04 Uncertainty inference(continuous)

Unit 4 Uncertainty Inference - Continuous

Wang Yuan-Kai 王元凱ykwangmailsfjuedutw

httpwwwykwangtw

Department of Electrical Engineering Fu Jen Univ輔仁大學電機工程系

2006~2011

Bayesian Networks

Fu Jen University Department of Electrical Engineering Wang Yuan-Kai Copyright

Reference this document as Wang Yuan-Kai ldquoUncertainty Inference - Continuous

Lecture Notes of Wang Yuan-Kai Fu Jen University Taiwan 2011

王元凱 Unit - Uncertainty Inference (Continuous) p


Goal of this Unit Review basic concepts of

statistics in terms of Image processing Pattern recognition

2



Related Units Previous unit(s) Probability Review

Next units Uncertainty Inference (Discrete) Uncertainty Inference (Continuous)

3



Self-Study Artificial Intelligence a modern

approach Russell amp Norvig 2nd Prentice Hall

2003 pp462~474 Chapter 13 Sec 131~133

統計學的世界墨爾著鄭惟厚譯天下文化2002

深入淺出統計學 D Grifiths 楊仁和譯2009 Orsquo Reilly

4



55

Contents1 Gaussian helliphelliphelliphelliphelliphelliphelliphelliphelliphelliphellip 62 Gaussian Mixtures 363 Linear Gaussian 804 Sampling 925 Markov Chain 1026 Stochastic Process 1067 Reference helliphelliphelliphelliphelliphelliphelliphelliphelliphelliphellip 114



6

1 Gaussian Distribution 11 Univariate Gaussian 12 Bivariate Gaussian 13 Multivariate Gaussian



7

Why Should We CareGaussians are as natural as

Orange Juice and SunshineWe need them to understand

mixture modelsWe need them to understand

Bayes Optimal ClassifiersWe need them to understand

Bayes Network



8

11 Univariate Gaussian Univaraite Gaussian is a

Gaussian with only one variable

2exp

21)(

2xxp

1]Var[ X0][ XE

Unit-variance Gaussian



9

General Univariate Gaussian

2

2

2)(exp

21)(

xxp

2]Var[ XμXE ][

=100

=15

bull It is also called Normal distributionbull Bell-shape curve



10

Normal Distribution

bull X ~ N()bull ldquoX is distributed as a Gaussian with

parameters and 2rdquobull In this figure X ~ N(100152)

=100

=15



11

A Live Demo and are two parameters of the

Gaussian Position parameter Shape parameter

Demo



12

Cumulative Distribution Function

x xxdxedxxpxF

22 2)(

21)()(

Density Function for the Standardized Normal Variate

0

005

01

015

02

025

03

035

04

045

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Den

sity

Cumulative Distribution Function for a Standardized Normal Variate

0010203040506070809

1

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Prob

abilt

y



13

The Error Functionbull Assume X ~ N(01)bull Define ERF(x) = P(Xltx)

= Cumulative Distribution of X

x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14

Using The Error FunctionAssume X ~ N(01)P(Xltx| 2) = )( 2

xERF



15

The Central Limit TheoremIf (X1 X2 hellip Xn) are iid continuous

random variablesThen define As n p(z) Gaussian with

mean E[Xi] and variance Var[Xi]

n

iin x

nxxxfz

121

1)(



16

Example ndashZero Mean Gaussian amp Noise Zero mean Gaussian N(0) Usually used as noise model in

images An image f(xy) with noise N(0)

means f(xy) = g(xy) + N(0)



17

12 Bivariate Gaussian



18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2

1 vector randomFor XX

X

22

21

1212

Σ

If X N( )



19

Gaussian Parameters )()(exp

||||2

1)( 121

21 μXμXXp T Σ

Σ

amp are Gaussianrsquos parameters

2

1

μ

22

21

1212

Σ

Position parameter Shape parameter



20

Graphical Illustration

1

2

1

2

X1

X2p(X)

X1

Position parameter Shape parameter 1 2

Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22

Axis-Aligned Gaussian X1 and X2 are independent or

uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2

σ1 gt σ2 σ1 lt σ2



23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24

Degenerated Gaussians

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25

Example ndash Clustering (14) Given a set of data points in a 2D

space Find the Gaussian distribution of

those points



26

Example ndash Clustering (24) A 2D space example Face verification of a person We use 2 features to verify the person Size Length

We get 1000 faceimages of the person

Each image has 2 features a data pointin the 2D space

Find the mean and range of 2 features



27

Example ndash Clustering (34)

x and y are dependent



28


x and y are almost independent




29

13 Multivariate Gaussian

Tm

m

XXX

X

XX

)( vector randomFor 212

1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31

Axis-Aligned Gaussians

m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33

Degenerate Gaussians

m

2

1

μ 0|||| Σ

x1

x2



34

Example ndash3-Variate Gaussian (12)

3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35


Assume a simple case ij=0 if inej

32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36

2 Gaussian Mixture Modelbull What is Gaussian Mixture

bull 2 Gaussians are mixed to be a pdfbull Why Gaussian Mixture

bull Single Gaussian is not enough Usually the distribution of your data

is assumed as one Gaussian Also called unimodal Gaussian

However sometimes the distribution of data is not a unimodal Gaussian



37

Why Is Unimodal Gaussian not Enough (13)

A univariate example Histogram of an image



38


Bivariate example

One Gaussian PDF



39


To solve it

Mixture of Three Gaussians



40

Gaussian Mixture Model(GMM)

21 Combine Multiple Gaussians

22 Formula of GMM23 Parameter Estimation

of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x

bull Unimodal Gaussian (Single Gaussian)

bull Multi-modal Gaussians (Multiple Gaussians)



42

Combine 2 Gaussians (14) Suppose Two Gaussians in 1-dimension

1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44

Combine 2 Gaussians (34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46

Combine 2 Gaussians with Weights (13)

If p(x) = frac12 p(x | C1) + frac12 p(x | C2)

p(x|Ci)dx = 1 p(x)dx = p(x|C1)dx + p(x|C2)dx = 1 + 1 = 2

p(x) = p(x | C1) + p(x | C2)

p(x)dx = frac12 p(x|C1)dx + frac12 p(x|C2)dx = 1

If p(x) = 1 p(x | C1) + 2 p(x | C2) 1+2=1 p(x)dx = 1 p(x|C1)dx + 2 p(x|C2)dx = 1



47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49

Combine 2 Gaussians with Different Mean Distances (12)

Suppose Two Gaussians in 1D

1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51

Combine 2 Gaussians with Different Weights (12)


1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53

2D Gaussian Combination (12)

1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55

More Gaussians As no of Gaussians M

increases it can represent any possible density By adjusting M and of

each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59

22 Formula of GMM A Gaussian mixture model

(GMM) is a linear combinationof M Gaussians

M

iii Cxpxp

1) |()(

bull P(x) is the probability of a point xbullx=(Cb Cr) or (RGB) or

bull i is mixing parameter (weight)bull p(x|Ci) is a Gaussian function



60

Comparison of Formula

In GMM p(x|Ci) means the probability of x in the i Gaussian component

1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61

Two Constraints of GMMbull i

bull p(x|Ci)bull It is normalized bull ie p(x|Ci)dx = 1

1 0 and 11

M

iii



62

The Problem (15) Now we know any density can be

obtained by Gaussian mixture Given the mixture function we can

plot its density But in reality what we need to do

in computer is We get a lot of data point =xj Rn

j=1hellipN with unknown density Can we find the mixture function of

these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

Histograms of =xj Rn j=1hellipN



64

The Problem (35) To find the mixture function

means to estimate the parameters of the mixture function Mixing parameters Gaussian component densities Mean vector i

Covariance matrix i Number of components M



65

The Problem (45)

1D Gaussian 2D Gaussian 3D Gaussian 1 2 3

() 1 3 6Total 2 5 9

1D GMM 2D GMM 3D GMM M M M 1 M 2 M 3 M

() 1 M 3 M 6 MTotal 3M 6M 10M

GMM with M Gaussians

A GaussianNo of Parameters



66

The Problem (55) That is given xj Rn j=1hellipN

M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(

Usually we use i to denote (i i)

M

iii xpxp

1) |()(

Solve i i i

Also called parameter estimation



67

23 Parameter Estimation Given Fixed M Data =xj Rn j=1hellipN We may calculate the histogram of

We want to find the parameters = (1 M 1 M 1 M)that best fit the histogram of data

Examples 1-D example xj R1

Two 2-D examples xj R2



68

1-D Example=15 -02 14 18 Histogram

bullNx=-25=10bullbullNx=15=40bull

2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7

M=4(equal variances)



70

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44ANEMIA PATIENTS AND CONTROLS

Red Blood Cell Volume

Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

2-D Example=(345402)



71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480

490LOG-LIKELIHOOD AS A FUNCTION OF EM ITERATIONS

EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

ANEMIA DATA WITH LABELS

Anemia Group

Control Group



79

Parameter Estimation of GMM

Two methods Maximum Likelihood Estimation

(MLE) Expectation Maximization (EM)

Discussed in Lecture Note 24



80

3 Linear Gaussian P(X) can belong to a distribution Ex Gaussian uniform Exponential

Gaussian mixture) P(Y|X) conditional probability Can the conditional probability

belong to a distribution Linear Gaussian describes The distribution of conditional

probability as Gaussian The dependence between random

variables



81

- +

N(y)

y

From Gaussian to Linear Gaussian (12)

If two variables x and y has linearrelationship we say y = ax + b a and b are parameters

If y belongs to Gaussian distribution y ~ N( ) = N(y ) But y = ax+b ax+b ~ N( ) = N(ax+b ) But we can write

N(y ax+b )

y becomes a random variable



82

Linear Gaussian=N(yax+b ) The meaning of linear Gaussian

N(y ax+b )

-3 - + +3

N(y)

y

bull When y=ax+b N(y) is the maximum probability

bull However yax+boccurs

bullwith lower probability bulldecayed as in Gaussian distributionax+b

=



83

P(y|x) = N(yax+b ) P(Xj=y|Xi=x)=P(y|x) If P(y|x)=N(y ax+b ) Xj varies linearly with Xi With Gaussian uncertainty Standard deviation is fixed

2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84

Example 1 (12) Illumination change of an image

g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b

bullTwo kinds of illumination change(same ab)bullReal light change g1bullChanged by image processing software g2

bullg1 = g2 NobullBecause of noise g1 is a random variable



85


f(xy) g1(xy)

g1 =af+b

bullBut g1 is a random variablebullIt means g1 undergoes a noisebullg1 ~ af + b + N(0) = N(af+b )bullOr P(g1|f) = N(af+b )



86

Extension Linear Transform X and Y are two vectors Y = (y1 y2 hellip ym)T

X = (x1 x2 hellip xm)T

X and Y are linearly dependent Y = AX + B Linear transform

If Y becomes a random vector P(Y|X)=N(Y AX+B sum)



87

Example 2 (15) Illumination change of color image

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B

bullF(xy)=(rF(xy) gF(xy) bF(xy)T

bullG(xy)=(rG(xy) gG(xy) bG(xy)T

G =AF+B



88

Example 2 (25) A simple case of G=AF+B aij=0 if inej

RG=a11RF+b1 GG=a22GF+b2 BG=a33BF+b3

001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89

Example 2 (35) If G has noises

F(xy) G1(xy)

G1 =AF+B

bullG1(xy) is a random vectorbullIt means G1 undergoes a noisebullG1 ~ AF + B + N(0sum)

= N(AF+B sum)bullOr P(G1|F) = N(AF+B sum)

333231

232221

131211



90

Example 2 (45) G1 ~ N(AF+B sum) N is a multivariate Gaussian (3-D)

333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211

bbagarabbagarabbagara

BAF

FFF

FFF

FFF



91

Example 2 (55) Assume a simple case aij=0 if inej ij=0 if inej

Then

P(rG|rF) = N(a11rF+b1 1) P(gG|gF) = N(a21gF+b2 2) P(bG|bF) = N(a31bF+b3 3)

100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92

4 Sampling Generate N samples S from P(X) S=(s1s2 hellip sN)

X can be a random variable or a random vector If X=x si=xi

If X=(x1x2hellipxn) si=(x1ix2

ihellipxni)

Why generate N samples Estimate probabilities by frequencies

NxXxXP

with samples)(



93

Example (12) A simple example coin toss Tossing the coin get head or tail It is a Boolean random variable coin = head or tail Random variable but not

random vector If it is unbiased coin head and tail

have equal probability A prior probability distribution

P(Coin) = lt05 05gt Uniform distribution

But we do not know it is unbiased



94

Example (22) Sampling in this example

= flipping the coin many times N eg N=1000 times Ideally 500 heads 500 tails P(head) = 5001000=05

P(tail) = 5001000=05 Practically 5001 heads 499 tails P(head) = 5011000=0501

P(tail) = 4991000=0499 By the sampling we can

estimate the probability distribution



95

Sampling (Math) For a Boolean random variable X P(X) is prior distribution

= ltP(x) P(x)gt Using a sampling algorithm to

generate N samples Say N(x) is the number of samples

that x is true N(x) of x is false

)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96

Sampling Algorithm It is the algorithm to Generate samples from a known

probability distribution Estimate the approximate

probability How does a sampling algorithm

generate a sample CC++ rand() Return 0 ~ RAND_MAX (32767)

Java random()

P



97

A Sampling Algorithm of the Coin Toss

Flip the coin 1000 times int coin_face

for (i=0 ilt1000 i++) if (rand() gt RAND_MAX2)

coin_face = 1else coin_face = 0

What kind of distribution



98

Sampling Algorithms for Many RVs (12)

3 Boolean random variables X Y Z (X=1 Y=0 Z=0) is called a sample int X Y Z

for (i=0 ilt1000 i++) if (rand() gt RAND_MAX2) X = 1

else X = 0if (rand() gt RAND_MAX2) Y = 1

else Y = 0 if (rand() gt RAND_MAX2) Z = 1

else Z = 0 X Y Z are all

uniform distribution



99


Y Z are not uniform distribution P(Y)=lt067 033gt

P(Z)=lt025075gt int X Y Z




else Z = 0



100

Various Sampling Algorithms For more complex P(X) we need

more complex sampling algo Stochastic simulation Direct Sampling Rejection sampling Reject samples disagreeing with

evidence Likelihood weighting Use evidence to weight samples

Markov chain Monte Carlo (MCMC) Also called Gibbs sampling



101

Example Approximate reasoning for

Bayesian networks TBU



102

5 Markov Chain Markov Assumption Each state at time t only

depends on the state at time t-1

Ex The weather today only depends on the weather of yesterday

(Andrei Andreyevich Markov)

X1 X2 X3

t=1 t=2 t=3



103

Deterministic vs Non-Deterministic

Deterministic patterns Traffic light FSMs hellip

Non-Deterministic patterns Weather Speech Tracking hellip



104

Example ndashWeather Prediction (12)

Only 3 possible weather states Sunny Cloudy Rainy

Transition Matrix A=Pr( today | yesterday)

Weather Today Sunny Cloudy Rainy

Sunny 05 025 025 Cloudy 0375 0125 0375

Weather

Yesterday Rainy 0125 0625 0375



105


Suppose we know the weather of previous days t=1 rainy t=2 sunny t=3 sunny t=4 cloudy

Predict the weather of day t=5

X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106

6 Stochastic Process

Also called Random Process It is a collection of random

variables For each t in the index set T X(t) is a

random variable Usually t refers to time and X(t) is

the state of the process at time t X(t) can be discrete or continuous



107

Graphical View of Stochastic Process



108

Statistics of Stochastic Process

Mean of X(t) Variance standard deviation of X(t) Frequency distribution of X(t) P(X) Conditional probability of X(t)

P(X(t) | X(t-1))



109

Markov Chain A Markov chain is a stochastic

process where P(X(t+1) | X(t) X(t-1) hellip X(0))

= P(X(t+1) | X(t)) or P(Xn+1 | Xn Xn-1 hellip X0)

= P(Xn+1 | Xn) Next state depends only on current

state Future and past are conditionally

independent given current



110

Higher-order Markov Chain Second-order Markov chain P(X(t+1) | X(t) X(t-1) hellip X(0))

= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111

Stationary Process The probability distribution of X is

independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112

Doubly Stochastic Process Hidden variable

Y1 Y3

X1 X2 X3

Y2



113

Example - Video Facial expression recognition TBU



114

7 Reference Moments

Hu M K [1962] ldquoVisual Pattern Recognition by Moment Invariantsrdquo IRE Trans Info Theory vol IT-8 pp 179-187

Gonzalez RC and R E Woods [2001] Digital Image Processing 2nd Prentice Hall pp659-660 672-675

L Rocha et al ldquoImage Moments-Based Structuring and Tracking of Objectsrdquo Proceedings of the XV Brazilian Symposium on Computer Graphics and Image Processing 2002



115

Reference Gaussian mixture

C M Bishop Neural Networks for Pattern Recognition Oxford University Press 1995

McLachlan and Peel Finite Mixture Models John Wiley amp Sons

Rennie ldquoA Short Tutorial on Using EM with Mixture Modelsrdquo MIT Tech Report 2004 httpwwwaimitedu peoplejrenniewritingmixtureEMpdf

Tomasi ldquoEstimating Gaussian Mixture Density with EM a tutorialrdquo Duke University

Y Weiss Motion segmentation using EM ndash a short tutorial 1997 httpwwwcshujiacil˜yweissemTutorialpdf



116

Reference Linear Gaussian

Russell amp Norvig Artificial Intelligence a modern approach 2nd Prentice Hall 2003Sec 143 pp501-503

Sampling RussellampNorvig Artificial Intelligence a modern

approach 2nd Prentice Hall 2003Sec 145 pp511-518

Stochastic process Probability Random variables and Random

Signal Principles 4e Peebles McGraw Hill 2001



117

General References 統計學的世界鄭惟厚譯天下文化2002

Statistics concepts and controversies 5th D S Moore 2001

Sampling Chap 1-4 隨機程序與機率楊政穎等譯滄海

Probability Random variables and Random Signal Principles 4e Peebles McGraw Hill 2001 Moment Chap 3 Random process Chap 67



Goal of this Unit Review basic concepts of

statistics in terms of Image processing Pattern recognition

2





3





2003 pp462~474 Chapter 13 Sec 131~133



4



55




6




7





Bayes Network



8



2exp

21)(

2xxp

1]Var[ X0][ XE




9


2

2

2)(exp

21)(

xxp

2]Var[ XμXE ][

=100

=15




10

Normal Distribution



=100

=15



11



Demo



12


x xxdxedxxpxF

22 2)(

21)()(


0

005

01

015

02

025

03

035

04

045

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Den

sity


0010203040506070809

1

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Prob

abilt

y



13



x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117









3





2003 pp462~474 Chapter 13 Sec 131~133



4



55




6




7





Bayes Network



8



2exp

21)(

2xxp

1]Var[ X0][ XE




9


2

2

2)(exp

21)(

xxp

2]Var[ XμXE ][

=100

=15




10

Normal Distribution



=100

=15



11



Demo



12


x xxdxedxxpxF

22 2)(

21)()(


0

005

01

015

02

025

03

035

04

045

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Den

sity


0010203040506070809

1

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Prob

abilt

y



13



x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117









2003 pp462~474 Chapter 13 Sec 131~133



4



55




6




7





Bayes Network



8



2exp

21)(

2xxp

1]Var[ X0][ XE




9


2

2

2)(exp

21)(

xxp

2]Var[ XμXE ][

=100

=15




10

Normal Distribution



=100

=15



11



Demo



12


x xxdxedxxpxF

22 2)(

21)()(


0

005

01

015

02

025

03

035

04

045

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Den

sity


0010203040506070809

1

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Prob

abilt

y



13



x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







55




6




7





Bayes Network



8



2exp

21)(

2xxp

1]Var[ X0][ XE




9


2

2

2)(exp

21)(

xxp

2]Var[ XμXE ][

=100

=15




10

Normal Distribution



=100

=15



11



Demo



12


x xxdxedxxpxF

22 2)(

21)()(


0

005

01

015

02

025

03

035

04

045

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Den

sity


0010203040506070809

1

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Prob

abilt

y



13



x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







6




7





Bayes Network



8



2exp

21)(

2xxp

1]Var[ X0][ XE




9


2

2

2)(exp

21)(

xxp

2]Var[ XμXE ][

=100

=15




10

Normal Distribution



=100

=15



11



Demo



12


x xxdxedxxpxF

22 2)(

21)()(


0

005

01

015

02

025

03

035

04

045

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Den

sity


0010203040506070809

1

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Prob

abilt

y



13



x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







7





Bayes Network



8



2exp

21)(

2xxp

1]Var[ X0][ XE




9


2

2

2)(exp

21)(

xxp

2]Var[ XμXE ][

=100

=15




10

Normal Distribution



=100

=15



11



Demo



12


x xxdxedxxpxF

22 2)(

21)()(


0

005

01

015

02

025

03

035

04

045

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Den

sity


0010203040506070809

1

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Prob

abilt

y



13



x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







8



2exp

21)(

2xxp

1]Var[ X0][ XE




9


2

2

2)(exp

21)(

xxp

2]Var[ XμXE ][

=100

=15




10

Normal Distribution



=100

=15



11



Demo



12


x xxdxedxxpxF

22 2)(

21)()(


0

005

01

015

02

025

03

035

04

045

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Den

sity


0010203040506070809

1

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Prob

abilt

y



13



x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







9


2

2

2)(exp

21)(

xxp

2]Var[ XμXE ][

=100

=15




10

Normal Distribution



=100

=15



11



Demo



12


x xxdxedxxpxF

22 2)(

21)()(


0

005

01

015

02

025

03

035

04

045

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Den

sity


0010203040506070809

1

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Prob

abilt

y



13



x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







10

Normal Distribution



=100

=15



11



Demo



12


x xxdxedxxpxF

22 2)(

21)()(


0

005

01

015

02

025

03

035

04

045

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Den

sity


0010203040506070809

1

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Prob

abilt

y



13



x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







11



Demo



12


x xxdxedxxpxF

22 2)(

21)()(


0

005

01

015

02

025

03

035

04

045

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Den

sity


0010203040506070809

1

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Prob

abilt

y



13



x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







12


x xxdxedxxpxF

22 2)(

21)()(


0

005

01

015

02

025

03

035

04

045

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Den

sity


0010203040506070809

1

-5 -4 -3 -2 -1 0 1 2 3 4 5

Standard Deviations

Prob

abilt

y



13



x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







13



x

z

dzzpxERF )()(

x

z

dzz2

exp21 2



14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







14


xERF



15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







15




n

iin x

nxxxfz

121

1)(



16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







16






17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







17




18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







18

The Formula

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

2

1

μ

2


X

22

21

1212

Σ

If X N( )



19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







19


||||2

1)( 121

21 μXμXXp T Σ

Σ


2

1

μ

22

21

1212

Σ




20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







20


1

2

1

2

X1

X2p(X)

X1


Principal axis



21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







21

General Gaussian

X1

X2

2

1

μ

22

21

1212

Σ

X1

X2



22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







22


uncorrelated

X1

X2

2

1

μ

22

12

00

Σ

X1

X2




23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







23

Spherical Gaussian

2

2

00

Σ

X1

X2

2

1

μ



24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







24


)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

0|||| Σ

X1

X2

2

1

μ



25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







25



those points



26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







26







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







27





28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







28






29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







29


Tm

m

XXX

X

XX


1

X

If X N( )

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp

m

2

1

μ

mmm

m

m

221

222

21

11212

Σ



30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







30

Gaussian Parameters



m

2

1

μ

mmm

m

m

221

222

21

11212

Σ

)()(exp||||)2(

1)( 121

21

2μxΣμx

Σx T

mp



31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







31


m

2

1

μ

m

m

2

12

32

22

12

00000000

000000000000

Σ

X1

X2

X1

X2



32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







32

Spherical Gaussians

m

2

1

μ

2

2

2

2

2

00000000

000000000000

Σ

x1

x2



33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







33


m

2

1

μ 0|||| Σ

x1

x2



34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







34


3

2

1

μ

32

3231

2322

21

131212

Σ

)()(exp||||)2(

1)( 121

21

23 μxΣμx

Σx Tp



35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







35



32

22

12

000000

Σ

)()(exp||||)2(

1)( 121

21

23

μxΣμxΣ

x Tp



36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







36








37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







37





38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







38


Bivariate example

One Gaussian PDF



39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







39


To solve it




40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







40




of GMM



41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







41


1

2 1 2

1 1( ) exp22

Tnp x x x

11 1 12 1 2

1

12 2 22 1 2

2

1 1( ) exp22

1 1 exp22

Tn

Tn

p x x x

x x





42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







42


1 21 2( ) ( | ) ( | )p x p x p x

2

2

1| exp 1222

ii i

ii

xp x i

p(x) = p(x | C1) + p(x | C2)

p(x|Ci)dx = 1



43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







43

1-D Example (24)

2

2

2

2

1 ( 4)( ) exp( )2 032 03

1 ( 64) exp( )2 052 05

xp x

x

1=4 1=032=64 2=05

1=062=04

Given x=52

2

2

2

1 (5 4)( 5) exp( )2 032 03

1 (5 64) exp( )2 052 05

p x



44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







44


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

Gaussian Mixture

N(01)=p(x|01)N(31)=p(x|31)

p(x)=p(x|01)+p(x|31)



45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







45


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x)=p(x|01)+p(x|34)



46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







46




p(x) = p(x | C1) + p(x | C2)





47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







47


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05

x

p(x)

N(01) N(31)

2 Gaussians

N(01)=p(x|01)N(31)=p(x|31)

p(x) = frac12 p(x|01)+ frac12 p(x|31)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01) N(31)



48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







48


-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

052 Gaussians

x

p(x)

N(01)

N(34)

N(01)=p(x|01)N(34)=p(x|34)

p(x) = frac12 p(x|01)+ frac12 p(x|34)

-4 -3 -2 -1 0 1 2 3 4 5 6 70

005

01

015

02

025

03

035

04

045

05Gaussian Mixture

x

p(x)

N(01)

N(34)



49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







49



1 21 1

1 22 2( ) ( | ) ( | )p x p x p x

Let = 1 Let =



50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







50

=1 =2

=3 =4




51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







51



1 21 2( ) 075 ( | ) 025 ( | )p x p x p x

Let = 1 Let =



52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







52

=1 =2

=3 =4




53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







53


1 1 1 2 2 1

4 0 4 0( | ) (00) ( | ) (03)

0 4 0 4p x C p x C



54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







54


p(x) = p(x|C1) + p(x|C2)



55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







55



each Gaussians

1

1

( ) ( | )

( | )

M

i iiM

i i ii

p x p x C

p x



56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







56

-5 0 5 100

05

1

15

2

Component Modelsp(

x)

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)

5 Gaussians



57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







57



58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







58



59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







59



M

iii Cxpxp

1) |()(





60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







60



1

2 1 21 1( ) exp

22

Tnp x x x

1

12 1 2

1

( ) ( | )

1exp22

M

i ii

MTi

i i ini i

p x p x C

x x

Gaussian

GMM



61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







61



1 0 and 11

M

iii



62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







62






these data points



63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







63

-5 0 5 100

05

1

15

2

Component Models

p(x)

5 Gaussians

-5 0 5 100

01

02

03

04

05

Mixture Model

x

p(x)




64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







64






65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







65

The Problem (45)


() 1 3 6Total 2 5 9


() 1 M 3 M 6 MTotal 3M 6M 10M





66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







66


M

iiii xpxp

111 ) |()(

M

iiii xpxp

122 ) |()(

M

iiiNiN xpxp

1) |()(


M

iii xpxp

1) |()(

Solve i i i




67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







67







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







68



2

2

1 ( 15)( ) exp( )2 132 13xp x

=15 =13

ParameterEstimation



69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







69

(Izenman ampSommer)

(Basford et al)

M=3

M=7M=7




70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







70

33 34 35 36 37 38 39 437

38

39

4

41

42

43



Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion




71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







71

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 1

Initialization



72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







72

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 3



73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







73



74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







74

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 10



75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







75

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 15



76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







76

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion

EM ITERATION 25



77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







77

0 5 10 15 20 25400

410

420

430

440

450

460

470

480


EM Iteration

Log-

Like

lihoo

d



78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







78

33 34 35 36 37 38 39 437

38

39

4

41

42

43

44


Red

Blo

od C

ell H

emog

lobi

n C

once

ntra

tion


Anemia Group

Control Group



79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







79







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







80





variables



81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







81

- +

N(y)

y




N(y ax+b )




82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







82


N(y ax+b )

-3 - + +3

N(y)

y




=



83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







83


2

2

2))((exp

21

)(

)|(

baxy

baxyN

xXyXP ij

XiXj

P(Xj | Xi)



84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







84


g(xy) =af(xy)+b

f(xy) g(xy)

g =af+b





85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







85


f(xy) g1(xy)

g1 =af+b




86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







86







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







87


3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G

F(xy) G(xy)

G =AF+B



G =AF+B



88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







88



001

100030002

F

F

F

G

G

G

bgr

bgr

3

2

1

333231

232221

131211

bbb

bgr

aaaaaaaaa

bgr

F

F

F

G

G

G



89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







89


F(xy) G1(xy)

G1 =AF+B



333231

232221

131211



90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







90


333231

232221

131211

)()(exp||||2

1)( 121

21 μXμXXp T Σ

Σ

3333231

2232221

1131211


BAF

FFF

FFF

FFF



91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







91


Then


100030002

A

200080004

333

222

111

3333231

2232221

1131211

bbabgabra


BAF

F

F

F

FFF

FFF

FFF



92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







92




ihellipxni)


NxXxXP

with samples)(



93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







93








94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







94








95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







95





)(ˆ)( )(ˆ)( xPN

xNxPN

xN

)()(lim )()(lim xPN

xNxPN

xNNN



96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







96





Java random()

P



97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







97








98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







98










99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







99







else Z = 0



100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







100







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







101





102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







102





X1 X2 X3

t=1 t=2 t=3



103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







103






104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







104





Sunny 05 025 025 Cloudy 0375 0125 0375

Weather




105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







105




X5

t=5

X1

t=1

R

X2

t=2

S

X3

t=3

S

X4

t=4

C



106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







106








107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







107




108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







108



P(X(t) | X(t-1))



109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







109









110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







110


= P(X(t+1) | X(t) X(t-1))

X1

t=1

X2

t=2

X3t=3

X4

t=4

Third-order n-order



111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







111


independent of t

X1

t=1

X2

t=2

X3t=3

X4

t=4



112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







112


Y1 Y3

X1 X2 X3

Y2



113




114

7 Reference Moments






115









116









117







113




114

7 Reference Moments






115









116









117







114

7 Reference Moments






115









116









117







115









116









117







116









117







117





Education

04 Uncertainty inference(continuous)