36
Bayesian Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Embed Size (px)

Citation preview

Page 1: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Bayesian Statistics and R

Peng Ding,School of Mathematical Sciences,

Peking Univ.

December 16, 2008

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 2: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Introduction

• What is Probability?• Frequentist: LLN,CLT• Bayesian: Prior and Posterior• What is Statistics?

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 3: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Statistical Methodology

• Moment Estimation( Karl Pearson)• Maximum Likelihood Estimation( Gauss, R. A. Fisher)• Bayesian Method( Bayes)• Empirical Bayesian( Robbins)• ...?

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 4: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Frequentist vs Bayesian

• Frequentist:· parameters are constant to be estimate· point estimation and interval estimation• Bayesian:· parameters are random variables· Prior + Model→ Posterior· all information are contained in posterior distribution

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 5: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Bayes’ Formula and Bayesian Statistics

• Bayes’ Formula:If θ ∈ Θ has prior distribution π(θ), and the observed data ycomes from conditional distribution p(y |θ). Then the posteriordistribution of θ given y is

π(θ|y) =p(y |θ)π(θ)∫

Θ p(y |θ)π(θ)dθ

• Bayesian Statictics:· choose prior· model observed data· inference based on posterior distribution

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 6: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

How to choose Prior?

• Congugate Prior Distribution:· Prior and Posterior have the same form· Congugate Prior for Exponential Family

p(yi |θ) = f (yi )g(θ) exp{

φ(θ)Tu(yi )}

p(y |θ) =

(n∏

i=1

f (yi )

)g(θ)n exp

{φ(θ)T

n∑

i=1

u(yi )

}

Choose Prior as:

p(θ) ∝ g(θ)η exp{φ(θ)Tν}The Posterior is:

p(θ|y) ∝ g(θ)n+η exp{φ(θ)T (ν +n∑

i=1

u(yi ))}

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 7: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

How to choose Prior?

• Non-informative Prior Distribution:· Baysian Assumption

p(θ) ∝ constant, θ ∈ Θ

· Jefferys Prior

π(θ) ∝ |I (θ)|1/2

where I (θ) is the Fisher Information Matrix.

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 8: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

How to explore the Posterior?

• Direct Caculation:· The posterior has explicit and simple form!• Simulation the Posterior:· Sampling from the posterior distribution· Markov Chain Monte Carlo( MCMC):

Gibbs Sampler and Metropolis- Hasting Algorithm

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 9: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Gibbs Sampling

• X ∼ π(x), x = (x1, ..., xn)

• Initial value x (0) = (x(0)1 , ..., x

(0)n )

• The t-th iteration:· Sample x

(t)1 ∼ π(x1|x (t−1)

2 , ..., x(t−1)n );

· ...· Sample x

(t)i ∼ π(xi |x (t)

1 , ..., x(t)i−1, x

(t−1)i+1 , ..., x

(t−1)n );

· ...· Sample x

(t)n ∼ π(xn|x (t)

1 , ..., x(t)n−1).

• Under some regular conditions, the distribution of x converges tothe stationary distribution of the Markov Chain: π(x).

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 10: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

M-H Algorithm

• Given an irreducible transition probability q(·, ·),• Given a function

α(·, ·) = min{1,π(x ′)q(x ′, x)

π(x)q(x , x ′)}, 0 < α ≤ 1.

• At time t, X (t) = x ,· Generate a potential transition x → x ′ by q(x , ·);· With probability α(x , x ′), accept x ′; with probability 1− α(x , x ′)stay at x .• Under some regular conditions, π(x) is the stationary distributionof this Markov Chain.

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 11: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Bayesian Statistics in R

• MCMCpack

library(lattice)library(coda)library(MASS)library(MCMCpack)

• Famous software WinBUGS: Bayesian inference Using GibbsSampling.

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 12: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Exmaple 1: Binomial Distribution

• Modelp(y |θ) ∝ θy (1− θ)n−y

• Priorp(θ) ∝ θα−1(1− θ)β−1

• Posteriorp(θ|y) ∝ θα+y−1(1− θ)β+n−y−1

• R code

posterior <-MCbinomialbeta(y=3,n=12,alpha=1,beta=1,mc=5000)summary(posterior)plot(posterior)

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 13: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Result for Binomial Distribution

1. Empirical mean and standard deviation for each variable,plus standard error of the mean:

Mean SD Naive SE Time-series SE0.286442 0.116600 0.001649 0.001924

2. Quantiles for each variable:

2.5% 25% 50% 75% 97.5%0.08972 0.19926 0.27805 0.36167 0.53906

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 14: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Posterior for Binomial Parameter

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 15: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Exmaple 2: Poisson Distribution

• Model

p(y |λ) ∝n∏

i=1

λyi e−λ

• Priorp(λ) ∝ eβλλα−1

• Posteriorλ|y ∼ Gamma(α + ny , β + n)

• R code

y<-rpois(1000,lambda=2)posterior <- MCpoissongamma(y, 15, 1, 5000)summary(posterior)plot(posterior)

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 16: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Posterior for Poisson Parameter

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 17: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Exmaple 3: Normal Distribution with Variance known

• Model

p(y |µ) ∝ exp{− 1

2σ2

n∑

i=1

(yi − µ)2}

• Prior

p(µ) ∝ exp{− 1

2τ20

(µ− µ0)2}

• Posteriorµ|y ∼ N(µ1, τ

21 )

where

µ1 =

µ0

τ20

+ nyσ2

1τ20

+ 1σ2

, y =n∑

i=1

yi/n,

1

τ21

=1

τ20

+1

σ2.

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 18: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Exmaple 3: Normal Distribution with Variance known

• R code

y<-rnorm(1000,5,1)posterior <-MCnormalnormal(y, sigma2=1, mu0=0,tau20=100, mc=5000)summary(posterior)plot(posterior)

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 19: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Posterior for Normal Parameter: µ

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 20: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Exmaple 4: Normal Distribution with unknown Variance

• Model

p(y |µ) ∝ exp{− 1

2σ2

n∑

i=1

(yi − µ)2}

• Prior( Semi-Congugate)

µ ∼ N(b0,B20 )

σ2 ∼ Inverse− χ2(2c0, 2d0)

• R code

y<-rnorm(1000,2,1)posterior<-MCMCregress(y~1, b0 = 0, B0 =0, c0= 0.001, d0 = 0.001)summary(posterior)plot(posterior)

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 21: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Posterior for Normal Parameter: µ and σ

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 22: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Exmaple 5: Multinomial Distribution

• Model

p(y |θ) ∝n∏

i=1

θyii

• Prior

p(θ|α) ∝n∏

i=1

θαi−1i

• R code

posterior <-MCmultinomdirichlet(c(727,583,137), c(1,1,1), mc=10000)summary(posterior)plot(posterior)

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 23: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Posterior for Multinomial Parameter

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 24: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Generalized Linear Model

• E (y |x) = g−1(βT x)• Different g(·)–Link Function, different models· Linear Regression: g(t) = t: E (y |x) = βT x

· Logistic Regression: g(t) = log( t1−t ): log( P(y=1|x)

1−P(y=1|x)) = βT x

· Probit Regression: g(t) = Φ−1(t): Φ−1(P(y = 1|x)) = βT x· Poisson Regression: g(t) = log(t): log(E (y |x)) = βT x

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 25: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Exmaple 6: Linear Regression

X<-rnorm(100,2,1)Y<-1+2*X+rnorm(100,0,1)posterior <- MCMCregress(Y~X,b0 = 0, B0 = 0,

c0 = 0.001, d0 = 0.001,verbose=1000)plot(posterior)summary(posterior)

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 26: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Posterior for Linear Regression

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 27: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Exmaple 7: Logistic Regression

x<-rnorm(1000)y<-rbinom(1000,1,exp(1-x)/(1+exp(1-x)))posterior <-MCMClogit(y~x, b0=0, B0=.001)plot(posterior)summary(posterior)

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 28: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Posterior for Logistic Regression

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 29: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Exmaple 8: Probit Regression

y<-rbinom(1000,1,pnorm(1-x))posterior <- MCMCprobit(y~x, b0=0,B0=.001)plot(posterior)summary(posterior)

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 30: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Posterior for Probit Regression

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 31: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Exmaple 9: Poisson Regression

x<-rnorm(100)y<-rpois(100,exp(1+x))posterior <- MCMCpoisson(y ~x)plot(posterior)summary(posterior)

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 32: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Posterior for Probit Regression

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 33: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Other Models

• Gaussian Mixture Model• Latent Class Analysis• Hierachical Models• Perhaps any parametric models: Examples in WinBUGS• Bayesians believes that all inference and more is Bayesianterritory.—Bayesian Nonparametrics, J. K. Ghosh and R. V.Ramamoorthi, Springer(2003)

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 34: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Reference and recommendatory books

• Andrew Gelman, John B Carlin, Hal S Stern and Donald BRubin( 2004), Bayesian Data Analysis, Chapman&Hall/CRC• Martin A. Tanner( 1996), Tools for Statistical Inference:Methods for Exploration of Posterior Distribution and LikelihoodFunctions, Springer• Bradley P. Carlin and Thomas A. Lious(2000), Bayes andEmpirical Bayes Methods for Data Analysis, Chapman&Hall/CRC• Mao Shi-song, Wang Jing-long and Pu Xiao-long( 2006),Advanced Mathematical Statistics, Higher Education Press(inChinese).

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 35: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

Acknowledgement

• I would like to thank the organizers of the first R conference inChina.• Also I am grateful to my mentor Professor Zhi Geng for hisintroduction of GLM and Bayesian Methods.

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R

Page 36: Bayesian Statistics and R - 统计之都 Statistics and R Peng Ding, School of Mathematical Sciences, Peking Univ. December 16, 2008 Peng Ding, School of Mathematical Sciences, Peking

The End

Thank you!

Peng Ding, School of Mathematical Sciences, Peking Univ. Bayesian Statistics and R