Bayesian Regression An Introduction Using R
J Guzmán, PhD
January 2011
Multiple Regression • Model:
µ|X,β = β0 + β1x1 + … + βkxk
y | X,β = β0 + β1x1 + … + βkxk + ε
ε ~ N(0, σ2) • k predictors or explanatory variables • n observations
Multiple Regression • Matrix notation:
y |β,σ2, X ~ N(Xβ, σ2I)
• X matrix of explanatory variables • β vector of regression coefficients • y response or variable of interest
vector • ε vector of normally distributed random
errors
Example: Birds’ Extinction • Albert, 2009. Bayesian Computation with R,
2d. Edition. Springer. • Birds’ extinction – data from 16 islands • species – name of bird species • time – average time of extinction on the
islands • nesting – average number of nesting pairs • size – species’ size, 1 = large; 0 = small • status – species’ status; 1 = resident,
0 = migrant
Example
library(LearnBayes) data(birdextinct) attach(birdextinct) birdextinct[1:15, ]
Linear Model • Graphical display: log.time = log(time) plot(log.time ~ nesting)
• Linear Model t.lm = lm(log.time ~ nesting + size + status, x = T, y = T )
summary(t.lm)
Birds’ Extinction
• From output: a. longer extinction times for species with
large no. of nesting pairs
b. smaller extinction times for large birds
c. longer extinction times for resident birds
Bayesian Regression
• Model: y | β, σ2, X ~ N(Xβ, σ2I) • Two unknown parameters: β & σ2
• Assume un-informative joint prior: f(β, σ2) ∝ 1/σ2
• Joint posterior: f(β, σ2 | y) ∝ f(β | y, σ2) × f(σ2 | y)
Bayesian Regression • Marginal posterior distribution of β,
conditional on error variance σ2 is Multivariate Normal with:
Mean = b = (XtX)-1Xty var(β) = σ2 ⋅ (XtX)-1
• Marginal posterior distribution of σ2 is Inverse Gamma[(n - k)/2, S/2]
S = (y - Xb)t (y - Xb)
Prediction
• To predict future yp , based on vector xp
• Conditional on β & σ2, yp ~ N(xpβ, σ2)
• Posterior predictive distribution f(yp|y) = ∫f(yp|β, σ2)f(β, σ2|y)dβdσ2
Computation • Simulate from joint posterior of β & σ2:
– Simulate value of σ2 from f(σ2| y) – Simulate value of β from f(β | y, σ2)
• Use Albert’s blinreg( ) to perform simulation
• To predict mean response µy use Albert’s blinregexpected( )
• To predict future response yp use Albert’s blinregpred( )
blinreg( )
• To sample from joint posterior of β & σ2 • Inputs: y, X & no. of simulations m par.sample = blinreg(t.lm$y, t.lm$x, 5000)
S = sum(t.lm$residuals^2) shape=t.lm$df.residual/2; rate = S/2 library(MCMCpack) sigma2 = rinvgamma(1, shape, scale = 1/rate)
Computation • β is simulated from Multivariate
Normal: Mean = b = (XtX)-1Xty var(β) = σ2 ⋅ (XtX)-1
MSE = sum(t.lm$residuals^2)/ t.lm$df.residual
vbeta = vcov(t.lm)/MSE beta = rmnorm(1, t.lm$coef,
vbeta*sigma2)
Graphical Display
op = par(mfrow = c(2,2)) hist(par.sample$beta[, 2],
main= "Nesting", xlab=expression(beta[1]))
hist(par.sample$beta[, 3], main= "Size", xlab=expression(beta[2]))
Graphical Display
hist(par.sample$beta[, 4], main= "Status", xlab = expression(beta[3]))
hist(par.sample$sigma, main= "Error SD", xlab=expression(sigma))
par(op)
Parameters’ Summary
• Compute 2.5th , 50th & 97.5th percentiles
apply(par.sample$beta, 2, quantile, c(.025, .5, .975))
quantile(par.sample$sigma,
c(.025, .5, .975))
Mean Response Covariate
Set
Nesting
Size
Status A
4
Small
Migrant
B
4
Small
Resident
C
4
Large
Migrant
D
4
Large
Resident
Mean Response
cov1 = c(1, 4, 0, 0) cov2 = c(1, 4, 1, 0) cov3 = c(1, 4, 0, 1) cov4 = c(1, 4, 1, 1) X1 =rbind(cov1,cov2,cov3,cov4) mean.draw = blinregexpected(X1,
par.sample)
Mean Response
op = par(mfrow = c(2,2)) hist(mean.draw[, 1],
main = "Covariate Set A", xlab = "Log Time")
hist(mean.draw[, 2],
main = "Covariate Set B", xlab = "Log Time")
Mean Response hist(mean.draw[, 3],
main = "Covariate Set C", xlab = "Log Time")
hist(mean.draw[, 4],
main="Covariate Set D", xlab="Log Time")
par(op)
Means Response’ Summary
• Compute 2.5th , 50th & 97.5th percentiles
apply(mean.draw, 2, quantile, c(.025, .5, .975))
Future Response pred.draw=blinregpred(X1, par.sample) op = par(mfrow = c(2,2)) hist(pred.draw[, 1], main="Covariate Set A", xlab="Log Time")
hist(pred.draw[, 2], main="Covariate Set B", xlab="Log Time")
Future Response
hist(pred.draw[, 3], main="Covariate Set C", xlab="Log Time")
hist(pred.draw[, 2], main="Covariate Set D", xlab="Log Time")
par(op)
Future Response’ Summary
• Compute 2.5th , 50th & 97.5th percentiles
apply(pred.draw, 2, quantile, c(.025, .5, .975))