Upload
duongnguyet
View
218
Download
2
Embed Size (px)
Citation preview
3. The F Test for ComparingReduced vs. Full Models
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 1 / 43
Assume the Gauss-Markov Model with normal errors:
y = Xβ + ε, ε ∼ N(0, σ2I).
Suppose C(X0) ⊂ C(X) and we wish to test
H0 : E(y) ∈ C(X0) vs. HA : E(y) ∈ C(X) \ C(X0).
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 2 / 43
The “reduced” model corresponds to the null hypothesis andsays that E(y) ∈ C(X0), a specified subspace of C(X).
The “full” model says that E(y) can be anywhere in C(X).
For example, suppose
X0 =
111111
and X =
1 0 01 0 00 1 00 1 00 0 10 0 1
.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 3 / 43
In this case, the reduced model says that all 6 observations havethe same mean.
The full model says that there are three groups of twoobservations. Within each group, observations have the samemean. The three group means may be different from oneanother.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 4 / 43
For this example, let µ1, µ2, and µ3 be the elements of β in thefull model, i.e., β = [µ1, µ2, µ3]
′. Then, for the full model,
E(y) = Xβ =
1 0 01 0 00 1 00 1 00 0 10 0 1
µ1
µ2
µ3
=
µ1
µ1
µ2
µ2
µ3
µ3
, and
H0 : E(y) ∈ C(X0) vs. HA : E(y) ∈ C(X) \ C(X0)
is equivalent to
H0 : µ1 = µ2 = µ3 vs. HA : µi 6= µj, for some i 6= j.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 5 / 43
For the general case, consider the test statistic
F =y′(PX − PX0)y/ [rank(X)− rank(X0)]
y′(I − PX)y/ [n− rank(X)].
To show that this statistic has an F distribution, we will use thefollowing fact:
PX0PX = PXPX0 = PX0 .
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 6 / 43
There are many ways to see that this fact is true. First,
C(X0) ⊂ C(X) =⇒ Each column of X0 ∈ C(X)
=⇒ PXX0 = X0.
Thus,
PXPX0 = PXX0(X′0X0)−X′0 = X0(X′0X0)
−X′0= PX0 .
This implies that
(PXPX0)′ = P′X0
=⇒ P′X0P′X = P′X0
=⇒ PX0PX = PX0 . 2
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 7 / 43
Alternatively,
∀ a ∈ Rn, PX0a ∈ C(X0) ⊂ C(X).
Thus, ∀ a ∈ Rn, PXPX0a = PX0a.
This implies PXPX0 = PX0.
Transposing both sides of this equality and using symmetry ofprojection matrices yields
PX0PX = PX0 . 2
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 8 / 43
Alternatively, C(X0) ⊂ C(X) =⇒ XB = X0 for some B becauseevery column of X0 must be in C(X).
Thus,
PX0PX = X0(X′0X0)−X′0PX = X0(X′0X0)
−(XB)′PX
= X0(X′0X0)−B′X′PX = X0(X′0X0)
−B′X′
= X0(X′0X0)−(XB)′ = X0(X′0X0)
−X′0 = PX0 .
PXPX0 = PXX0(X′0X0)−X′0 = PXXB(X′0X0)
−X′0
= XB(X′0X0)−X′0 = X0(X′0X0)
−X′0 = PX0 .
2Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 9 / 43
Note that PX − PX0 is a symmetric and idempotent matrix:
(PX − PX0)′ = P′X − P′X0
= PX − PX0 .
(PX − PX0)(PX − PX0) = PXPX − PXPX0 − PX0PX + PX0PX0
= PX − PX0 − PX0 + PX0
= PX − PX0 .
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 10 / 43
Now back to determining the distribution of
F =y′(PX − PX0)y/ [rank(X)− rank(X0)]
y′(I − PX)y/ [n− rank(X)].
An important first step is to note that
F =y′(
PX−PX0σ2
)y/ [rank(X)− rank(X0)]
y′( I−PX
σ2
)y/ [n− rank(X)]
.
Now we can show that the numerator is a chi-square randomvariable divided by its degrees of freedom, independent of thedenominator, which is a central chi-square random variabledivided by its degrees of freedom. Once we show all thesethings, we will have established that the statistic F has an Fdistribution (see Slide Set 1 Slide 35).
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 11 / 43
We will use the following result from Slide Set 1.
Suppose Σ is an n× n positive definite matrix.
Suppose A is an n× n symmetric matrix of rank m such thatAΣ is idempotent (i.e., AΣAΣ = AΣ).
Then y ∼ N(µ,Σ) =⇒ y′Ay ∼ χ2m(µ
′Aµ/2).
For the numerator of our F statistic, we have
µ = Xβ, Σ = σ2I, A =
(PX − PX0
σ2
), and
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 12 / 43
m = rank(A) = rank(
PX − PX0
σ2
)= rank(PX − PX0)
= tr(PX − PX0) = tr(PX)− tr(PX0)
= rank(PX)− rank(PX0) = rank(X)− rank(X0).
(Multiplying by a nonzero constant does not affect the rank of amatrix. Rank is the same as trace for idempotent matrices.Trace of a difference is the same as the difference of traces. Therank of a projection matrix is equal to the rank of the matrixwhose column space is projected onto.)
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 13 / 43
To verify that Σ is positive definite, note that for any a ∈ Rn \ {0},
a′Σa = a′(σ2I)
a = σ2a′a = σ2n∑
i=1
a2i > 0.
To verify that AΣ is idempotent, we have
AΣ =
(PX − PX0
σ2
)(σ2I) = PX − PX0 .
Thus, we have
y′(
PX − PX0
σ2
)y ∼ χ2
rank(X)−rank(X0)
(12β′X′
(PX − PX0
σ2
)Xβ
).
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 14 / 43
Denominator Distribution
From the result on Slide Set 2 Slide 19, we have
y′(
I − PX
σ2
)y ∼ χ2
n−rank(X).
(This fact can be proved using the result on Slide Set 1 Slide 31,following the same type of argument used to show thedistribution of the numerator.)
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 15 / 43
Independence of Numerator and Denominator
By Slide Set 1 Slide 38,
y′(
PX−PX0σ2
)y is independent of y′
( I−PXσ2
)y because
(PX − PX0
σ2
)(σ2I)
(I − PX
σ2
)=
1σ2 (PX − PXPX − PX0 + PX0PX)
=1σ2 (PX − PX − PX0 + PX0) = 0.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 16 / 43
Thus, it follows that
F =y′(PX − PX0)y/ [rank(X)− rank(X0)]
y′(I − PX)y/ [n− rank(X)]
∼ Frank(X)−rank(X0),n−rank(X)
(β′X′(PX − PX0)Xβ
2σ2
).
If H0 is true, i.e. if E(y) = Xβ ∈ C(X0), then the noncentralityparameter is 0 because
(PX − PX0)Xβ = PXXβ − PX0Xβ
= Xβ − Xβ = 0.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 17 / 43
In general, the noncentrality parameter quantifies how far themean of y is from C(X0) because
β′X′(PX − PX0)Xβ
= β′X′(PX − PX0)′(PX − PX0)Xβ
= || (PX − PX0)Xβ ||2= || PXXβ − PX0Xβ ||2
= || Xβ − PX0Xβ ||2= || E(y)− PX0E(y) ||2 .
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 18 / 43
Note that
y′(PX − PX0)y = y′ [(I − PX0)− (I − PX)] y
= y′(I − PX0)y− y′(I − PX)y
= SSEREDUCED − SSEFULL.
Also rank(X)−rank(X0)
= [n− rank(X0)]− [n− rank(X)]
= DFEREDUCED − DFEFULL,
where DFE = Degrees of Freedom for Error.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 19 / 43
Thus, the F statistic has the familiar form
(SSEREDUCED − SSEFULL)/(DFEREDUCED − DFEFULL)
SSEFULL/DFEFULL.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 20 / 43
Equivalence of F Tests
It turns out that this reduced vs. full model F test is equivalent tothe F test for testing
H0 : Cβ = d vs. HA : Cβ 6= d
with an appropriately chosen C and d.
The equivalence of these tests is proved in STAT 611.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 21 / 43
Example: F Test for Lack of Linear Fit
Suppose a balanced, completely randomized design is used toassign 1, 2, or 3 units of fertilizer to a total of 9 plots of land.
The yield harvested from each plot is recorded as the response.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 22 / 43
Let yij denote the yield from the jth plot that received i units offertilizer (i, j = 1, 2, 3).
Suppose all yields are independent and yij ∼ N(µi, σ2) for all
i, j = 1, 2, 3.
If y =
y11
y12
y13
y21
y22
y23
y31
y32
y33
, then E(y) =
µ1
µ1
µ1
µ2
µ2
µ2
µ3
µ3
µ3
.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 23 / 43
Suppose we wish to determine whether there is a linearrelationship between the amount of fertilizer applied to a plot andthe expected value of a plot’s yield.
In other words, suppose we wish to know if there exists realnumbers β1 and β2 such that
µi = β1 + β2(i) for all i = 1, 2, 3.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 24 / 43
Consider testing H0 : E(y) ∈ C(X0) vs. HA : E(y) ∈ C(X) \ C(X0),where
X0 =
1 11 11 11 21 21 21 31 31 3
and X =
1 0 01 0 01 0 00 1 00 1 00 1 00 0 10 0 10 0 1
.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 25 / 43
Note H0 : E(y) ∈ C(X0)⇐⇒ ∃ β ∈ R2 3 E(y) = X0β
⇐⇒ ∃[β1
β2
]∈ R2 3
µ1
µ1
µ1
µ2
µ2
µ2
µ3
µ3
µ3
=
1 11 11 11 21 21 21 31 31 3
[β1
β2
]=
β1 + β2(1)β1 + β2(1)β1 + β2(1)β1 + β2(2)β1 + β2(2)β1 + β2(2)β1 + β2(3)β1 + β2(3)β1 + β2(3)
⇐⇒ µi = β1 + β2(i) for all i = 1, 2, 3.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 26 / 43
Note E(y) ∈ C(X)⇐⇒ ∃ β ∈ R3 3 E(y) = Xβ
⇐⇒ ∃
β1
β2
β3
∈ R3 3
µ1
µ1
µ1
µ2
µ2
µ2
µ3
µ3
µ3
=
1 0 01 0 01 0 00 1 00 1 00 1 00 0 10 0 10 0 1
β1
β2
β3
=
β1
β1
β1
β2
β2
β2
β3
β3
β3
.
This condition clearly holds with βi = µi for all i = 1, 2, 3.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 27 / 43
The alternative hypothesis
HA : E(y) ∈ C(X) \ C(X0)
is equivalent to
HA : There do not exist β1, β2 ∈ R such that
µi = β1 + β2(i) ∀ i = 1, 2, 3.
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 28 / 43
Because the lack of fit test is a reduced vs. full model F test, wecan also obtain this test by testing
H0 : Cβ = d vs. HA : Cβ 6= d
for appropriate C and d.
β =
µ1
µ2
µ3
C =? d =?
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 29 / 43
R Code and Output
> x=rep(1:3,each=3)> x[1] 1 1 1 2 2 2 3 3 3>> y=c(11,13,9,18,22,23,19,24,22)>> plot(x,y,pch=16,col=4,xlim=c(.5,3.5),+ xlab="Fertilizer Amount",+ ylab="Yield",axes=F,cex.lab=1.5)> axis(1,labels=1:3,at=1:3)> axis(2)> box()
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 30 / 43
●
●
●
●
●
●
●
●
●
Fertilizer Amount
Yie
ld
1 2 3
1015
20
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 31 / 43
> X0=model.matrix(˜x)> X0
(Intercept) x1 1 12 1 13 1 14 1 25 1 26 1 27 1 38 1 39 1 3
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 32 / 43
> X=model.matrix(˜0+factor(x))> X
factor(x)1 factor(x)2 factor(x)31 1 0 02 1 0 03 1 0 04 0 1 05 0 1 06 0 1 07 0 0 18 0 0 19 0 0 1
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 33 / 43
> proj=function(x){+ x%*%ginv(t(x)%*%x)%*%t(x)+ }>> library(MASS)> PX0=proj(X0)> PX=proj(X)
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 34 / 43
> Fstat=(t(y)%*%(PX-PX0)%*%y/1)/+ (t(y)%*%(diag(rep(1,9))-PX)%*%y/(9-3))> Fstat
[,1][1,] 7.538462>> pvalue=1-pf(Fstat,1,6)> pvalue
[,1][1,] 0.03348515
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 35 / 43
> reduced=lm(y˜x)> full=lm(y˜0+factor(x))>> rvsf=function(reduced,full)+ {+ sser=deviance(reduced)+ ssef=deviance(full)+ dfer=reduced$df+ dfef=full$df+ dfn=dfer-dfef+ Fstat=(sser-ssef)/dfn/+ (ssef/dfef)+ pvalue=1-pf(Fstat,dfn,dfef)+ list(Fstat=Fstat,dfn=dfn,dfd=dfef,+ pvalue=pvalue)+ }
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 36 / 43
> rvsf(reduced,full)$Fstat[1] 7.538462
$dfn[1] 1
$dfd[1] 6
$pvalue[1] 0.03348515
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 37 / 43
> anova(reduced,full)Analysis of Variance Table
Model 1: y ˜ xModel 2: y ˜ 0 + factor(x)
Res.Df RSS Df Sum of Sq F Pr(>F)1 7 78.2222 6 34.667 1 43.556 7.5385 0.03349 *---Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 38 / 43
> test=function(lmout,C,d=0){+ b=coef(lmout)+ V=vcov(lmout)+ dfn=nrow(C)+ dfd=lmout$df+ Cb.d=C%*%b-d+ Fstat=drop(
t(Cb.d)%*%solve(C%*%V%*%t(C))%*%Cb.d/dfn)+ pvalue=1-pf(Fstat,dfn,dfd)+ list(Fstat=Fstat,pvalue=pvalue)+ }> test(full,matrix(c(1,-2,1),nrow=1))$Fstat[1] 7.538462$pvalue[1] 0.03348515
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 39 / 43
SAS Code and Output
data d;input x y;cards;
1 111 131 92 182 222 233 193 243 22;run;
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 40 / 43
proc glm;class x;model y=x;contrast ’Lack of Linear Fit’ x 1 -2 1;
run;
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 41 / 43
The SAS System
The GLM Procedure
Dependent Variable: y
Sum ofSource DF Squares Mean Square F Value Pr > F
Model 2 214.2222222 107.1111111 18.54 0.0027
Error 6 34.6666667 5.7777778
Corrected Total 8 248.8888889
R-Square Coeff Var Root MSE y Mean
0.860714 13.43684 2.403701 17.88889
Source DF Type I SS Mean Square F Value Pr > F
x 2 214.2222222 107.1111111 18.54 0.0027
Source DF Type III SS Mean Square F Value Pr > F
x 2 214.2222222 107.1111111 18.54 0.0027
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 42 / 43
Contrast DF Contrast SS Mean Square F Value Pr > F
Lack of Linear Fit 1 43.55555556 43.55555556 7.54 0.0335
Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 43 / 43