8/10/2019 ho.max.lik1

1/3

8. The principle of maximum likelihood and the derivation of the likelihood ratio test statistic

(Rewritten with Dan Yarlett, TA)

Suppose we wish to estimate the parameter of a coin,p= Prob (Head on a toss). We might

toss the coin n= 100 times and observe r= ! Heads. " fami#iar estimate ofpisn

rp=$ = 0.!. %n what

sense is this estimate,n

rp=$ , the &best' estimate ofp ne answer that is a#read* fami#iar to us is thatp$ is an unbiasedestimate ofp+ i.e., this choice of estimate #eads to a perfect fit between the expectedand

the observednumber of Heads. We now derive another answer b* app#*ing the princip#e of maimum

#i-e#ihood.

he probabi#it* (or #i-e#ihood, l) of observing theparticular se/uence of r Hs and n-rs that

we observed is

rnr pppl = )1()( .

Likelihood, l(p), as function of p, of

sequence of r Hs & 20-r Ts, r = 8, 10, 12

p

!8"#$02%02

Likelihood

000001%

000001#

0000012

0000010

0000008

000000%

000000#

0000002

00000000

n = 20 tosses

r&n = $

r&n = #

r&n = %

o i##ustrate the properties of this #i-e#ihood function, we p#ot it in the above 2igure for a specific se/uence

of rHeads and n-rai#s, n= 30 and r= 4, 10 and 13. "s can be seen in the 2igure, for given rand n, l(p)

varies between 0, whenp= 0 orp= 1, and a maimum, l*, whenp = r/n. %t is not hard to show, in genera#

(e.g., b* using the 5a#cu#us), that l(p)is alwaysa maimum whenn

rp= . herefore, usingn

rp=$ asour estimate ofpmaximiesthe observed #i-e#ihood of the data, and this is wh* we ca## this estimate the

&maximum likelihood (ML) estimate o p.' %t can be shown b* mathematica# arguments that, in genera#,67 estimates have ver* desirab#e properties. 2or eamp#e, even in cases where an 67 estimate is biased,

it can be shown that (i) as the samp#e si8e becomes ver* #arge, the 67 estimate tends to be unbiased, and

(ii) the variance of the 67 estimate is the sma##est among a set of reasonab#e, competing estimates 9 e.g.,

the variance of the 67 estimate is no greater than that of the unbiased #inear estimate.

7et l*denote the maximum possibleva#ue of l, i.e., the va#ue obtained b* puttingn

rp=

8/10/2019 ho.max.lik1

2/3

rnr

n

rn

n

rl

=: .

We can confirm in the above 2igure that this maimum depends on nand r. ;ow suppose that, before weco##ect the data, we have a nu## h*pothesis thatphas the specific va#ue,p!(e.g.,p!= 0.!). "fter we co##ect

the data, we can now sa* that the #i-e#ihood of the data, i "!were true, wou#d be

( ) ( ) rnr

ppl

= 000 1 .

We -now that l!cannot be greater than l*, but if l!were not si#niicantly lessthan l*, we wou#d agree that

H0is a tenab#e h*pothesis and retain it. However, if l!weresi#niicantly lessthan l*, we wou#d agree that

H0is an untenab#e h*pothesis and resing the fact that lo#(ab) = lo#(a) & lo#(b), which imp#ies

that lo#(ak) = klo#(a), it can be seen that

3

8/10/2019 ho.max.lik1

3/3

+

=

)(

)1(#n)(#n

:#n 000

0 rn

pnrn

r

npr

l

lL ,

where n' stands for &natura# #ogarithms' or ogs to the base e'. o genera#i8e this epression forL!, #et

us rep#ace the observed fre/uencies, rand n-r, b* 'i, and the epected fre/uencies, np!and n(-p!), b*i.

hen we get

= i

i

i

i ')

'L #n0 .

he test statistic we actua##* use is notL!, but rather 93L!, because it can be shown that the distribution of

93L!is chi9s/uare with df e/ua# to the number of parameters that were set in H0. o sum up, the teststatistic,

==

i

i

i

i ')

'L* #n33 03

,

has the chi9s/uare distribution under the nu## thatp = p!. "s it turns out, +tends to have approimate#*

the same va#ue as the fami#iar, Pearson chi9s/uare inde,

=i i

ii

)

)' 3

3 )( .

Hence the resu#ts of the #i-e#ihood ratio test are genera##* the same as those of the Pearson chi9s/uare test.

?