31
Exps Kxqwlqj lq KljkGlphqvlrqdo Gdwd Mhurph K1 Iulhgpdq ) Qlfkrodv L1 Ilvkhu Rfwrehu 5;/ 4<<; Devwudfw Pdq| gdwd dqdo|wlf txhvwlrqv fdq eh irupxodwhg dv +qrlv|, rswlpl}dwlrq sureohpv1 Wkh| h{solflwo| ru lpsolflwo| lqyroyh qglqj vlpxowdqhrxv frpelqdwlrqv ri ydoxhv iru d vhw ri +lqsxw, yduldeohv wkdw lpso| xqxvxdoo| odujh +ru vpdoo, ydoxhv ri dqrwkhu ghvljqdwhg +rxwsxw, yduldeoh1 Vshflfdoo|/ rqh vhhnv d vhw ri vxeuhjlrqv ri wkh lqsxw yduldeoh vsdfh zlwklq zklfk wkh ydoxh ri wkh rxwsxw yduldeoh lv frqvlghudeo| odujhu +ru vpdoohu, wkdq lwv dyhudjh ydoxh ryhu wkh hqwluh lqsxw grpdlq1 Lq dgglwlrq lw lv xvxdoo| ghvluhg wkdw wkhvh uhjlrqv eh ghvfuledeoh lq dq lqwhusuhwdeoh irup lqyroylqj vlpsoh vwdwhphqwv +uxohv, frqfhuqlqj wkh lqsxw ydoxhv1 Wklv sdshu suhvhqwv d surfhgxuh gluhfwhg wrzdugv wklv jrdo edvhg rq wkh qrwlrq ri sdwlhqw uxoh lqgxfwlrq1 Wklv sdwlhqw vwudwhj| lv frqwudvwhg zlwk wkh juhhg| rqhv xvhg e| prvw uxoh lqgxfwlrq phwkrgv/ dqg vhpl0juhhg| rqhv xvhg e| vrph sduwlwlrqlqj wuhh whfkqltxhv vxfk dv FDUW1 Dssolfdwlrqv lqyroylqj vflhqwlf dqg frpphufldo gdwd edvhv duh suhvhqwhg1 Nh|zrugv= Gdwd Plqlqj/ qrlv| ixqfwlrq rswlpl}dwlrq/ fodvvlfdwlrq/ dvvrfldwlrq/ uxoh lqgxfwlrq1 41 Lqwurgxfwlrq Wkh sxusrvh ri pdq| gdwd dqdo|vhv fdq eh ylhzhg lq wkh frqwh{w ri suhglfwlrq1 Wkh gdwd edvh frqwdlqv uhshdwhg revhuydwlrqv ri d ghvljqdwhg rxwsxw yduldeoh | dorqj zlwk vlpxowdqhrxv ydoxhv ri dgglwlrqdo lqsxw yduldeohv { @+{ >{ > >{ ,1 Wkh jrdo lv wr xvh wkhvh gdwd i|> { j +414, wr ghwhuplqh olnho| ydoxhv ri | iru vshflhg +ixwxuh, ydoxhv ri wkh lqsxwv {1 Vxssrvlqj wkh gdwd +414, lv d udqgrp vdpsoh iurp vrph +xqnqrzq, mrlqw glvwulexwlrq zlwk suredelolw| ghqvlw| s+|> {,/ wklv jrdo fdq eh fkdudfwhul}hg dv wu|lqj wr rewdlq wkh suredelolw| ghqvlw| ri |0ydoxhv dw hdfk { s+| m {,@ s+|> {, U s+|> {, g| zklfk lv prvw vlpso| +li lqfrpsohwho|, ghvfulehg e| lwv uvw prphqw i +{,@ H^| m {‘@ ] |s+| m {, g|= +415, Wklv txdqwlw| +415, lv zhoo nqrzq wr plqlpl}h phdq0vtxduhg suhglfwlrq huuru dw hdfk { i +{, @ duj plq H^+| i , m {=

ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

  • Upload
    others

  • View
    3

  • Download
    0

Embed Size (px)

Citation preview

Page 1: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

Exps Kxqwlqj lq Kljk�Glphqvlrqdo Gdwd

Mhurph K1 Iulhgpdq � ) Qlfkrodv L1 Ilvkhu �

Rfwrehu 5;/ 4<<;

DevwudfwPdq| gdwd dqdo|wlf txhvwlrqv fdq eh irupxodwhg dv +qrlv|, rswlpl}dwlrq sureohpv1 Wkh|

h{solflwo| ru lpsolflwo| lqyroyh �qglqj vlpxowdqhrxv frpelqdwlrqv ri ydoxhv iru d vhw ri +�lqsxw�,yduldeohv wkdw lpso| xqxvxdoo| odujh +ru vpdoo, ydoxhv ri dqrwkhu ghvljqdwhg +�rxwsxw�, yduldeoh1Vshfl�fdoo|/ rqh vhhnv d vhw ri vxeuhjlrqv ri wkh lqsxw yduldeoh vsdfh zlwklq zklfk wkh ydoxh riwkh rxwsxw yduldeoh lv frqvlghudeo| odujhu +ru vpdoohu, wkdq lwv dyhudjh ydoxh ryhu wkh hqwluh lqsxwgrpdlq1 Lq dgglwlrq lw lv xvxdoo| ghvluhg wkdw wkhvh uhjlrqv eh ghvfuledeoh lq dq lqwhusuhwdeohirup lqyroylqj vlpsoh vwdwhphqwv +�uxohv�, frqfhuqlqj wkh lqsxw ydoxhv1 Wklv sdshu suhvhqwv dsurfhgxuh gluhfwhg wrzdugv wklv jrdo edvhg rq wkh qrwlrq ri �sdwlhqw� uxoh lqgxfwlrq1 Wklv sdwlhqwvwudwhj| lv frqwudvwhg zlwk wkh juhhg| rqhv xvhg e| prvw uxoh lqgxfwlrq phwkrgv/ dqg vhpl0juhhg|rqhv xvhg e| vrph sduwlwlrqlqj wuhh whfkqltxhv vxfk dv FDUW1 Dssolfdwlrqv lqyroylqj vflhqwl�fdqg frpphufldo gdwd edvhv duh suhvhqwhg1

Nh|zrugv= Gdwd Plqlqj/ qrlv| ixqfwlrq rswlpl}dwlrq/ fodvvl�fdwlrq/ dvvrfldwlrq/ uxoh lqgxfwlrq1

41 LqwurgxfwlrqWkh sxusrvh ri pdq| gdwd dqdo|vhv fdq eh ylhzhg lq wkh frqwh{w ri �suhglfwlrq�1 Wkh gdwd edvhfrqwdlqv uhshdwhg revhuydwlrqv ri d ghvljqdwhg �rxwsxw� yduldeoh | dorqj zlwk vlpxowdqhrxv ydoxhv ridgglwlrqdo �lqsxw� yduldeohv { @ +{ � > { � > � � �> { � ,1 Wkh jrdo lv wr xvh wkhvh gdwd

i| � >{ � j � � +414,wr ghwhuplqh olnho| ydoxhv ri | iru vshfl�hg +ixwxuh, ydoxhv ri wkh lqsxwv {1 Vxssrvlqj wkh gdwd +414,lv d udqgrp vdpsoh iurp vrph +xqnqrzq, mrlqw glvwulexwlrq zlwk suredelolw| ghqvlw| s+|>{,/ wklv jrdofdq eh fkdudfwhul}hg dv wu|lqj wr rewdlq wkh suredelolw| ghqvlw| ri |0ydoxhv dw hdfk {

s+| m{, @ s+|>{,U s+|>{,g|

zklfk lv prvw vlpso| +li lqfrpsohwho|, ghvfulehg e| lwv �uvw prphqw

i+{, @ H^| m{` @]

| s+| m{,g|= +415,

Wklv txdqwlw| +415, lv zhoo nqrzq wr plqlpl}h phdq0vtxduhg suhglfwlrq huuru dw hdfk {

i+{, @ dujplq� H^+| � i, � m{`=���� ���������������������� �!��" #$��" %&#'���)(*�������)� ��� (,+-" �)�����'./%�%&��0 �&�$��� ���213���������&45��� ���)� ���$(,6/�)" 7��&�$# " �98�45��� ����� ��� (:4513.<;!=�>�?�@�AB C9D �FE/#$� �!�A #$�����)� ���$(-A �&()GIHJ 1K��L�MONQP��!� D �����!� " %���0SRQL��)� ���$���!��" ���T��%�" ���)%&��#&4U+-��%WV���(YXZ��[Y\!]�4U^/����� D MK8_()��4`^���acb_\�\&>)4U./G)#$� � ��0 " �)AB ^O" %�VdA ef" # D ���WE/%���" #&A 1K��L9MONgA .K6�H

Page 2: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

H{suhvvlqj wkh rxwsxw yduldeoh dv| @ H^| m{` . +| �H^| m{`, @ i+{, . % +416,

rqh vhhv wkdw wklv suhglfwlrq sureohp fdq eh fdvw dv rqh ri ixqfwlrq hvwlpdwlrq zkhuh wkh jrdo lv wrdssur{lpdwh +�ohduq�, d ghwhuplqlvwlf �wdujhw� ixqfwlrq i+{, iurp d vhw ri revhuydwlrqv zkhuh lwv ydoxh+dw hdfk {, lv fruuxswhg e| udqgrp qrlvh %1 Wklv qrlvh uhsuhvhqwv wkh udqgrp glvwulexwlrq ri wkh rxwsxw| derxw lwv phdq ydoxh i+{,/ dw hdfk {/ dqg fkdudfwhul}hv wkh idfw wkdw vshfli|lqj d vlpxowdqhrxv vhwri lqsxw ydoxhv grhv qrw vshfli| d xqltxh |0ydoxh> rwkhu idfwruv/ qrw fdswxuhg e| wkh vhw ri phdvxuhglqsxwv/ lq xhqfh wkh rxwsxw ydoxh1

Dowkrxjk uhvwulfwlqj dwwhqwlrq wr wkh �uvw prphqw +415, juhdwo| vlpsol�hv wkh sureohp/ lw lv vwlood iruplgdeoh rqh1 Wkh sureohp ri dffxudwho| dssur{lpdwlqj d jhqhudo ixqfwlrq ri pdq| dujxphqwv/hyhu|zkhuh zlwklq vrph grpdlq ri lqsxw ydoxhv/ edvhg rq vdpsohg gdwd +zlwk ru zlwkrxw qrlvh, uhpdlqvd gl!fxow rqh1 Dv vxfk/ lw kdv ehhq wkh irfxv ri pxfk uhvhdufk lq wkh �hogv ri pdwkhpdwlfv/ qhxudoqhwzrunv/ pdfklqh ohduqlqj/ dqg vwdwlvwlfv ^ vhh Oruhqw} +4<;9,/ Elvkrs +4<<8,/ Ulsoh| +4<<9,/ Plwfkhoo+4<<:,/ dqg Zdked +4<<3,`1 Wkh vhw ri ixqfwlrqv dphqdeoh wr dffxudwh dssur{lpdwlrq zlwk fxuuhqwphwkrgrorj| lv vwloo uhodwlyho| vpdoo dqg wkhuh pd| zhoo eh pdq| wdujhw ixqfwlrqv wr eh hqfrxqwhuhglq sudfwlfh wkdw uhpdlq hoxvlyh1

Riwhq/ ixqfwlrq dssur{lpdwlrq lv dssolhg lq vlwxdwlrqv iru zklfk wkh dfwxdo gdwd dqdo|wlf jrdo lvidu pruh prghvw> wkh lqwhuhvw lv lq vrph surshuw| ri wkh wdujhw ixqfwlrq1 D frpprq surfhgxuh lqvxfk vlwxdwlrqv lv wr dwwhpsw wr hvwlpdwh wkh wdujhw i+{, hyhu|zkhuh lq wkh lqsxw vsdfh dqg dvfhuwdlqwkh surshuw| ri lqwhuhvw iurp wkh uhvxowlqj hvwlpdwh ai+{,1 Iuhtxhqwo| krzhyhu/ wklv vwudwhj| fdq ehfrxqwhu�surgxfwlyh lq wkdw dq dowhuqdwlyh rqh irfxvhg gluhfwo| rq hvwlpdwlqj wkh surshuw| ri lqwhuhvwpd| jlyh ulvh wr kljkhu dffxudf|1

Rqh h{dpsoh ri wklv skhqrphqrq lv +5 0 fodvv, fodvvl�fdwlrq1 Khuh wkh rxwsxw yduldeoh | dvvxphvwzr ydoxhv/ | @ 3 lqglfdwlqj wkh �uvw fodvv dqg | @ 4 lqglfdwlqj wkh vhfrqg1 Wkh wdujhw ixqfwlrq lv

i+{, @ H^| m{` @ Su+| @ 4 m{, +417,dqg wkh rswlpdo +plqlpxp huuru, fodvvl�fdwlrq ghflvlrq lv

| @ 4+i+{, A 4@5,> +418,zkhuh 4+�, wdnhv wkh ydoxh rqh zkhq lwv dujxphqw lv wuxh dqg }hur rwkhuzlvh1 Lw lv frpprq wr dsso|ixqfwlrq dssur{lpdwlrq phwkrgrorj| lq wklv vlwxdwlrq/ lqvhuwlqj wkh hvwlpdwh ai+{, lq +418, wr pdnhsuhglfwlrqv a|1 Ydsqln +4<<8, dqg Iulhgpdq +4<<:, kdyh vkrzq wkdw wklv fdq surylgh srru shuirupdqfhuhodwlyh wr surfhgxuhv wkdw wu| wr gluhfwo| hvwlpdwh wkh ghflvlrq erxqgdu| i+{, @ 4@5/ dqg vrphwlphvohdgv wr frxqwhu�lqwxlwlyh uhvxowv1 Iru h{dpsoh lpsurylqj wkh txdolw| ri wkh ixqfwlrq hvwlpdwh fdqdfwxdoo| ghjudgh fodvvl�fdwlrq shuirupdqfh1

Dqrwkhu h{dpsoh lv +xqlyduldwh, ghqvlw| hvwlpdwlrq zkhuh wkh wdujhw i+{, lv wkh uhodwlyh suredelolw|ri dq revhuydwlrq dw {1 Wkh surshuw| ri lqwhuhvw lv wkh fxpxodwlyh glvwulexwlrq ixqfwlrq

I +{, @] �

���

i+{ � , g{ � = +419,

Lqvhuwlqj wkh rswlpdo nhuqho ghqvlw| hvwlpdwh ri i+{, lq +419, ohdgv wr d orzhu dffxudf| hvwlpdwh riI +{, wkdq vlpso| xvlqj wkh udz gdwd dv wkh ghqvlw| hvwlpdwh

aI +{, @ 4Q

�[��� �

4+{ � � {,

^Kdoo +4<;<,`1 Wkh udz gdwd lv ri frxuvh d yhu| pxfk srruhu hvwlpdwh ri wkh ghqvlw| i+{, lwvhoi1Dqrwkhu +wulyldo, h{dpsoh lv zkhq wkh surshuw| ri lqwhuhvw lv wkh phdq ri wkh wdujhw ixqfwlrq ryhu

wkh hqwluh lqsxw vsdfh�i @

]

i+{, s+{,g{= +41:,

5

Page 3: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

Khuh wkh vdpsoh rxwsxw phdq�| @ 4

Q�[��� �

| � +41;,

lv olnho| wr surylgh kljkhu dffxudf| wkdq wdnlqj wkh phdq ri vrph hvwlpdwh ai+{, ghulyhg iurp wkh gdwd1

51 Ixqfwlrq rswlpl}dwlrqIxqfwlrq dssur{lpdwlrq riwhq lv dssolhg lq frqwh{wv zkhuh wkh surshuw| ri lqwhuhvw lv wkh rswlpdo+pd{lpxp ru plqlpxp, ydoxhv ri wkh wdujhw i+{, +415,1 Vshfl�fdoo|/ rqh vhhnv d vxeuhjlrq ri wkhvsdfh ri lqsxw ydoxhv zlwklq zklfk wkh dyhudjh ydoxh ri wdujhw lv pxfk odujhu +ru vpdoohu, wkdw lwvdyhudjh ryhu wkh hqwluh lqsxw vsdfh +41:,1 Vlqfh plqlpl}lqj d ixqfwlrq fdq eh dfklhyhg e| pd{lpl}lqjlwv qhjdwlyh/ rqo| pd{lpl}dwlrq lv frqvlghuhg khuh zlwkrxw orvv ri jhqhudolw|1 Ohw V� eh wkh vhw ri doosrvvleoh ydoxhv iru wkh lqsxw yduldeoh { �

i{ � 5 V � j �� � � = +514,Wkh lqglylgxdo V � frxog uhsuhvhqw uhdo +shukdsv glvfuhwh, ydoxhv/ ru fdwhjrulfdo +xqrughudeoh, ydoxhv1Wkh hqwluh lqsxw grpdlq V fdq wkhq eh uhsuhvhqwhg e| wkh q0glphqvlrqdo +rxwhu, surgxfw vsdfh

V @ V � � V � � � � � � V � = +515,Wkh jrdo lv wr �qg d vxeuhjlrq U ri wkh lqsxw grpdlq V/ U � V/ iru zklfk

�i � @ dyh ����� i+{, @]

�����i+{, s+{,g{!

]

�����s+{,g{ AA �i +516,

zkhuh �i lv wkh dyhudjh ryhu wkh hqwluh lqsxw vsdfh +41:,1 Dq lpsruwdqw surshuw| ri dq| vxfk vxeuhjlrqlv lwv vl}h +�vxssruw�,

� � @]

�����s+{,g{= +517,

Dv zloo eh vhhq wkhuh lv jhqhudoo| d wudgh0r� ehwzhhq wkh ydoxhv ri �i � dqg � � lq wkdw odujhu ydoxhv riwkh odwwhu whqg wr ohdg wr vpdoohu ydoxhv ri wkh iruphu1

Vwudljkwiruzdug hvwlpdwhv ri wkhvh txdqwlwlhv +516, +517, uhvshfwlyho| zloo eh xvhg=a� � @ 4

Q[

�������4+{ � 5 U,> �| � @ 4

Q � a� �[

�������| � > +518,

zkhuh i| � j � � duh wkh revhuyhg +qrlv|, rxwsxw ydoxhv +414, +416,1 Dowkrxjk i| � j � � duh xvhg wr shuirupwkh dyhudjhv lw lv lpsruwdqw wr nhhs lq plqg wkdw wkh txdqwlw| ri lqwhuhvw lv �i � +516,1 Wklv zloo kdyhlpsruwdqw lpsolfdwlrqv rq wkh vwudwhj| hpsor|hg1

61 Gluhfw dssolfdwlrqvWkhuh duh pdq| flufxpvwdqfhv lq zklfk dwwhqwlrq lv gluhfwo| irfxvhg rq wkh h{wuhphv ri wkh wdujhwixqfwlrq i+{, +416,1 Lq iruhfdvwlqj ixwxuh uhwxuqv ri �qdqfldo vhfxulwlhv rqh lv jhqhudoo| lqwhuhvwhg lqlghqwli|lqj wkrvh wkdw surylgh wkh kljkhvw uhwxuq1 Khuh wkh rxwsxw | lv wkh uhwxuq dqg wkh lqsxwv {

pljkw eh sdvw uhwxuqv dqg ydulrxv hfrqrplf idfwruv1 Wkh vxssruw � � +517, uhsuhvhqwv wkh iudfwlrq risrwhqwldo ehwv1 Lq pdunhw uhvhdufk | pljkw uhsuhvhqw vrph fxvwrphu ehkdylru dqg wkh lqsxwv { zrxogeh ydulrxv ghprjudsklf yduldeohv1 Wkh vxssruw � � lv wkh vl}h ri wkh lghqwl�hg pdunhw vhjphqw1 Lqphglfdo dssolfdwlrqv wkh rxwsxw frxog eh d phdvxuh ri vhyhulw| ri dq looqhvv dqg wkh lqsxwv ydulrxvv|pswrpv dqg phglfdo phdvxuhphqwv1 Wkh jrdo zrxog eh wr lghqwli| wkh fkdudfwhulvwlfv ri wkh prvwvhyhuho| loo sdwlhqwv/ shukdsv iru ixuwkhu gldjqrvwlf whvwlqj ru hvshfldoo| djjuhvvlyh wuhdwphqw1 Khuhwkh vxssruw lv wkh iudfwlrq ri sdwlhqwv iru zklfk vxfk whvwlqj ru wuhdwphqw lv ihdvleoh1

6

Page 4: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

Lq lqgxvwuldo surfhvv frqwuro rqh lv riwhq lqwhuhvwhg lq pd{lpl}lqj vrph phdvxuh ri txdolw| +vwuhqjwk/gxudelolw|/ hwf1, ri wkh uhvxowlqj surgxfw1 Khuh | lv wkh txdolw| phdvxuh dqg wkh lqsxwv { duh ydulrxvsdudphwhuv wkdw frqwuro wkh surfhvv +frqfhqwudwlrqv ri ydulrxv fkhplfdov/ whpshudwxuhv/ pl{lqj wlphv/hwf1,1 Vrphwlphv wkhuh duh ydulrxv fkdudfwhulvwlf txdqwlwlhv i} � j �

� dvvrfldwhg zlwk d surgxfw +rshudwlqjyrowdjhv/ fxuuhqwv/ lpshgdqfhv, wkdw kdyh dvvrfldwhg wdujhw ydoxhv iw � j � � 1 Wkh jrdo lv wr �qg ydoxhvri wkh surfhvv frqwuro sdudphwhuv { wkdw uhvxow lq fkdudfwhulvwlf ydoxhv vlpxowdqhrxvo| forvh wr wkhluwdujhwv1 Khuh | frxog eh wdnhq wr eh

| @ ��[

� � �+} � � w � , � =

Lq wkhvh surfhvv frqwuro dssolfdwlrqv wkh vxssruw � � lv jhqhudoo| ri qr gluhfw lqwhuhvw1 Lw fdq eh ylhzhgdv d �phwd�0sdudphwhu ri wkh surfhgxuh zkrvh ydoxh lv wdnhq wr eh dv vpdoo dv srvvleoh frqvlvwhqw zlwkpd{lpl}lqj �i � +516, dv rssrvhg wr �| � +518,1

Lq doo ri wkhvh dssolfdwlrqv/ dv zhoo dv prvw rwkhuv lq gdwd dqdo|vlv/ wkh rxwsxw yduldeoh | uhsuhvhqwvqrlv| phdvxuhphqwv ri wkh wdujhw i+{,1 Wkh vhw ri revhuyhg lqsxwv vhogrp frpsohwho| fkdudfwhul}h doosrvvleoh idfwruv wkdw lq xhqfh ydoxhv ri wkh rxwsxw1

71 Lqgluhfw dssolfdwlrqvEhvlghv wkrvh wkdw gluhfwo| irfxv rq rswlpl}dwlrq/ wkhuh duh pdq| rwkhu frpprqo| dssolhg gdwd dqdo|wlfsurfhgxuhv wkdw fdq eh fdvw zlwklq wkdw iudphzrun1 Rqh vxfk h{dpsoh lv fodvvl�fdwlrq1 Khuh wkh dfwxdorxwsxw txdqwlw| dvvxphv +xqrughudeoh, fdwhjrulfdo ydoxhv if � j � � 1 Lq wklv fdvh wkhuh duh N +�gxpp|�,rxwsxw yduldeohv i| � @ 4+fodvv @ f � ,j �

� 1 Wkh lqsxwv { uhsuhvhqw wkh suhglfwru yduldeohv1 Wkh jrdo lvwr lghqwli| wkrvh uhjlrqv ri wkh lqsxw vsdfh zlwklq zklfk dq revhuydwlrq lv prvw olnho| wr eh iurp rqhri wkh lqglylgxdo fodvvhv1 Wkhvh duh wkh uhjlrqv zkhuh wkh fruuhvsrqglqj wdujhw

i � +{, @ H^| � m{` @ Su+| � @ 4 m{,lv odujhu wkdq wkdw ri dq| rwkhu fodvv1 Wkxv/ fodvvl�fdwlrq fdq eh ylhzhg dv �qglqj uhjlrqv lq zklfkhdfk i � +{, lv uhodwlyho| odujh1 Wkh vxssruw +517 ,+518, ri hdfk ri wkh uhjlrqv fdq eh uhjdughg dv phwd0sdudphwhuv mrlqwo| rswlpl}hg wr pd{lpl}h fodvvl�fdwlrq dffxudf| li suhglfwlrq lv wkh jrdo/ ru fkrvhq wrdlg lqwhusuhwdelolw| lq pruh gdwd dqdo|wlf vlwxdwlrqv1

Dqrwkhu frpprqo| dssolhg gdwd dqdo|wlf surfhgxuh lv foxvwhulqj1 Khuh wkh jrdo lv wr �qg uhjlrqvri wkh gdwd vsdfh wkdw duh uhodwlyho| ghqvho| srsxodwhg1 Wkdw lv/ wkh gdwd suredelolw| ghqvlw| s+{,lv odujh frpsduhg wr vrph ghvljqdwhg uhihuhqfh ghqvlw| s � +{, xvxdoo| +exw qrw dozd|v, wdnhq wr eh dxqlirup glvwulexwlrq ryhu wkh udqjh ri wkh gdwd1 Uhjlrqv zkhuh wkh udwlr

u+{, @ s+{,!s � +{, +714,lv odujh uhsuhvhqw orfdo frqfhqwudwlrqv ri gdwd lq h{fhvv ri wkdw suhglfwhg e| s � +{, +�foxvwhuv�,1 Uhjlrqvzkhuh u+{, lv vpdoo +�u+{, odujh, uhsuhvhqw orfdo odfn ri gdwd +�krohv�, wkdw dovr pd| eh ri lqwhuhvw1

Wkh udwlr +714, fdq eh pd{lpl}hg e| dvvljqlqj dq rxwsxw ydoxh | @ 4 wr hdfk gdwd revhuydwlrq1 DPrqwh Fduor vdpsoh ri frpsdudeoh vl}h lv wkhq jhqhudwhg iurp wkh uhihuhqfh glvwulexwlrq s � +{, dqgdvvljqhg wkh rxwsxw ydoxhv | @ 31 Wkh wdujhw ixqfwlrq

i+{, @ H^| m{` @ u+{,!+4 . u+{,,+ryhu wkh srrohg gdwd, lv prqrwrqlf lq u+{, vr wkdw lwv pd{lpd2plqlpd uhsuhvhqw foxvwhuv2krohv1

D sureohp forvho| uhodwhg wr foxvwhulqj lv dvvrfldwlrq1 Khuh rqh vhhnv uhjlrqv ri wkh gdwd vsdfhwkdw duh pruh2ohvv ghqvho| srsxodwhg wkdq zrxog eh wkh fdvh li wkh gdwd yduldeohv zhuh lqghshqghqwri hdfk rwkhu1 Lq wklv fdvh s � +{, lv wdnhq wr eh wkh ghqvlw| forvhvw wr wkdw ri wkh gdwd s+{, xqghu wkhfrqvwudlqw ri lqghshqghqfh

s � +{, @�\

� � �s � +{ � ,> +715,

7

Page 5: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

zkhuh hdfk idfwru lq wkh surgxfw lv wkh pdujlqdo glvwulexwlrq ri wkh uhvshfwlyh gdwd yduldeohs � +{ � , @

]

s+{, \��� �� �

g{��� = +716,

D udqgrp vdpsoh iurp wklv lqghshqghqw ghqvlw| +715, +716, lv hdvlo| jhqhudwhg iurp wkh gdwd lwvhoi1Hdfk {� 0ydoxh iru hdfk +�Prqwh Fduor�, revhuydwlrq lv udqgrpo| vhohfwhg zlwk htxdo suredelolw| iurpwkh froohfwlrq ri doo {� gdwd ydoxhv1

Dv wkh deryh glvfxvvlrq looxvwudwhv pdq| gdwd dqdo|vlv sureohpv fdq eh fdvw lq dq rswlpl}dwlrqiudphzrun zkhuh wkh remhfwlyh ixqfwlrq lv revhuyhg zlwk vxshulpsrvhg qrlvh1 Surfhgxuhv wkdw orfdwhuhjlrqv ri wkh lqsxw vsdfh zkhuh wklv wdujhw dvvxphv uhodwlyho| odujh ydoxhv uhsuhvhqw +lq sulqflsdo,srwhqwldo vroxwlrqv wr wkhvh sureohpv1

81 Lqwhusuhwdelolw|Lq ghvfulswlyh gdwd dqdo|vlv lqwhusuhwdelolw| ehfrphv dq lpsruwdqw lvvxh1 Rqh zrxog olnh wr uhvwulfwvroxwlrqv wr wkrvh wkdw fdq eh ghvfulehg dqg lqwhusuhwhg lq whupv ri lpsruwdqw fkdudfwhulvwlfv ri wkhsureohp/ hyhq li wklv pd| vdful�fh vrph srzhu1 Iru wkh sureohp frqvlghuhg khuh/ wklv lpsolhv wkdw wkhvroxwlrq uhjlrq U +516, eh vshfl�hg e| vlpsoh vwdwhphqwv +orjlfdo frqglwlrqv, lqyroylqj wkh ydoxhv riwkh lqglylgxdo lqsxw yduldeohv i{� j � � 1 Vxfk �uxohv� wdnh wkh irup

U @�^

� � �E � = +814,

Wkdw lv/ wkh vroxwlrq uhjlrq lv wdnhq wr eh wkh xqlrq ri d vhw ri vlpso| gh�qhg vxeuhjlrqv iE � j �� 1 Ohw

v � � uhsuhvhqw d vxevhw ri wkh srvvleoh ydoxhv ri lqsxw yduldeoh {� > wkdw lv iv � � � V � j � � zkhuh hdfk V �+514, uhsuhvhqwv doo srvvleoh {� 0ydoxhv1 Wkhq hdfk E � +814, lv wdnhq wr eh d �er{�

E � @ v � � � v � � � � � � � v � � +815,zlwklq wkh hqwluh lqsxw vsdfh +515,1 Hdfk er{ +815, lv ghvfulehg e| wkh lqwhuvhfwlrq ri vxevhwv ri ydoxhvri hdfk lqsxw

{ 5 E � @�_

� � �+{� 5 v � � ,= +816,

Iru lqsxwv wkdw duh uhdo +shukdsv glvfuhwho|, ydoxhg/ wkh vxevhwv duh uhsuhvhqwhg e| frqwljxrxvvxelqwhuydov

v � � @ ^w �� � > w �� � `= +817,Wkxv/ wkh surmhfwlrq ri d er{ E � rq wkh vxevsdfh ri uhdo ydoxhg lqsxwv lv d k|shu0uhfwdqjoh1 Irufdwhjrulfdo lqsxwv wkh lqglylgxdo vxevhwv ri ydoxhv v � � duh h{solflwo| gholqhdwhg1 Qrwh wkdw iru wkh fdvhlq zklfk wkh vxevhw ri ydoxhv +uhdo ru fdwhjrulfdo, lv lq idfw wkh hqwluh vhw v � � @ V � / wkh fruuhvsrqglqjidfwru {� 5 V � fdq eh rplwwhg iurp wkh er{ gh�qlwlrq +816,1 Lq wklv fdvh lw wdnhv wkh vlpsohu irup

{ 5 E � @ _

����� �� �+{� 5 v � � ,= +818,

Wkh sduwlfxodu lqsxw yduldeohv { � iru zklfk v � � 9@ V � duh vdlg wr eh wkrvh wkdw �gh�qh� wkh er{ E � 1Dv dq h{dpsoh/ iru wkh pdunhwlqj gdwd edvh ghvfulehg lq Vhfwlrq 48 wkh uxoh

{ 5 E � @;

?

=

4; ? djh ? 67 )pdulwdo vwdwxv 5 ivlqjoh/ olylqj wrjhwkhu0qrw pduulhgj )krxvhkroghu vwdwxv @ uhqw

gh�qhv rqh ri wkh er{hv frpsulvlqj wkh vxeuhjlrq dvvrfldwhg zlwk kljk iuhtxhqf| ri ylvlwlqj eduv dqgqljkw foxev1

8

Page 6: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

91 FryhulqjIurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv+818, iurp wkh gdwd +414, wkdw froohfwlyho| fryhu wkh uhjlrq ri wkh lqsxw vsdfh zkhuh wkh wdujhw i+{,dvvxphv odujh ydoxhv1 Jlyhq dq dojrulwkp iru frqvwuxfwlqj er{hv iurp gdwd +vhh Vhfwlrq :, wklv fdq ehdffrpsolvkhg e| �fryhulqj� ^vhh Plwfkhoo+4<<:,`1 Wkh vdph er{ frqvwuxfwlrq dojrulwkp lv uhshdwhgo|dssolhg lq d vhtxhqwldo pdqqhu wr vxevhwv ri wkh gdwd1 Wkh �uvw er{ E � lv lqgxfhg xvlqj wkh hqwluhgdwd vhw +414,1 Wkh vhfrqg E � lv frqvwuxfwhg xvlqj wkh ruljlqdo gdwd zlwk wkrvh revhuydwlrqv fryhuhge| wkh �uvw er{ uhpryhg= E � � i| � >{ � m{ � @5 E � j1 Dw wkh Nwk lwhudwlrq d er{ E � lv lqgxfhg xvlqj wkhgdwd uhpdlqlqj diwhu wkh uhprydo ri doo revhuydwlrqv fryhuhg e| wkh N � 4 suhylrxvo| lqgxfhg er{hv=E � � i| � >{ � m{ � @5 V

�� �

� � � E � j1 Wklv frqwlqxhv xqwlo hlwkhu wkh hvwlpdwhg wdujhw phdqv zlwklq wkher{hv

�| � @ dyh^| � m{ � 5 E � ) { � @5�� �̂

� � �E � ` +914,

ehfrph wrr vpdoo/ vd| ohvv wkdq wkh joredo phdq �| +41;,/ ru wkhlu lqglylgxdo vxssruw

� � @ 4Q

�[��� �

4+{ � 5 E � ) { � @5�� �̂

� � �E � , +915,

ehfrphv wrr vpdoo1Wkh vhw ri er{hv lqgxfhg lq wklv pdqqhu fdq wkhq eh xvhg wr irup wkh �qdo uhjlrq +814, dffruglqj

wr wkh vshfl�f gdwd dqdo|wlf jrdo1 Rqh pljkw vhohfw wkrvh zkrvh phdq +914, lv juhdwhu wkdq vrphwkuhvkrog �| �

U @ ^

�� ��� ����E � >

ru shukdsv fkrrvh wkh vxevhw wkdw |lhogv wkh odujhvw uhjlrq phdq �| � +518, iru d vshfl�hg vxssruw � �

� � @�[

� � �� � * � � =

Khuh i� � j duh wkh lqglylgxdo er{ vxssruwv +915,1 Dowhuqdwlyho|/ rqh fdq uhjdug wkh vhtxhqfh ri lqgxfhger{hv/ dorqj zlwk wkhlu uhvshfwlyh phdqv/ dv dq rughuhg vhw +�ghflvlrq olvw�, ^Ulyhvw +4<;:,`

i�| � > E � j �� = +916,

Dq lqsxw srlqw { wkdw lv fryhuhg e| pruh wkdq rqh er{ lv dvvljqhg wr wkh rqh dsshdulqj �uvw lq wkholvw/ dqg dvvrfldwhg zlwk lwv er{ phdq1

:1 Er{ lqgxfwlrqFhqwudo wr wkh rswlpl}dwlrq dojrulwkp lv wkh surfhgxuh xvhg wr frqvwuxfw wkh lqglylgxdo er{hv1 Jlyhqwkh gdwd +ru d vxevhw wkhuhri,/ lwv jrdo lv wr surgxfh d er{ E zlwklq zklfk wkh wdujhw phdq

�i � @]

��� �i+{,s+{,g{!

]

��� �s+{,g{ +:14,

lv dv odujh dv srvvleoh frqvlvwhqw zlwk wkh lqwhusuhwdelolw| frqvwudlqw +818,1 Wkh vwudwhj| hpsor|hg lvrqh ri �sdwlhqw� wrs0grzq vxffhvvlyh uh�qhphqw iroorzhg e| erwwrp�xs uhfxuvlyh h{sdqvlrq1

9

Page 7: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

:141 Wrs�grzq shholqjWklv �uvw skdvh ri wkh er{ lqgxfwlrq vwudwhj| ehjlqv zlwk d er{ E wkdw fryhuv doo ri wkh gdwd1 Dwhdfk vwhs +lwhudwlrq, d vpdoo vxeer{ e zlwklq wkh fxuuhqw er{ E lv uhpryhg1 Wkh sduwlfxodu vxeer{e � fkrvhq iru uhprydo/ wr surgxfh wkh qh{w +vpdoohu, er{ lq wkh vhtxhqfh/ lv wkh rqh wkdw |lhogv wkhodujhvw rxwsxw phdq ydoxh zlwklq wkh qh{w er{ E � e �

e � @ duj pd{� ����� ��� dyh^| � m{ � 5 E � e`= +:15,

Khuh F+e, uhsuhvhqwv d fodvv ri srwhqwldo vxeer{hv holjleoh iru uhprydo1 Wkh fxuuhqw er{ lv wkhq xsgdwhgE # E � e � +:16,

dqg wkh surfhgxuh uhshdwhg rq wklv qhzhu vpdoohu er{1 Wklv �shholqj� dzd| ri vpdoo vxeer{hv frqwlqxhvxqwlo wkh vxssruw zlwklq wkh fxuuhqw er{ � � idoov ehorz vrph wkuhvkrog ydoxh � �

� � @ 4Q

�[��� �

4+{ � 5 E, � � � = +:17,

Wkh txdqwlw| � � lv d �phwd�0sdudphwhu ri wkh surfhgxuh1 D fkrlfh iru lwv ydoxh lqyroyhv erwk vwdwlvwlfdodqg sureohp grpdlq ghshqghqw frqvlghudwlrqv1 Wkhvh duh glvfxvvhg lq Vhfwlrq <1

Wkh fodvv ri er{hv F+e, holjleoh iru shholqj +:15, lv glfwdwhg e| wkh lqwhusuhwdelolw| frqvwudlqw +818,1Hdfk holjleoh vxeer{ e lv gh�qhg e| d vlqjoh lqsxw yduldeoh {� 1

+d, Uhdo ydoxhg lqsxwv hdfk surylgh wzr holjleoh vxeer{hv/ e � � dqg e � � / zklfk erughu wkh uhvshfwlyhorzhu dqg xsshu erxqgdulhv ri wkh fxuuhqw er{ E rq wkh mwk +uhdo, lqsxw

e � � @ i{ m{� ? {� � � � j +:18,e � � @ i{ m{� A {� � � � � � j=

Khuh {� � � � lv wkh �0txdqwloh ri wkh {� 0ydoxhv iru gdwd zlwklq wkh fxuuhqw er{/ dqg { � � � � � � lv wkhfruuhvsrqglqj +4� �,0txdqwloh1 Wkh txdqwlw| � lv dqrwkhu phwd0sdudphwhu xvxdoo| wdnhq wr eh txlwhvpdoo +� � 3=4,1 Sduwlfxodu ydoxhv duh glfwdwhg e| vwdwlvwlfdo frqvlghudwlrqv ghwdlohg lq Vhfwlrq ;1

+e, Hdfk fdwhjrulfdo yduldeoh {� frqwulexwhv d vhw ri holjleoh er{hv/ rqh iru hdfk ri lwv ydoxhv v �zlwklq wkh fxuuhqw er{

e � @ i{ m{ � @ v � j> v� 5 V � = +:19,Wkh frpsohwh fodvv F+e, ri holjleoh vxeer{hv +:15, lv wkh froohfwlrq ri doo ri wkrvh gh�qhg rq wkh

uhvshfwlyh lqsxw yduldeohv1 Wkh gdwd zlwklq hdfk rqh lv +lq wxuq, surylvlrqdoo| uhpryhg dqg wkh phdqryhu wkh uhpdlqlqj revhuydwlrqv lq wkh fxuuhqw er{ fdofxodwhg1 Wkh gdwd zlwklq wkh vxeer{ e � wkdwjlyhv ulvh wr wkh odujhvw vxfk +uhpdlqlqj, phdq lv wkhq shupdqhqwo| uhpryhg wr gh�qh wkh qh{w vpdoohuer{ +:16,1

Wkh xvh ri ydulrxv w|shv ri shholqj surfhgxuhv lq wkh frqwh{w ri pxowlyduldwh vwdwlvwlfdo dqdo|vlvzdv lqwurgxfhg e| Eduqhww +4<:9, dqg hpsor|hg e| Juhhq +4<;4,1 Grqrkr dqg Jdvnr +4<<5, xvhgwklv frqfhsw wr gh�qh urexvw d!qh htxlyduldqw hvwlpdwruv ri pxowlyduldwh orfdwlrq1:151 Erwwrp�xs sdvwlqjWkh jrdo ri wkh wrs�grzq shholqj dojrulwkp lv d er{ fryhulqj d vxeuhjlrq ri wkh lqsxw yduldeoh vsdfhzlwklq zklfk wkh wdujhw phdq +:14, lv uhodwlyho| odujh1 Wkh erxqgdulhv +818, ri wklv er{ duh ghwhuplqhge| sduwlfxodu ydoxhv ri wkrvh yduldeohv wkdw gh�qhg wkh vxeer{hv +:18, +:19, fkrvhq iru shholqj +:15,dw wkh ydulrxv vwdjhv ri wkh wrs�grzq uh�qhphqw surfhgxuh1 H{fhsw iru wkh odvw rqh/ wkhvh �qdo er{erxqgdulhv zhuh ghwhuplqhg dw hduolhu vwhsv ri wkh shholqj vhtxhqfh zlwkrxw nqrzohgjh ri odwhu shhovwkdw ixuwkhu uh�qhg wkh erxqgdulhv rq rwkhu lqsxw yduldeohv1 Lw lv wkhuhiruh srvvleoh wkdw wkh �qdo er{fdq eh lpsuryhg e| uhdgmxvwlqj vrph ri lwv erxqgdulhv1 Wklv lv grqh wkurxjk d erwwrp�xs �sdvwlqj�vwudwhj|1

:

Page 8: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

Wklv sdvwlqj dojrulwkp lv edvlfdoo| wkh lqyhuvh ri wkh shholqj surfhgxuh1 Vwduwlqj zlwk wkh shholqjvroxwlrq/ wkh fxuuhqw er{ E lv lwhudwlyho| hqodujhg e| sdvwlqj rqwr lw d vpdoo vxeer{ e � / E # EV e � 1Wkh vpdoo vxeer{ e � fkrvhq lv wkh rqh iurp dq holjleoh fodvv e 5 F+e, wkdw pd{lpl}hv wkh rxwsxw phdqlq wkh qhz +odujhu, er{1 Wkh holjleoh fodvv ri sdvwlqj vxeer{hv lv gh�qhg dqdorjrxvo| wr wkrvh xvhgiru shholqj +:18, +:19,1 Iru hdfk lqsxw yduldeoh { � / holjleoh vxeer{hv e duh erxqghg e| wkh fxuuhqwer{ erxqgdulhv rq doo rwkhu yduldeohv i{ � � j � � �� � 1 Iru fdwhjrulfdo yduldeohv { � / ydoxhv v� +:19, qrwuhsuhvhqwhg lq wkh fxuuhqw er{ E gh�qh vxeer{hv e holjleoh iru sdvwlqj1 Iru uhdo ydoxhg { � / holjleohvxeer{hv duh uhsuhvhqwhg e| vpdoo lqwhuydov h{whqglqj wkh xsshu dqg orzhu erxqgdulhv ri E rq wkdwyduldeoh1 Wkh zlgwkv ri hdfk ri wkhvh lqwhuydov lv fkrvhq vr dv wr frqwdlq �Q � revhuydwlrqv/ zkhuh �lv wkh shholqj iudfwlrq +:18, dqg Q � lv wkh qxpehu ri revhuydwlrqv lq wkh fxuuhqw er{ E1

Erwwrp0xs sdvwlqj lv lwhudwlyho| dssolhg/ vxffhvvlyho| hqodujlqj wkh fxuuhqw er{/ xqwlo wkh dgglwlrqri wkh qh{w vxeer{ e � fdxvhv wkh rxwsxw phdq �| � � ��� wr ghfuhdvh1 Dw wkdw srlqw sdvwlqj vwrsv dqg wkhfxuuhqw er{ E ehfrphv wkh vroxwlrq1 Dowkrxjk erwwrp�xs sdvwlqj xvxdoo| surylghv vrph lpsuryh0phqw/ lw vhogrp kdv d gudpdwlf h�hfw1 Wkhuh duh rffdvlrqv krzhyhu zkhq lw grhv surgxfh vxevwdqwldolpsuryhphqw wr wkh shholqj vroxwlrq1

;1 Sdwlhqw uxoh lqgxfwlrqWkhuh duh wzr sulqflsdo phwd0sdudphwhuv wkdw frqwuro er{ lqgxfwlrq> wkh| duh wkh shholqj iudfwlrq �+:18, dqg wkh vxssruw � � +:17, ri wkh shholqj vroxwlrq1 Vwdwlvwlfdo shuirupdqfh frqvlghudwlrqv jryhuqwkh fkrlfh ri d ydoxh iru �1 Wkhvh duh glvfxvvhg lq wklv vhfwlrq1 Fkrrvlqj d ydoxh iru � � lqyroyhv erwkvwdwlvwlfdo dqg dssolfdwlrq grpdlq ghshqghqw frqvlghudwlrqv1 Wkhvh duh glvfxvvhg lq Vhfwlrq <1

Lq rughu wr glvfxvv vwdwlvwlfdo shuirupdqfh d irupdo +lghdol}hg, jrdo pxvw eh vshfl�hg dv zhoo dv dphdvxuh ri krz zhoo wkdw jrdo lv dfklhyhg1 Iru d jlyhq ydoxh ri � � wkh irupdo jrdo ri er{ lqgxfwlrqfdq eh h{suhvvhg dv pd{lpl}lqj wkh phdq ri wkh wdujhw ixqfwlrq i+{, zlwklq d er{ +:14, zlwk uhvshfwwr wkh er{ sdudphwhuv v � � +816,/ xqghu d frqvwudlqw rq er{ vxssruw/ � @ � � 1 Wkdw lv/

�i � @ pd{�

dyh^i+{, m{ 5 E`> � � @ � � = +;14,

Ohw ai eh wkh vroxwlrq +hvwlpdwh, lqgxfhg e| wkh shholqj dojrulwkp zkhq dssolhg wr gdwd1 Wkhq rqhphdvxuh ri shuirupdqfh lv dyhudjh +h{shfwhg, phdq0vtxduhg huuru

H^ �i � � ai ` � @ + �i � �H ai, � .H^ ai �H ai ` � = +;15,Wkh h{shfwhg ydoxh +;15, uhsuhvhqwv wkh vtxduhg0huuru ri ai dyhudjhg ryhu doo gdwd vhwv ri vl}h Q wkdwfrxog kdyh ehhq uhdol}hg iurp wkh v|vwhp xqghu vwxg|1 Wkh dfwxdo gdwd dw kdqg +414, lv suhvxphg wreh rqh ri wkhvh gdwd vhwv gudzq dw udqgrp1 Wkh txdqwlw| H ai lv wkh dyhudjh ryhu doo gdwd vhwv ri wkhhvwlpdwhg vroxwlrq iru hdfk rqh1 Wkh �uvw whup rq wkh uljkw ri +;15, lv wkh ghyldwlrq ri wklv dyhudjhiurp wkh wuxwk dqg uhsuhvhqwv wkh vtxduhg v|vwhpdwlf huuru +�eldv0vtxduhg�,1 Wkh vhfrqg uljkw kdqgwhup lv wkh yduldqfh ri wkh vroxwlrqv ryhu doo gdwd vhwv1 Lw fkdudfwhul}hv wkh lqvwdelolw| ri wkh surfhgxuh1Kljk lqvwdelolw| lpsolhv wkdw wkh vroxwlrq lv yhu| vhqvlwlyh wr vpdoo fkdqjhv lq wkh gdwd1 Wklv qrwrqo| uhgxfhv dffxudf|/ exw lw lv hvshfldoo| wurxeohvrph iurp wkh shuvshfwlyh ri lqwhusuhwdwlrq1 Lw lvgdqjhurxv wr sodfh kljk frq�ghqfh rq lqgxfhg ihdwxuhv +l1h1 er{ erxqgdulhv, li voljkw fkdqjhv wr wkhgdwd vwurqjo| d�hfw wkh qdwxuh dqg2ru h{lvwhqfh ri wkrvh ihdwxuhv1;141 IudjphqwdwlrqWrs0grzq vxffhvvlyh uh�qhphqw surfhgxuhv vxfk dv shholqj fdq eh ylhzhg dv vwhhshvw dvfhqw zlwk dolplwhg qxpehu ri vwhsv1 Hdfk lwhudwlrq surgxfhv wkh vwhs wkdw lv hvwlpdwhg wr surylgh wkh juhdwhvworfdo lqfuhdvh lq wkh remhfwlyh ixqfwlrq/ khuh er{ phdq1 Wklv �juhhg|� vwhs lv vhogrp rswlpdo lq whupvri ehlqj wkh rqh wkdw pryhv forvhvw wr wkh xowlpdwh vroxwlrq1 Li d yhu| odujh qxpehu ri vwhsv lv srvvleohwkhq wklv odfn ri rswlpdolw| uhsuhvhqwv rqo| d frpsxwdwlrqdo lvvxh> d odujh hqrxjk qxpehu ri juhhg|vwhsv zloo hyhqwxdoo| duulyh dw d pd{lpxp1

;

Page 9: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

Lq wkh fdvh ri wrs0grzq vxffhvvlyh uh�qhphqw/ d odujh qxpehu ri vwhsv lv qrw srvvleoh1 Wklv lv gxhwr gdwd iudjphqwdwlrq1 Hdfk vwhs uhgxfhv wkh dprxqw ri gdwd dydlodeoh wr wkh qh{w vwhs1 Dw vrphsrlqw wkhuh lv qrw hqrxjk uhpdlqlqj gdwd wr frqwlqxh1 Wkh er{ vxssruw � kdv idoohq ehorz wkh vwrsslqjwkuhvkrog � � 1

Iru d jlyhq gdwd vhw vl}h Q wkh qxpehu ri vwhsv lv ghwhuplqhg e| wkh dprxqw ri iudjphqwdwlrq+�juhhg�, shuplwwhg dw hdfk rqh1 Zlwk d juhhg| vwudwhj| d odujh iudfwlrq ri wkh gdwd lv uhpryhg dw hdfkvwhs doorzlqj iru rqo| d vpdoo qxpehu ri lwhudwlrqv1 Wklv uhvwulfwv wkh delolw| ri wkh surfhgxuh lq odwhuvwhsv wr frpshqvdwh iru hduo| vwhsv wkdw pd| kdyh ehhq kljko| vxerswlpdo1 Wklv fdq kdyh ghwulphqwdoh�hfw rq erwk eldv dqg yduldqfh +;15, ohdglqj wr lqfuhdvhg phdq vtxduhg huuru1 Eldv lv lqgxfhg zkhqwkh ehvw vwhs dw dq hduo| lwhudwlrq lv xqdpeljxrxv exw/ gxh wr wkh qdwxuh ri wkh wdujhw ixqfwlrq/ lvplvgluhfwhg1 Yduldqfh lv lqgxfhg zkhq wkh ehvw vwhs lv dpeljxrxv> wkhuh duh vhyhudo srvvlelolwlhv wkdwsurylgh vlplodu orfdo lpsuryhphqw lq wdujhw phdq1 Lq wklv fdvh wkh fkrlfh dprqj wkhp lv gulyhq odujho|e| wkh qrlvh/ pdnlqj lw kljko| vhqvlwlyh wr vpdoo fkdqjhv lq wkh gdwd1 Zlwk d juhhg| vwudwhj| wkhuhduh rqo| d ihz vwhsv vr hdfk rqh kdv d odujh h�hfw rq wkh �qdo rxwfrph1 Wkhuh lv olwwoh rssruwxqlw| wrplwljdwh wkh gdpdjh fdxvhg e| d edg vwhs1

Lq wkh frqwh{w ri er{ lqgxfwlrq yhu| juhhg| vwudwhjlhv srvh dqrwkhu srwhqwldo sureohp1 Rqo| dvpdoo qxpehu ri lqsxw yduldeohv fdq eh lqyroyhg lq gh�qlqj wkh er{ erxqgdulhv +818,1 Lq rughu irudq lqsxw wr sduwlflsdwh lw pxvw eh fkrvhq iru shholqj dw ohdvw rqfh lq wkh wrs0grzq vhtxhqfh1 Li wkhqxpehu ri lqsxwv q lv odujh dqg wkh qxpehu ri vwhsv vpdoo/ rqo| d vpdoo iudfwlrq ri wkh lqsxw yduldeohvfdq eh lqyroyhg1 Zkloh wklv pd| uhsuhvhqw dq lqwhusuhwdwlrqdo dgydqwdjh/ lw fdq lqwurgxfh vhyhuh eldvdqg yduldqfh zkhq pruh wkdq d ihz lqsxwv duh uhtxluhg wr lvrodwh wkh wdujhw pd{lpxp1;151 SdwlhqfhWkh sureohp ri iudjphqwdwlrq zlwk wrs0grzq uh�qhphqw surfhgxuhv fdq eh plwljdwhg e| dgrswlqj d�sdwlhqw� vwudwhj|> rqo| d vpdoo iudfwlrq ri wkh gdwd lq wkh fxuuhqw er{ lv uhpryhg dw hdfk lwhudwlrq1Wklv pdnhv hdfk lqglylgxdo vwhs ohvv lpsruwdqw wr wkh �qdo rxwfrph +yduldqfh, dqg shuplwv juhdwhurssruwxqlw| iru odwhu vwhsv wr wdnh dgydqwdjh ri vwuxfwxuh xqfryhuhg e| hduolhu rqhv dqg2ru wr frp0shqvdwh iru wkrvh wkdw zhuh plvgluhfwhg +eldv,1 Dovr/ pruh lqsxw yduldeohv duh shuplwwhg +exw qrwuhtxluhg, wr hqwhu lqwr wkh gh�qlwlrq ri wkh lqgxfhg er{1

Iru uhdo ydoxhg lqsxw yduldeohv/ wkh ghjuhh ri sdwlhqfh ri wkh er{ lqgxfwlrq dojrulwkp +Vhfwlrq :14,lv frqwuroohg e| wkh ydoxh ri wkh shholqj sdudphwhu � +:18,1 Li doo lqsxwv +fkrvhq iru shholqj, duh uhdozlwk glvwlqfw ydoxhv wkh qxpehu ri vwhsv +shhov, O lv

O @ orj� � ! orj+4� �,=Wkh deryh glvfxvvlrq vxjjhvwv fkrrvlqj wkh vpdoohvw ydoxh srvvleoh iru � lq rughu wr dfklhyh wkhpd{lpxp ghjuhh ri sdwlhqfh1 Wklv zrxog eh � @ 4@Q � zkhuh Q � lv wkh qxpehu ri revhuydwlrqv lqwkh fxuuhqw er{> rqh revhuydwlrq lv shhohg r� dw hdfk vwhs1

Krzhyhu wkh sdudphwhu � vhuyhv dqrwkhu uroh> lw lv dovr d vprrwklqj sdudphwhu1 Wkh jrdo ri hdfkshho lv wr pd{lpl}h wkh dyhudjh ri wkh wdujhw ixqfwlrq i+{, lq wkh qh{w vpdoohu er{ E�e1 Wkh yduldqfhri wkh hvwlpdwh ri wklv lv sursruwlrqdo wr

4�ydu^% m{ 5 e` +;16,

zkhuh % lv wkh udqgrp qrlvh +416,1 Vlqfh wkh jrdo lv wr kdyh wkh shholqj gulyhq e| vljqdo i+{, udwkhuwkdq wkh qrlvh %/ wkh ydoxh ri � fdqqrw eh pdgh wrr vpdoo1 Wkhuh lv d wudgh�r� ehwzhhq wkh jrdov risdwlhqfh dqg dffxudwh shholqj wkdw ghshqgv rq wkh vljqdo wr qrlvh udwlr dqg wrwdo vdpsoh vl}h1 Hpslulfdohylghqfh vr idu vxjjhvwv wkdw wkh shholqj surfhgxuh lv qrw vhqvlwlyh wr wkh ydoxh ri � surylghg lw lv qrwwrr odujh> sdwlhqfh dsshduv wr eh wkh pruh lpsruwdqw jrdo1 Ydoxhv lq wkh udqjh 3=38 � � � 3=4 vhhpwr zrun zhoo1 Phwkrgv iru hvwlpdwlqj dq rswlpdo ydoxh iurp wkh gdwd dw kdqg duh glvfxvvhg lq Vhfwlrq461

Iru lqsxw yduldeohv wkdw duh fdwhjrulfdoo| ydoxhg/ ru uhdo zlwk d vpdoo qxpehu ri glvwlqfw ydoxhv/ wkhghjuhh ri sdwlhqfh fdqqrw eh frqwuroohg zlwk suhflvlrq1 Doo revhuydwlrqv zlwk lghqwlfdo ydoxhv rq dq

<

Page 10: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

lqsxw yduldeoh duh frqvlghuhg wrjhwkhu1 Sdwlhqfh lv hqfrxudjhg e| pdnlqj rqo| d vlqjoh ydoxh holjleohiru shholqj dw hdfk vwhs1 Wkxv/ shholqj rq yduldeohv zlwk pruh +glvfuhwh, ydoxhv lqgxfhv pruh sdwlhqfh1Phwkrgv iru hqfrxudjlqj sdwlhqfh lq wklv frqwh{w duh glvfxvvhg lq Vhfwlrq 471 Rzlqj wr lwv hpskdvlv rqsdwlhqfh/ wkh surfhgxuh ghvfulehg khuhlq lv uhihuuhg wr dv d sdwlhqw uxoh lqgxfwlrq phwkrg +�SULP�,1

<1 Vwrsslqj uxohShholqj vwrsv zkhq wkh vxssruw � � ri wkh fxuuhqw er{ E lv ehorz d fkrvhq wkuhvkrog � � +:17,1 Fkrlfhri d ydoxh iru � � ghshqgv rq wkh gdwd dqdo|wlf jrdo1 Vrphwlphv wkh jrdo lv d �srlqw� hvwlpdwh ri wkhorfdwlrq ri wkh ixqfwlrq pd{lpxp1 Wkh lqgxvwuldo surfhvv frqwuro sureohpv glvfxvvhg lq Vhfwlrq 6duh h{dpsohv1 Lq wklv fdvh rqh zrxog olnh wkh ydoxh ri � � wr eh dv vpdoo dv srvvleoh frqvlvwhqw zlwkdffxudwh hvwlpdwlrq ri d pd{lpxp ri i+{,1 Ehfdxvh wkh ixqfwlrq lv phdvxuhg zlwk qrlvh +416, lw fdqeh frxqwhu surgxfwlyh wr doorz wkh ydoxh ri � � wr ehfrph duelwudulo| vpdoo1 Wkh shholqj surfhgxuhxvhv wkh rxwsxw phdq �| � +:15, dv dq hvwlpdwh ri wkh ixqfwlrq phdq �i � +:14, lq hdfk vxffhvvlyh er{1Wklv hvwlpdwh ehfrphv ohvv uholdeoh zlwk vpdoohu ydoxhv ri er{ vxssruw � � dqg wkh shholqj surfhgxuhfdq eh glvwudfwhg e| qrlvh> wkh ydoxh ri �| � fdq frqwlqxh wr lqfuhdvh zkhuhdv wkh ydoxh ri �i � dfwxdoo|ghfuhdvhv1 Wklv �ryhu0�wwlqj� skhqrphqrq lv frpprq wr doo surfhgxuhv wkdw rswlpl}h rq udqgrp gdwd1

<141 Furvv�ydolgdwlrqD frpprq surfhgxuh xvhg wr ghdo zlwk ryhu0�wwlqj lv furvv0ydolgdwlrq1 Wkh gdwd vhw lv udqgrpo|sduwlwlrqhg lqwr wzr sduwv/ d �ohduqlqj� gdwd vhw dqg d �whvw� gdwd vhw1 W|slfdoo| wkh ohduqlqj vhw lvwdnhq wr eh wzlfh wkh vl}h ri wkh whvw vhw1 Wkh shholqj surfhgxuh lv dssolhg wr wkh ohduqlqj gdwd zlwkd yhu| vpdoo ydoxh iru � � / doorzlqj iru yhu| vpdoo er{ vxssruwv1 Wkh whvw gdwd vhw lv wkhq xvhg wrhvwlpdwh wkh rxwsxw phdq �| � lq hdfk vxffhvvlyh er{ ri wkh shholqj vhtxhqfh lqgxfhg rq wkh wudlqlqjgdwd1 Li wkh qrlvh dvvrfldwhg zlwk wkh whvw gdwd lv lqghshqghqw ri wkdw iru wkh wudlqlqj gdwd wklv zloosurgxfh xqeldvhg hvwlpdwhv ri wkh wdujhw phdq �i � lq hdfk er{1 Wkh er{ zlwk wkh odujhvw dvvrfldwhg+whvw, phdq lv wdnhq dv wkh hvwlpdwhg rswlpdo rqh frqwdlqlqj wkh wdujhw pd{lpxp1

<151 Dssolfdwlrq grpdlqRiwhq/ dssolfdwlrq grpdlq frqvlghudwlrqv sodfh frqvwudlqwv rq wkh vxssruw ri wkh lqgxfhg er{1 Wkhvxssruw ri wkh furvv0ydolgdwhg hvwlpdwh ri wkh rswlpdo er{ pd| eh wrr vpdoo wr eh xvhixo1 Lw pd|lghqwli| fkdudfwhulvwlfv ri �qdqfldo vhfxulwlhv zlwk yhu| kljk uhwxuq/ exw wkh vxssruw pd| eh wrr vpdoowr doorz vx!flhqwo| iuhtxhqw ehwwlqj1 D kljko| ghvludeoh pdunhw vhjphqw pd| eh orfdwhg exw eh wrrvpdoo wr eh zruwk wdujhwlqj1 Lq vrph dssolfdwlrqv wkhuh pd| eh d sur�wdelolw| wkuhvkrog dqg rqhvhhnv er{hv zlwk odujhvw vxssruw zkrvh phdq lv deryh wkdw wkuhvkrog1 Lq doo ri wkhvh fdvhv wkhuh pd|eh er{hv wkdw kdyh pxfk odujhu vxssruw wkdq wkh hvwlpdwhg rswlpdo rqh/ exw rqo| voljkwo| vpdoohuhvwlpdwhg wdujhw phdq1 Wkhvh �vxerswlpdo� er{hv pd| eh pruh xvhixo lq sduwlfxodu dssolfdwlrqv1

Dssolfdwlrq grpdlq mxgjhphqwv vxfk dv wkh wudgh�r� ehwzhhq er{ phdq dqg vxssruw duh ehvwpdgh e| wkh xvhu1 Wkh xvhu fdq eh suhvhqwhg zlwk wkh phdq dqg vxssruw ri hdfk ri wkh O er{hv lqwkh +qhvwhg, shholqj vhtxhqfh iE � j � � /

i�| � > � � j � � > +<14,dv frpsxwhg rq wkh +ohiw rxw, whvw gdwd vhw1 Wkh rqh prvw dssursuldwh iru wkh dssolfdwlrq dw kdqgfdq wkhq eh fkrvhq1 Wklv pd| eh wkh �rswlpdo� rqh zlwk odujhvw hvwlpdwhg phdq ru rqh wkdw kdv dvpdoohu phdq exw odujhu vxssruw1 Diwhu wklv fkrlfh lv pdgh/ erwwrp�xs sdvwlqj +Vhfwlrq :15, fdq wkhqeh dssolhg wr ixuwkhu lpsuryh wkh phdq�vxssruw wudgh�r�1

Iljxuh <14 vkrzv d vfdwwhu sorw ri �| � djdlqvw � � +<14, iru wkh �uvw vhtxhqfh ri er{hv lqgxfhg iurpwkh jhrorj| gdwd ghvfulehg lq Vhfwlrq 441 Vxfk d sorw zloo eh uhihuuhg wr dv d �shholqj wudmhfwru|�1Wkh er{ phdq �| � lv vhhq wr vwhdglo| lqfuhdvh zlwk ghfuhdvlqj vxssruw � � zlwk wkh h{fhswlrq ri wkuhhvpdoo expsv1 Wkh rswlpdo vroxwlrq lv �| ��� @ 3=<: dw � ��� @ 3=35;1 Li wkh vxssruw ri wklv er{ lv wrrvpdoo dqrwkhu fdqglgdwh pljkw eh �| ��� @ 3=<8 dw � ��� @ 3=3;41 Lw kdv voljkwo| orzhu hvwlpdwhg phdq exwdoprvw wkuhh wlphv wkh vxssruw1 Rwkhu fkrlfhv pd| uhsuhvhqw pruh xvhixo dowhuqdwlyhv dqg wkh xvhu

43

Page 11: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

support

box

mea

n

0.0 0.2 0.4 0.6 0.8 1.0

0.6

0.7

0.8

0.9

Iljxuh <14= Iluvw shholqj wudmhfwru| iru wkh jhrorj| gdwd1 Phdq yv1 vxssruw iru er{hv lqgxfhg e|wrs�grzq shholqj1 Prvw dssursuldwh wudgh�r� lv fkrvhq e| wkh xvhu1

lv iuhh wr fkrrvh wkh rqh prvw dssursuldwh1 Lqyroylqj wkh xvhu lq wklv pdqqhu khosv lqvxuh wkdw wkhlqgxfhg er{hv ehvw vhuyh wkh sduwlfxodu gdwd dqdo|wlf jrdo1

431 Uhgxqgdqw lqsxw yduldeohvLq dgglwlrq wr wdujhw phdq dqg vxssruw/ d er{ E lv fkdudfwhul}hg e| wkh vhw ri lqsxw yduldeohv wkdwgh�qh lw +818,1 Wkh �frpsoh{lw|� ri d er{ fdq eh gh�qhg dv wkh fduglqdolw| ri wklv vhw1 Iru d jlyhqphdq dqg vxssruw/ ohvv frpsoh{ er{hv duh suhihuuhg gxh wr wkhlu vlpsohu lqwhusuhwdwlrq1 Dv qrwhg lqVhfwlrq ;15 dq lpsruwdqw surshuw| ri sdwlhqw uxoh lqgxfwlrq lv wkdw lw doorzv pruh frpsoh{ er{hv wreh lqgxfhg1 Wklv uhsuhvhqwv dq lpsruwdqw dgydqwdjh zkhq vxfk frpsoh{lw| lv uhtxluhg/ uhgxflqj eldv1Krzhyhu/ lw kdv wkh vlgh h�hfw wkdw qrqhvvhqwldo lqsxwv fdq dovr hqwhu lqwr wkh er{ gh�qlwlrq1 Wkhvh�uhgxqgdqw� lqsxwv fdxvh lqfuhdvhg frpsoh{lw|/ wkhuhe| forxglqj lqwhusuhwdwlrq zlwkrxw uhgxflqj eldv1

Wkhuh duh wzr phfkdqlvpv wkdw fdq fdxvh uhgxqgdqw lqsxwv/ qrlvh dqg froolqhdulw|1 Wkh qrlvh %+416, lqgxfhv yduldqfh lq wkh hvwlpdwh ri wkh ehvw shholqj yduldeoh dw hdfk lwhudwlrq1 Wklv fdq fdxvhluuhohydqw yduldeohv wr eh vhohfwhg rzlqj wr vdpsolqj xfwxdwlrqv1 Dv glvfxvvhg lq Vhfwlrq ;15 wklvlv wkh prwlydwlrq iru qrw fkrrvlqj wkh shholqj sdudphwhu � wrr vpdoo +;16,1 Dowkrxjk h�hfwlyh/ wklvvwudwhj| fdqqrw shuihfwo| vfuhhq rxw luuhohydqw lqsxw yduldeohv1

Froolqhdulw| rffxuv zkhq wkhuh lv d kljk ghjuhh ri fruuhodwlrq dprqj wzr ru pruh lqsxw yduldeohvzlwklq wkh vroxwlrq er{1 Lq wklv fdvh/ rqo| rqh ri wkh yduldeohv lq wkh fruuhodwhg jurxs lv qhhghg dvsduw ri wkh er{ gh�qlwlrq/ qdpho| wkh rqh zlwk wkh prvw uhvwulfwlyh udqjh ri ydoxhv1 E| yluwxh ri lwvfruuhodwlrqv zlwk wkh rwkhuv lq wkh jurxs/ uhvwulfwlqj lwv udqjh dorqh vhuyhv wr uhvwulfw wkh udqjhv ri wkhrwkhu yduldeohv1

Zkhwkhu lqgxfhg e| qrlvh ru froolqhdulw|/ uhgxqgdqw lqsxw yduldeohv fdq eh sur�wdeo| ghohwhg iurpwkh vhw +818, gh�qlqj wkh er{1 Lq wkh fdvh ri luuhohydqw lqsxwv lqgxfhg e| qrlvh wklv fdq vhuyh wrlqfuhdvh dffxudf| lq whupv ri surylglqj kljkhu er{ phdq dqg vxssruw1 Hyhq zkhq wklv lv qrw wkh fdvhwkh lqwhusuhwdwlyh ydoxh dvvrfldwhg zlwk uhprylqj pdujlqdoo| uhohydqw lqsxwv fdq vrphwlphv rxwzhljkd vpdoo ghfuhdvh lq hvwlpdwhg er{ phdq1 Vxfk ghflvlrqv duh olnho| wr ghshqg rq sureohp vshfl�fv dqg/

44

Page 12: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

dv zlwk fkrrvlqj er{ vxssruw/ duh ehvw mxgjhg e| wkh xvhu1 Grpdlq nqrzohgjh fdq vrphwlphv khos lqghflglqj zkhwkhu sduwlfxodu lqsxw yduldeohv wkdw dsshdu pdujlqdo rq vwdwlvwlfdo jurxqgv dfwxdoo| duholnho| wr eh uhohydqw ru qrw1

Lq rughu wr pdnh vxfk mxgjhphqwv d vwdwlvwlfdo phdvxuh lv uhtxluhg ri hdfk lqsxw yduldeoh*v uhohydqfhlq gh�qlqj wkh er{1 Rqh vxfk phdvxuh lv wkh ghfuhdvh lq er{ phdq zkhq wkh yduldeoh lv uhpryhg1Ohw {� eh rqh ri wkh yduldeohv lqyroyhg lq wkh er{ gh�qlwlrq +818,1 Lw fdq eh uhpryhg e| uhsodflqj lwvfruuhvsrqglqj vxevhw ri ydoxhv v � � e| wkh vhw ri doo srvvleoh {� 0ydoxhv V � /

v � � # V � = +4314,Wkh fruuhvsrqglqj ghfuhdvh lq er{ phdq lv wkhq uhfrughg1 Wkh gh�qlqj lqsxw zlwk wkh vpdoohvw vxfkghfuhdvh lv ghhphg wr eh wkh ohdvw uhohydqw rqh dqg lv +surylvlrqdoo|, uhpryhg1 Wklv surfhgxuh lvwkhq uhshdwhg rq wkh uhpdlqlqj lqsxwv lghqwli|lqj wkh vhfrqg ohdvw uhohydqw lqsxw yduldeoh dqg lw lvsurylvlrqdoo| ghohwhg1 Wklv frqwlqxhv xqwlo wkhuh lv rqo| rqh lqsxw yduldeoh ohiw lq wkh er{ gh�qlwlrq1Wklv rqh lv uhjdughg dv wkh prvw uhohydqw er{ yduldeoh1 Wkh vhtxhqfh ri ghohwhg lqsxwv lqgxfhg lq wklvpdqqhu rughuv wkh lqsxw yduldeohv lq dvfhqglqj uhohydqfh wr er{ wkh gh�qlwlrq1

Wdeoh 4314Vhtxhqwldo yduldeoh uhohydqfh iru wkh jhrorj| gdwd1

ydu1 �| �3 3=<79 3=3:54 3=<85 3=3;49 3=<74 3=43;5 3=:88 3=6738 3=8;6 4=3

Wdeoh 4314 suhvhqwv wkh uhvxowv ri wklv edfnzdug vwhszlvh holplqdwlrq surfhgxuh rq wkh er{ +�| @3=<8> � @ 3=3:, ri wkh jhrorj| gdwd shholqj wudmhfwru| vkrzq lq Ilj1 <141 Lw lv vhhq wr eh gh�qhg e|irxu +rxw ri 44, lqsxw yduldeohv m 5 i4> 5> 8> 9j1 Wkh �uvw urz ri Wdeoh 4314 uhsuhvhqwv wklv er{ dqg wkhqh{w irxu urzv uhsuhvhqw wkh uhvxowv ri uhprylqj hdfk gh�qlqj yduldeoh +lq wxuq, lq rughu ri hvwlpdwhguhohydqfh1 Wkh �uvw froxpq odehov wkh yduldeoh uhpryhg dqg wkh qh{w wzr duh uhvshfwlyho| wkh er{phdq dqg vxssruw dv d uhvxow ri wkh uhprydo/ hvwlpdwhg iurp wkh +ohiw rxw, whvw gdwd1

Rqh vhhv iurp Wdeoh 4314 wkdw uhprylqj { � iurp wkh er{ gh�qlwlrq dfwxdoo| surylghv d voljkwlqfuhdvh lq hvwlpdwhg er{ phdq1 Uhprylqj { � dqg { � uhvxowv lq d voljkw ghfuhdvh zkhuhdv uhprylqj{ � / { � / dqg { � surgxfhv d gudpdwlf ghfuhdvh ri wdujhw phdq lqvlgh wkh er{ qrz gh�qhg rqo| e| { � 1 Rifrxuvh/ dovr uhprylqj { � surgxfhv wkh vwduwlqj er{ hqforvlqj doo ri wkh gdwd1 Qrwh wkdw er{ vxssruwdozd|v lqfuhdvhv dv yduldeohv duh uhpryhg1

Jlyhq d er{ fkrvhq iurp wkh shholqj wudmhfwru| +Ilj1 <14, wkh fruuhvsrqglqj uhvxowv iurp wkhedfnzdug vwhszlvh surfhgxuh/ dv uhsuhvhqwhg lq Wdeoh 4314/ fdq eh hpsor|hg e| wkh xvhu wr mxgjhwudgh0r�v ehwzhhq er{ frpsoh{lw|/ phdq/ dqg vxssruw1 Lq wklv zd| wkh xvhu*v grpdlq nqrzohgjh dvzhoo dv gdwd dqdo|wlf uhtxluhphqwv fdq eh wdnhq lqwr dffrxqw1 Iru wklv sduwlfxodu h{dpsoh +Wdeoh 4314,wkh fkrlfh lv udwkhu reylrxv1 Uhprylqj { � dqg { � surgxfhv d er{ zlwk rqh kdoi wkh frpsoh{lw| dqg83( pruh vxssruw wkdq wkh ruljlqdoo| lqgxfhg rqh/ zlwk rqo| voljkwo| orzhu hvwlpdwhg wdujhw phdq1 Lqpdq| rwkhu vlwxdwlrqv/ krzhyhu/ wkh fkrlfhv duh qrw vr reylrxv dqg wkh xvhu*v nqrzohgjh dqg mxgjhphqwehfrph ydoxdeoh dvvhwv1

441 Jhrorj| gdwdWkh gdwd vhw xvhg iru looxvwudwlrq lq wkh suhfhglqj wzr vhfwlrqv frqvlvwv ri d vdpsoh ri Q @ 4664:jduqhwv froohfwhg iurp durxqg wkh zruog ^Jul!q hw1 do1 +4<<:,`1 D jduqhw lv d frpsoh{ Fd0Pj0Ih0Fuvlolfdwh wkdw frpprqo| rffxuv dv d plqru skdvh lq urfnv pdnlqj xs wkh Hduwk*v pdqwoh1 Wkh lqsxwyduldeohv duh wkh frqfhqwudwlrqv ri ydulrxv fkhplfdov phdvxuhg rq hdfk jduqhw

{ @ iWlR � / Fu � R � / IhR/ PqR/ PjR/ FdR/ ]q/ Jd/ Vu/ \/ ]uj1

45

Page 13: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

Rqh gdwd dqdo|wlf remhfwlyh zdv wr frqwudvw wkh fkhplfdo frpsrvlwlrqv ri jduqhwv zlwk gl�huhqw sodwh�whfwrqlf vhwwlqjv1 Wkuhh vxfk sodwh�vhwwlqjv duh uhsuhvhqwhg lq wkh gdwd= +4, dqflhqw vwdeoh vklhogv/ +5,Surwhur}rlf vklhog duhdv/ dqg +6, |rxqj rurjhqlf ehowv1 Wkh rxwsxw yduldeoh fkrvhq iru looxvwudwlrq zdvdq lqglfdwru ri ehlqj iurp wkh �uvw vhwwlqj

| @ 4+sodwh0whfwrqlf vhwwlqj @ dqflhqw vwdeoh vklhogv,=Wkh joredo rxwsxw phdq lv �| @ 3=8<> 8<( ri wkh jduqhwv lq wkh vdpsoh kdg wklv vhwwlqj1

Wkh �uvw er{ lqgxfhg e| wkh SULP surfhgxuh xvlqj wkh fkrlfhv ghvfulehg lq wkh suhylrxv vhfwlrqvlv

{ 5 E � @ Fu � R � � 3=95 ) PjR � 3=9:= +4414,Khuh wkh olplwv duh vwdwhg lq whupv ri wkh txdqwlohv ri wkh fruuhvsrqglqj lqglylgxdo frqfhqwudwlrqv ryhuwkh vdpsoh1 Wkxv wklv er{ fryhuv jduqhw vdpsohv iru zklfk wkh frqfhqwudwlrq ri Fu � R � lv dprqj lwv6;( kljkhvw ydoxhv dqg wkdw iru PjR dvvxphv ydoxhv lq lwv wrs 66(1 Wkh rxwsxw phdq lq wklv er{ lv�| � @ 3=<7 zlwk vxssruw � � @ 3=44= Wkdw lv/ wklv er{ fryhuv 44( ri wkh gdwd dqg <7( ri wkh jduqhwvzlwklq lw zhuh dqflhqw vwdeoh vklhogv1 Wkh qh{w wzr er{hv lqgxfhg wkurxjk wkh fryhulqj surfhgxuhrxwolqhg lq Vhfwlrq 9 duh

{ 5 E � @ Jd � 3=;9 ) 3=3< � \ � 3=<5> +4415,{ 5 E � @ Fu � R � � 3=76 ) PjR � 3=9< ) FdR � 3=7:

zlwk uhvshfwlyh rxwsxw phdqv �| � @ 3=;3 dqg �| � @ 3=;61 Wkh uhvshfwlyh vxssruwv duh � � @ 3=43 dqg� � @ 3=3<1 Wkh �uvw wkuhh er{hv U @ E � VE � VE � froohfwlyho| fryhu 5<( ri wkh gdwd +� � @ 3=5<,zlwk wdujhw phdq �| � @ 3=;91 Lq wkh hqwluh vdpsoh wkh uhodwlyh rggv ri d jduqhw kdylqj wkh dqflhqwvwdeoh vklhogv vhwwlqj lv 4177 zkhuhdv lq wkh lqgxfhg uhjlrq U wkh fruuhvsrqglqj rggv duh 91471 Wkxv/wkh rggv udwlr kdv ehhq lqfuhdvhg e| ryhu d idfwru ri irxu1 Wkh wkuhh uxohv +4414, +4415, surylgh dsduvlprqlrxv dqg hdvlo| lqwhusuhwhg ghvfulswlrq ri wkh lqgxfhg uhjlrq1

Lq wkh jhrorj| gdwd edvh wkhuh duh 7<6 jduqhwv zlwk wkh wklug sodwh�whfwrqlf vhwwlqj/ frpsulvlqjrqo| 61:( ri wkh hqwluh vdpsoh1 Wkxv/ wkh rxwsxw

| @ 4+sodwh0whfwrqlf vhwwlqj @ |rxqj rurjhqlf ehowv, +4416,fdq eh ylhzhg dv d �uduh hyhqw� skhqrphqrq/ �| � @ 3=36:1 Dsso|lqj wkh SULP surfhgxuh wr +4416,surgxfhg wkh +�uvw, er{

{ 5 E � @;

?

=

Fu � R � � 3=49 ) PjR � 3=75 )FdR � 3=5< ) Jd � 3=8< )3=:6 � \ � 3=<9=

+4417,

Lwv phdq ydoxh lv �| � @ 3=;5 zlwk vxssruw � � @ 3=34:1 Wkxv/ doprvw 73( ri wkh |rxqj rurjhqlf ehowvjduqhwv duh frqwdlqhg lq wklv yhu| vpdoo er{ frpsulvlqj rqo| 41:( ri wkh hqwluh vdpsoh1 Wkh uhodwlyhrggv ri �qglqj wklv w|sh ri jduqhw lq wkh er{ duh lqfuhdvhg e| ryhu d idfwru ri 433 iurp wkdw ri wkhhqwluh gdwd edvh1 Wkh qh{w wzr lqgxfhg er{hv +qrw vkrzq, kdyh phdq dqg vxssruw ydoxhv +�| � @ 3=45/� � @ 3=37, dqg +�| � @ 3=4</ � � @ 3=358, zlwk fruuhvsrqglqj rggv udwlr lqfuhdvhv ri 6188 dqg 9144uhvshfwlyho|1 Wkhvh uhvxowv vxjjhvw wkdw d vxevwdqwldo iudfwlrq ri wkh |rxqj rurjhqlf ehowv jduqhwvkdyh fkhplfdo frpsrvlwlrqv yhu| glvwlqfw iurp wkrvh ri wkh rwkhu wzr vhwwlqjv1 Wkh lqgxfhg uxohv +h1j1+4417,, surylgh lqvljkw dv wr wkh qdwxuh ri wkhvh glvwlqfwlrqv1

451 Plvvlqj ydoxhvLq wkh jhrorj| gdwd edvh wkhuh zhuh qr plvvlqj ydoxhv> doo frqfhqwudwlrq phdvxuhphqwv zhuh suhvhqwiru hdfk jduqhw1 Krzhyhu lq rwkhu frqwh{wv vxfk dv pdunhwlqj gdwd/ vdpsoh vxuyh| txhvwlrqqdluhv/ dqghyhq lq vrph vflhqwl�f gdwd vhwv/ plvvlqj ydoxhv duh d sureohp1 Lq vxfk vhwwlqjv d vwudwhj| lv qhhghgwr hqdeoh wkh surfhgxuh wr h�hfwlyho| surfhvv wkh dydlodeoh gdwd dqg qrw eh glvwudfwhg e| ydoxhv wkdw

46

Page 14: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

duh qrw suhvhqw1 Vxfk d vwudwhj| vkrxog dffrpsolvk wzr jrdov1 Iluvw/ li wkhuh lv dq dvvrfldwlrq ehwzhhqwkh rxwsxw yduldeoh dqg wkh suredelolwlhv ri plvvlqj ydoxhv rq ydulrxv lqsxwv/ wklv vkrxog eh xvhg wrdgydqwdjh1 Vhfrqg/ hyhq lq wkh devhqfh ri vxfk dvvrfldwlrqv/ rqh zrxog olnh wkh surfhgxuh wr h�hfwlyho|xvh ydoxhv ri lqsxw yduldeohv kljko| fruuhodwhg zlwk uhohydqw rqhv dv vxuurjdwhv zkhq wkh ydoxhv ri wkhfruuhvsrqglqj uhohydqw yduldeohv duh plvvlqj1

Lq wkh frqwh{w ri sdwlhqw uxoh lqgxfwlrq erwk jrdov fdq eh phw zlwk d vxusulvlqjo| vlpsoh vwudwhj|>wkh ydoxh {� @ plvvlqj lv vlpso| wdnhq dv dqrwkhu +ohjlwlpdwh, ydoxh wkdw lw fdq dvvxph1 Iru fdwhjrulfdoyduldeohv wklv uhsuhvhqwv qr fkdqjh wr wkh shholqj2sdvwlqj dojrulwkpv +Vhfwlrq :,1 Iru hdfk uhdo ydoxhgyduldeoh {� wkh qxpehu ri shholqj vxeer{hv e +:18, wkdw lw frqwulexwhv wr wkh holjleoh fodvv F+e, +:15,lv h{whqghg wr wkuhh

e � � @ i{ m{� ? { � � � � je � � @ i{ m{� A { � � � � � � j +4514,e � � @ i{ m{� @ plvvlqjj=

Wkh revhuydwlrqv lq wkh fxuuhqw er{ E iru zklfk {� @ plvvlqj duh ri frxuvh qrw xvhg wr frpsxwh { � � � �dqg {� � � � � � 1

Wklv vwudwhj| gluhfwo| phhwv wkh �uvw jrdo1 Dvvrfldwlrqv ehwzhhq wkh rxwsxw dqg plvvlqj lqsxwydoxhv duh fohduo| lqfrusrudwhg1 Lw dovr lqgluhfwo| phhwv wkh vhfrqg jrdo1 Wkh phdq |0ydoxh iru wkrvhrevhuydwlrqv plvvlqj d sduwlfxodu lqsxw ydoxh +{ � @ plvvlqj,/ wkurxjk d phfkdqlvp xquhodwhg wr wkhydoxh ri wkh rxwsxw/ lv olnho| wr eh forvh wr wkh joredo wdujhw phdq �|1 Dv vxfk/ wkh plvvlqj ydoxh lvxqolnho| wr eh fkrvhq iru shholqj/ hvshfldoo| lq wkh hduo| vwdjhv ri wkh wrs0grzq vhtxhqfh1 Li { � lvd kljko| uhohydqw lqsxw yduldeoh/ rwkhu lqsxwv vwurqjo| fruuhodwhg zlwk lw dsshdu uhohydqw e| yluwxhri wkdw fruuhodwlrq1 Vlqfh wkh ruljlqdo kljko| uhohydqw yduldeoh lv xqolnho| wr fkrrvh lwv plvvlqj ydoxhiru shholqj/ wkrvh yduldeohv fruuhodwhg zlwk lw wkdw frqwdlq ydoxhv zkhq wkrvh ri wkh ruljlqdo rqh duhplvvlqj kdyh lqfuhdvhg olnholkrrg ri ehlqj fkrvhq iru shholqj1

Wklv plvvlqj ydoxh vwudwhj| kdv wkh +shukdsv, xqsohdvdqw vlgh h�hfw ri forxglqj lqwhusuhwdwlrq1Zkhq +ixqgdphqwdoo|, luuhohydqw lqsxw yduldeohv duh fkrvhq iru shholqj ehfdxvh ri wkhlu vwurqj fruuh0odwlrqv zlwk uhohydqw rqhv wkh| duh lpsruwdqw wr wkh �qdo er{ gh�qlwlrq rqo| ehfdxvh ri wkh plvvlqjydoxhv rq wkh uhohydqw rqhv1 Wkh| vhuyh rqo| dv vxuurjdwhv dqg kdyh qr gluhfw lpsdfw rq wkh rxwsxwyduldeoh1 Lq wkh devhqfh ri plvvlqj ydoxhv wkh| zrxog kdyh ehhq pxfk ohvv olnho| wr eh fkrvhq iru shho0lqj dqg/ li fkrvhq/ pxfk pruh olnho| wr eh uhpryhg wkurxjk wkh uhgxqgdqw yduldeoh vwudwhj| rxwolqhglq Vhfwlrq 431

461 Pxowlsoh wudmhfwrulhvD shholqj wudmhfwru| +<14, +Ilj1 <14, lv lqgxfhg e| dq dssolfdwlrq ri wkh shholqj dojrulwkp1 Lw doorzvwkh xvhu wkh rssruwxqlw| wr fkrrvh wkh er{ phdq�vxssruw wudgh�r� dffruglqj wr mxgjhphqw dqgsurjudppdwlf qhhgv1 Wkh sduwlfxodu wudmhfwru| lqgxfhg ghshqgv xsrq wkh gdwd dqg wkh ghwdlov riwkh dojrulwkp1 Wkhvh ghwdlov lqyroyh ydoxhv ri phwd0sdudphwhuv wkdw frqwuro ydulrxv dvshfwv ri wkhsurfhgxuh1 Rqh vxfk lv wkh shholqj sdudphwhu � +:18,1 Rwkhuv duh glvfxvvhg lq Vhfwlrq 471 Dowkrxjk wkhsurfhgxuh lv idluo| vwdeoh/ gl�huhqw sdudphwhu ydoxhv fdq lqgxfh +dw ohdvw voljkwo|, gl�huhqw wudmhfwrulhvdqg wkh +�prgho vhohfwlrq�, lvvxh dulvhv ri zklfk rqh lv ehvw iurp wkh shuvshfwlyh ri wkh xvhu*v qhhgv1Dv zlwk rwkhu dvshfwv ri wkh surfhgxuh vxfk mxgjhphqwv duh ehvw pdgh e| wkh xvhu1

Wkh prgho vhohfwlrq vwudwhj| dgrswhg khuh lv wr vlpxowdqhrxvo| suhvhqw d odujh qxpehu ri wudmhf0wrulhv rq d vlqjoh sorw/ hdfk edvhg rq lwv sduwlfxodu vhw ri phwd0sdudphwhu ydoxhv1 Hdfk wudmhfwru|surgxfhv d vhw ri srlqwv olnh wkrvh ri Ilj1 <14 uhsuhvhqwlqj wkh phdq�vxssruw wudgh�r�v iru lwv qhvwhgvhtxhqfh ri er{hv1 Wkh xqlrq ri doo vxfk wudmhfwrulhv surgxfhv d vfdwwhu sorw zkhuh hdfk srlqw uhsuhvhqwvd er{ iurp rqh ri wkh lqglylgxdo wudmhfwrulhv1 Dv zlwk d vlqjoh wudmhfwru| +Ilj1 <14, wkh xvhu fkrrvhvwkh er{ uhsuhvhqwhg e| wkh srlqw rq wkh vfdwwhu sorw fruuhvsrqglqj wr wkh suhihuuhg phdq�vxssruwwudgh�r�1 Wkh lvvxh ri uhgxqgdqw lqsxw yduldeohv iru wkdw er{ fdq wkhq eh dgguhvvhg lq wkh pdqqhughvfulehg lq Vhfwlrq 43 +h1j1 Wdeoh 4314,1 Lq sudfwlfh vhyhudo vxfk er{hv duh lghqwl�hg uhsuhvhqwlqjsrwhqwldoo| surplvlqj phdq�vxssruw wudgh�r�v1 Hdfk rqh lv wkhq iroorzhg e| erwwrp�xs sdvwlqj dqg

47

Page 15: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

Support

Box

mea

n

0.0 0.2 0.4 0.6 0.8 1.0

23

45

67

8

Iljxuh 4614= Vlqjoh shholqj wudmhfwru| rq pdunhwlqj gdwd diwhu wklqqlqj/ wr eh frqwudvwhg zlwk wkhfruuhvsrqglqj pxowlsoh wudmhfwru| sorw lq wkh qh{w �jxuh1

wkhq uhgxqgdqw yduldeoh holplqdwlrq1 Wkh ryhudoo ehvw vroxwlrq/ dv mxgjhg e| wkh xvhu/ lv wkhq dgrswhgwr surgxfh wkh suhihuuhg er{ dw wklv vwdjh ri wkh fryhulqj vhtxhqfh +Vhfwlrq 9,1

Dgglwlrqdo wudmhfwrulhv fdq dovr eh lqgxfhg e| �expslqj� ^Wlevkludql dqg Nqljkw +4<<8,`1 Khuhrqh uhshdwhgo| prgl�hv wkh gdwd vxssolhg wr wkh dojrulwkp udwkhu wkdq prgli|lqj wkh dojrulwkp lwvhoi1Wkh sxusrvh lv wr plwljdwh lqvwdelolw| e| surgxflqj d odujh qxpehu ri udqgrpo| jhqhudwhg vroxwlrqv/hdfk rqh lqgxfhg e| udqgrpo| shuwxuelqj wkh ohduqlqj gdwd/ xvxdoo| wkurxjk �errwvwudsslqj� ^Hiurqdqg Wlevkludql +4<<8,`1 Wkh krsh lv wkdw rqh ri wkhvh udqgrp vroxwlrqv pd| +dfflghqwdoo|, wxuq rxw wreh ehwwhu +rq wkh ohiw0rxw whvw gdwd vhw, wkdq wkh rqh lqgxfhg rq wkh ruljlqdo ohduqlqj gdwd1 Dowkrxjklqwhqghg iru juhhg| surfhgxuhv vxfk dv ghflvlrq wuhhv wkdw duh lqkhuhqwo| yhu| xqvwdeoh ^Euhlpdq+4<<9,`/ expslqj pd| dovr vrphwlphv khos zlwk shholqj lq dqrpdorxv vlwxdwlrqv zkhuh wkhuh lv d kljkghjuhh ri lqkhuhqw dpeljxlw| dprqj wkh lqlwldo vhw ri shhov1 Lq vxfk fdvhv expslqj zloo whqg wr surgxfhd ydulhw| ri gl�huhqw lqlwldo shholqj vhtxhqfhv rqh ru pruh ri zklfk pd| ohdg wr d ehwwhu �qdo vhw rier{hv wkdq wkh vhtxhqfh ghulyhg iurp wkh ruljlqdo ohduqlqj gdwd vhw1

Zkhwkhu lqgxfhg e| prgli|lqj sdudphwhu ydoxhv ru expslqj/ d odujh qxpehu ri wudmhfwrulhv suh0vhqwhg rq d vlqjoh sorw fdq surgxfh d frqixvlqj slfwxuh1 Wkh odujh qxpehu ri srlqwv fdq revfxuhghwdlov pdnlqj lw gl!fxow wr fkrrvh d jrrg phdq0vxssruw wudgh0r�1 Wkh sorw fdq eh juhdwo| vlpsol�hge| �wklqqlqj� > wkh ydvw pdmrulw| ri srlqwv wkdw uhsuhvhqw xqolnho| fkrlfhv duh uhpryhg1 Dq| srlqw lwkdw uhsuhvhqwv d er{ zlwk vpdoohu phdq dqg vpdoohu vxssruw wkdq wkdw ri dqrwkhu er{ m

�| � ? �| � ) � � ? � � +4614,lv qrw olnho| wr eh vhohfwhg> rqh zrxog idyru wkh er{ uhsuhvhqwhg e| srlqw m1 Wkhuhiruh doo srlqwv wkdwduh grplqdwhg +4614, e| dqrwkhu srlqw rq wkh sorw fdq eh uhpryhg ohdylqj rqo| wkh xsshu hqyhorsh rixqgrplqdwhg srlqwv1 Wklv suhvhqwv d ohvv frqixvlqj slfwxuh pdnlqj lw hdvlhu iru wkh xvhu wr pdnh dqdssursuldwh fkrlfh1

Iljxuh 4614 vkrzv wkh vlqjoh wudmhfwru| surgxfhg e| shholqj xvlqj fulwhulrq +:15, rq wkh pdunhwlqjgdwd edvh ghvfulehg lq Vhfwlrq 48 +| @ qxpehu ri urxqg wuls ljkwv2|hdu,1 Iljxuh 4615 vkrzv dfruuhvsrqglqj sorw uhvxowlqj iurp 53 wudmhfwrulhv lqgxfhg e| 43 expsv rq hdfk ri wzr gl�huhqw shholqj

48

Page 16: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

Support

Box

mea

n

0.0 0.2 0.4 0.6 0.8 1.0

23

45

67

8

Iljxuh 4615= Pxowlsoh wudmhfwru| sorw rq pdunhwlqj gdwd diwhu wklqqlqj1 Frpsduhg wr wkh vlqjohwudmhfwru| sorw/ lw r�huv pruh phdq�vxssruw fkrlfhv dqg ehwwhu er{hv dw vpdoohu vxssruw ydoxhv1

fulwhuld +Vhfwlrq 47,1 Erwk sorwv kdyh ehhq wklqqhg +4614,1 Khuh/ iru � z 3=38 wkh wzr vhwv ri srlqwvwudfn hdfk rwkhu idluo| forvho|/ zlwk Ilj1 4615 r�hulqj d ulfkhu vhw ri dowhuqdwlyhv iurp zklfk wr fkrrvh1Iru � � 3=38 wkh pxowlsoh wudmhfwru| sorw surylghv d vhulhv ri er{hv zlwk pxfk kljkhu hvwlpdwhg rxwsxwphdq ydoxhv1 Wklv uhsuhvhqwv d fohdu dgydqwdjh li er{hv ri vpdoo vxssruw +� � 3=38, duh xvhixo lq wklvdssolfdwlrq1 Xvxdoo| wkh jdlqv surgxfhg e| hpsor|lqj pxowlsoh wudmhfwrulhv duh jhqhudoo| pruh prghvwwkdq wkdw revhuyhg khuh1 Sdwlhqw shholqj whqgv wr eh idluo| vwdeoh1 Krzhyhu/ iru prghudwho| vl}hggdwd vhwv +Q � 53333, zkhuh frpsxwdwlrq lv qrw d exughq/ pxowlsoh wudmhfwrulhv fdq surylgh lqvxudqfhdjdlqvw d srru fkrlfh ri phwd0sdudphwhu ydoxhv ru/ wkurxjk expslqj/ djdlqvw dq xqiruwxqdwh vhw ri+dpeljxrxv, shhov1

471 Shholqj fulwhuldWkh jxlglqj sulqflsoh ri wkh shholqj vwudwhj| +Vhfwlrq :14, lv sdwlhqfh +Vhfwlrq ;,1 Iru uhdo ydoxhglqsxw yduldeohv zlwk pdq| gl�huhqw ydoxhv/ wkh ghjuhh ri sdwlhqfh lv h�hfwlyho| frqwuroohg e| wkh phwd0sdudphwhu � +:18,1 Iru uhdo ydoxhg lqsxwv zlwk ihzhu glvwlqfw ydoxhv +pdq| wlhv,/ vxeer{hv e holjleohiru shholqj duh wdnhq wr kdyh vxssruw � � dv forvh dv srvvleoh wr wkh ydoxh ri �1 Wlhg ydoxhv duh dozd|vshhohg wrjhwkhu1 Iru fdwhjrulfdo yduldeohv wkh vxssruw � � ri holjleoh vxeer{hv +:19, fdqqrw eh gluhfwo|frqwuroohg1 Lq wkhvh odwwhu wzr fdvhv lw lv vwloo ghvludeoh wr hqfrxudjh sdwlhqfh wr wkh h{whqw srvvleoh1

47141 Vxeer{ fulwhuldWkh surfhgxuh glvfxvvhg lq Vhfwlrq :14 xvhv wkh wdujhw phdq �| � � � lq wkh qh{w vpdoohu er{ E�e +:15, dvwkh shholqj fulwhulrq wr eh pd{lpl}hg zlwk uhvshfw wr wkh vxeer{ e1 Wklv lv htxlydohqw wr pd{lpl}lqjwkh lpsuryhphqw

L+e, @ �| � � � � �| � +4714,ri wkdw ryhu wkh fxuuhqw er{ E1 Zkhq wkhuh duh vhyhudo frpshwlqj vxeer{hv zlwk vlplodu ydoxhv riL+e,/ wkh rqh zlwk wkh vpdoohvw vxssruw � � lv prvw ghvludeoh vlqfh lw lqgxfhv wkh prvw sdwlhqfh> pruh

49

Page 17: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

gdwd uhpdlqv iru ixuwkhu shholqj1 Wkhuhiruh/ lpsuryhphqw shu xqlw ri uhpryhg vxssruw � � ehfrphv dqlvvxh1

Sdwlhqfh fdq eh hqfrxudjhg e| xvlqj d prgl�hg shholqj fulwhulrqM+e, @ L+e, � S +� � , +4715,

zkhuh S +� � , lv d prqrwrqh ghfuhdvlqj ixqfwlrq ri lwv dujxphqw1 XvlqjS +� � , @ 4!� � +4716,

surgxfhv d fulwhulrq +4715, wkdw phdvxuhv lpsuryhphqw +4714, shu dprxqw ri uhpryhg vxssruw � � 1H{suhvvlqj +4714, dv

L+e, @ � �� � � � � ^�| � � �| � ` > +4717,

zkhuh �| � lv wkh rxwsxw phdq lq wkh shhohg vxeer{ e dqg � � lv wkh vxssruw ri wkh sduhqw E/ |lhogv wkhfulwhulrq

M+e, @ 4� � � � � ^�| � � �| � ` @ +�| � � � � �| � ,!� � = +4718,

Wkxv/ xvlqj +4716, lq +4715, surgxfhv dv d fulwhulrq wkh gl�huhqfh ehwzhhq wkh rxwsxw phdq ri wkh gdwduhpdlqlqj dqg wkdw ri wkh shhohg vxeer{1 Hyhq pruh sdwlhqfh fdq eh lqgxfhg e| fkrrvlqj

S +� � , @ +� � � � � ,!� � +4719,zklfk surgxfhv wkh phdq lq wkh shhohg vxeer{ �| � dv dq +htxlydohqw, fulwhulrq wr eh plqlpl}hg =

Hdfk ri wkh dowhuqdwlyhv +4714, +4716, +4719, fdq surgxfh +dw ohdvw vrphzkdw, gl�huhqw shholqjwudmhfwrulhv1 Doo ri wkhp fdq eh lqfoxghg wrjhwkhu dv sduw ri wkh pxowlsoh wudmhfwru| vwudwhj| ghvfulehglq Vhfwlrq 461 Lq wklv zd| wkh ehvw rqh iru wkh dssolfdwlrq dw kdqg fdq eh fkrvhq e| wkh xvhu147151 Lqsxw yduldeoh fulwhuldDoo ri wkh shholqj fulwhuld glvfxvvhg vr idu +4714, +4716, +4719, vwloo kdyh d juhhg| frpsrqhqw= wkh vxeer{e � wkdw rswlpl}hv wkhp lv fkrvhq wr surgxfh wkh qh{w er{ E � e � 1 Uhprylqj d gl�huhqw vxeer{ dw dsduwlfxodu vwhs pd| lq idfw surgxfh ehwwhu er{hv odwhu lq wkh shholqj vhtxhqfh1 Wkh jrdo ri d sdwlhqwvwudwhj| lv wr plwljdwh wklv h�hfw e| surylglqj d odujh qxpehu ri shhov lq wkh krsh wkdw odwhu vwhsv fdqfrpshqvdwh iru srru +juhhg|, fkrlfhv1 Expslqj fdq dovr khos lq wklv uhjdug1 Wkhvh fdq eh ylhzhgdv �sdvvlyh� vwudwhjlhv1 Rffdvlrqdoo|/ lq xqxvxdo vlwxdwlrqv/ wkh xvh ri pruh �surdfwlyh� vwudwhjlhv wrryhufrph juhhg fdq surgxfh ehwwhu wudmhfwrulhv1 Wkhvh fdq eh lqfoxghg dprqj wkh pxowlsoh wudmhfwrulhviurp zklfk wkh xvhu fdq fkrrvh1

Wkh deryh fulwhuld doo irfxv rq vxeer{hv e1 Wkh lqsxw yduldeohv i{ � j � � vhuyh rqo| wr gh�qh wkrvhholjleoh iru ghohwlrq +:18, +:19,1 D pruh surdfwlyh vwudwhj| frxog irfxv rq wkh lqsxw yduldeohv wkhpvhoyhv/dwwhpswlqj wr dvfhuwdlq zklfk rqhv dw hdfk vwhs duh prvw olnho| wr eh lq xhqflqj wkh wdujhw ixqfwlrqi+{, zlwklq wkh vxeuhjlrq gh�qhg e| wkh fxuuhqw er{/ { 5 E1 Rqh vxfk fulwhulrq wkdw uhtxluhv qrdgglwlrqdo frpsxwdwlrq lv

M� @ pd{ iM+e � ,j �plq iM+e � ,j +471:,zkhuh M+e, lv wkh ruljlqdo fulwhulrq +4715, dqg ie � j duh wkh vxeer{hv frqwulexwhg e| mwk lqsxw yduldeohwr wkh holjleoh vhw1 Wkh vxeer{ fkrvhq iru shholqj e� � � lv wkh rqh gh�qhg rq wkh rswlpdo lqsxw yduldeohm � @ dujpd{ � M� wkdw pd{lpl}hv wkh ruljlqdo fulwhulrq p � @ dujpd{ M+e� � ,1

Lq wklv frqwh{w +471:, lw fdq eh xvhixo wr gh�qh dq dgglwlrqdo +�fhqwudo�, vxeer{ dvvrfldwhg zlwkuhdo ydoxhg lqsxwv { �

e ��� @ i{ m{� � � � � {� � {� � � � � � j= +471;,Lqfoxglqj e ��� dorqj zlwk ie � � > e� � j +:18, lq +471:, pdnhv wkh surfhgxuh vhqvlwlyh wr lqsxw yduldeohv { �xsrq zklfk wkh wdujhw ixqfwlrq +zlwklq wkh fxuuhqw er{, kdv d vwurqj v|pphwulf frqyh{ ghshqghqfh/iru h{dpsoh i+{, @ { �

� / �4 � {� � 41 Lq wklv fdvh ghohwlqj hlwkhu h{wuhph vxeer{ e @ e ��� +:18,

4:

Page 18: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

ghfuhdvhv wkh phdq lq wkh uhpdlqlqj er{ �| � � � ? �| � 1 Iru wklv uhdvrq/ qrw lqfoxglqj e ��� +471;, lq+471:, zloo olnho| fdxvh wklv {� wr eh ljqruhg iru shholqj/ shukdsv lq idyru ri dq luuhohydqw lqsxw wkdwkdv qr h�hfw rq wkh rxwsxw | +�| � � � * �| � ,1 Qrwh wkdw e��� lv rqo| xvhg lq +471:, wr khos hydoxdwh wkhlpsruwdqfh ri hdfk uhdo ydoxhg lqsxw yduldeoh> lw lv qhyhu holjleoh iru dfwxdo ghohwlrq1

481 Pdunhwlqj gdwdLq wklv vhfwlrq wkh xvh ri SULP lv looxvwudwhg rq pdunhwlqj gdwd1 Lw frqvlvwv ri Q @ <73< txhvwlrqqdluhv+835 txhvwlrqv, �oohg rxw e| vkrsslqj pdoo fxvwrphuv lq wkh Vdq Iudqflvfr Ed| duhd ^Lpsdfw Uhvrxufhv/Lqf1 Froxpexv/ RK +4<;:,`1 Wkh �uvw 47 txhvwlrqv/ uhodwlqj wr ghprjudsklfv/ duh olvwhg lq Wdeoh 48141Wkhvh duh vhhq wr eh d pl{wxuh ri uhdo dqg fdwhjrulfdoo| ydoxhg yduldeohv1 Wkhuh duh pdq| plvvlqjydoxhv1 Wkh rwkhu 7;; txhvwlrqv duh frqfhuqhg zlwk ydulrxv frqvxphu ehkdylruv1 Wkhvh fdq eh xvhg dvrxwsxw yduldeohv | wr lghqwli| wkh ghprjudsklfv +lqsxw yduldeohv {, ri wkrvh uhvsrqghqwv zkr hqjdjhlq sduwlfxodu ehkdylruv1 Khuh zh fkrrvh iru looxvwudwlrq wkuhh fkdudfwhulvwlf ehkdylruv ri rqh ri wkhdxwkruv +MKI,1

Wdeoh 4814

Lqsxw yduldeohv iru wkh pdunhwlqj gdwd1Ydu1 Ghprjudsklf Qr1 ri ydoxhv Fdw14 vh{ 5 -5 pdulwdo vwdwxv 8 -6 djh 97 hgxfdwlrq 98 rffxsdwlrq < -9 lqfrph <: |hduv lq Ed| Duhd 8; pduulhg 0 gxdo lqfrphv 5 -< qxpehu lq krxvhkrog <43 qxpehu lq krxvhkrog ? 4; <44 krxvhkroghu vwdwxv 6 -45 w|sh ri krph 8 -46 hwkqlf fodvvl�fdwlrq ; -47 odqjxdjh lq krph 6 -

Wkh �uvw ehkdylru h{dplqhg lv iuhtxhqf| ri dlu wudyho dv fkdudfwhul}hg e| qxpehu ri urxqg wuls ljkwv shu |hdu1 Wkh joredo phdq ydoxh ryhu wkh hqwluh gdwd edvh lv �| @ 4=:1 Wkh �uvw wzr er{hvlqgxfhg e| SULP duh

| @ qxpehu ri ljkwv2|hdu> �| @ 4=:E � = �| � @ 7=5> � � @ 3=3;

hgxfdwlrq � 49 |hduvrffxsdwlrq 5 isurihvvlrqdo 2 pdqdjhuldo/ vdohv zrunhu/ krphpdnhujlqfrph A '83N/ ) 9@ plvvlqjqxpehu ri fkloguhq +? 4;, lq krph � 4

E � = �| � @ 6=5> � � @ 3=3:> Glvvlplodulw|= G+E � > E � , @ 3=471hgxfdwlrq A 45 |hduv/ ) 9@ plvvlqjlqfrph A '63N/ ) 9@ plvvlqj

4;

Page 19: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

4; ? djh ? 87pduulhg 2 gxdo lqfrphv 5 ivlqjoh/ pduulhg�rqh lqfrphj

Wkh er{ �glvvlplodulw|� gldjqrvwlf G+E � > E � , +4918, lv ghvfulehg lq Vhfwlrq 49161 Wkh �uvw er{lghqwl�hv wkh ghprjudsklfv ri d ;( pdunhw vhjphqw wkdw dyhudjhv 7=5 ljkwv shu |hdu/ dqg wkh vhfrqger{ dqrwkhu :( vhjphqw/ glvwlqfw iurp wkh �uvw/ zlwk doprvw grxeoh wkh joredo dyhudjh ri 4=:1 Wkhvher{hv yhuli| lqwxlwlrq> wkhuh duh qr uhdo vxusulvhv uhyhdohg e| wkhvh uxohv1

Wkh qh{w ehkdylru h{dplqhg lv wkdw ri rzqlqj d shw1 Iliw| wzr shufhqw ri wkh uhvsrqghqwv lqglfdwhgwkdw wkh| kdg d shw> wkh rggv ri d udqgrpo| vhohfwhg shuvrq lq wkh vdpsoh rzqlqj d shw duh urxjko|4241 Wkh �uvw wzr er{hv lqgxfhg e| SULP duh

| @ 4+kdyh d shw,1 �| @ 3=851E � = �| � @ 3=;3> � � @ 3=4:

djh � 77hgxfdwlrq � 47 |hduvolyh lq Ed| Duhd � 7 |hduvkrph 5 ikrxvh/ prelohjhwkqlf fodvv 5 iQdwlyh Dphulfdq/ Hdvw Lqgldq/Zklwh/ plvvlqjj

E � = �| � @ 3=:9> � � @ 3=3;/ Glvvlplodulw|= G+E � > E � , @ 3=771qxpehu ri fkloguhq +? 4;, lq krph A 3krxvhkrog vwdwxv 5 irzq/ olyh zlwk sduhqwv/ plvvlqjjhwkqlf fodvv 5 iQdwlyh Dphulfdq/ Hdvw Lqgldq/Zklwh/ plvvlqjj

SULP lghqwl�hg wzr pdunhw vhjphqwv froohfwlyho| fryhulqj 58( ri wkh hqwluh vdpsoh iru zklfk wkhrggv ri rzqlqj d shw duh urxjko| 8241 Shukdsv d vxusulvh khuh lv wkh lpsruwdqfh ri hwkqlf fodvv lqghwhuplqlqj olnholkrrg ri shw rzqhuvkls1

Wkh wklug h{dpsoh uhodwhv wr udglr olvwhqlqj kdelwv/ vshfl�fdoo| lqfolqdwlrq wr olvwhq wr NJR0DP

| @ olvwhq wr NJR0DP @;

?

=

3=3 qhyhu3=58 rffdvlrqdoo|4=3 uhjxoduo|1

Wkh joredo dyhudjh lv �| @ 3=431 Wkh �uvw wzr uxohv lqgxfhg e| SULP duhE � = �| � @ 3=56> � � @ 3=47

djh � 68olyh lq Ed| Duhd � 43 |hduv/ ) 9@ plvvlqjqxpehu ri fkloguhq lq krph � 4krxvhkroghu vwdwxv 9@ uhqwhwkqlf fodvv @5 iEodfn/ Sdfl�f Lvodqghujw|sh ri krph 5 ikrxvh/ frqgrj

E � = �| � @ 3=53> � � @ 3=3<> Glvvlplodulw|= G+E � > E � , @ 3=::1vh{ @ pdohpdulwdo vwdwxv 5 i pduulhg/ glyrufhg ru vhsdudwhg/ plvvlqjjrffxsdwlrq 5 isurihvvlrqdo 2 pdqdjhuldo/ vdohv zrunhu/ uhwluhgj

4<

Page 20: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

Wkhvh �uvw wzr er{hv fryhu d 56( pdunhw vhjphqw dqg lghqwli| ghprjudsklfv iru zklfk NJR0DPkdv ryhu wzlfh wkh olvwhqhu vkduh wkdq lq wkh hqwluh vdpsoh1

491 GldjqrvwlfvWkh edvlf rxwsxw ri wkh SULP surfhgxuh lv d vhw ri uxohv +818, gh�qlqj d vhulhv ri er{hv +814,1 Wkh|fdq eh xvhg wr lqihu vlpxowdqhrxv ydoxhv ri sduwlfxodu lqsxw yduldeohv wkdw uhodwh wr odujh ydoxhv ri wkhrxwsxw yduldeoh1 Wkhvh uxohv dovr gh�qh uhvshfwlyh vxevhwv ri wkh gdwd/ qdpho| wkrvh revhuydwlrqv wkdwolh zlwklq hdfk er{ E �

i| � >{ � m{ � 5 E � j= +4914,Vxppdu| vwdwlvwlfv edvhg rq wkhvh revhuydwlrqv +4914, fdq eh xvhg wr surylgh ixuwkhu lqvljkw dv wr wkhqdwxuh ri wkh lqgxfhg uhjlrq +814,149141 Vhqvlwlylw| dqdo|vlvDq lpsruwdqw lqjuhglhqw ri dq| gdwd dqdo|vlv lv ghwhuplqlqj wkh vhqvlwlylw| ri wkh remhfwlyh wr wkhlqgxfhg sdudphwhu ydoxhv1 Khuh wkh remhfwlyh lv +kljk, er{ phdq +:14, dqg wkh sdudphwhuv duh wkh er{erxqgdulhv +818,1 Wklv vhqvlwlylw| fdq eh lqihuuhg iurp wkh udwh ri fkdqjh ri wkh wdujhw phdq dw wkherxqgdu| ydoxhv rq hdfk lqsxw yduldeoh gh�qlqj wkh vroxwlrq iru hdfk er{1 Ohw E eh d vroxwlrq er{ dqgE � � wkh fruuhvsrqglqj +odujhu, er{ zlwk wkh mwk lqsxw yduldeoh {� uhpryhg iurp lwv gh�qlwlrq+4314,1Wkh ixqfwlrq �i� +{� , @ H^| m{� ) { 5 E � � ` +4915,uhsuhvhqwv wkh wdujhw phdq iru hdfk ydoxh ri { � / jlyhq wkh vroxwlrq erxqgdu| ydoxhv rq wkh rwkhuyduldeohv i{� � j� � �� � gh�qlqj wkh er{ +818,1 Wkh vorsh ri wklv ixqfwlrq g �i� @g{� dw wkh { � er{ erxqgdulhvsurylghv d phdvxuh ri wkh vhqvlwlylw| ri wkh er{ phdq wr wkrvh sduwlfxodu erxqgdu| ydoxhv1 Iru uhdo�ydoxhg yduldeohv dq hvwlpdwh ai� +{ � , ri +4915, fdq eh rewdlqhg e| vprrwklqj wkh rxwsxw yduldeoh | rq{ � iru wkrvh revhuydwlrqv frqwdlqhg lq wkh hqodujhg er{ E � �

ai� +{� , @ vprrwk+| � � yv1 { � � m { � 5 E � � ,= +4916,Iru fdwhjrulfdo yduldeohv/ +4916, lv vlpso| wkh rxwsxw phdq iru hdfk ri lwv ydoxhv/ { � 5 V � / lq E � � 1

Iljxuh 4914 vkrzv sorwv ri +4916, iru wkh irxu yduldeohv gh�qlqj wkh �uvw er{ E � ri wkh iuhtxhqw |hu h{dpsoh +| @ qxpehu ri ljkwv shu |hdu, ri Vhfwlrq 481 +Wkh lqsxw yduldeoh ydoxhv khuh duhgl�huhqw wkdq wkrvh lqglfdwhg lq Vhfwlrq 48 vlqfh hdfk ydoxh lq wkh gdwd vhw uhsuhvhqwv d frgh iru wkhfruuhvsrqglqj dfwxdo ydoxh1, Iru wkh fdwhjrulfdo yduldeoh +{� @ rffxsdwlrq 0 xsshu uljkw iudph, wkhydoxhv ri +4916, duh uhsuhvhqwhg e| yhuwlfdo eduv1 Dvvrfldwhg zlwk hdfk edu duh wzr txdqwlwlhv vkrzqehorz lw1 Wkh �uvw lv wkh ydoxh ri {� zlwk �P� lqglfdwlqj plvvlqj ydoxh1 Wkh vhfrqg lv wkh iudfwlrq+qhduhvw shufhqw, ri wkh revhuydwlrqv lq wkh fruuhvsrqglqj er{ E � � kdylqj wkdw sduwlfxodu ydoxh ri { � 1Wkh krul}rqwdo olqh dorqj wkh wrs ri wkh sorw lv wkh rxwsxw phdq lq wkh vroxwlrq er{ E/ surylghg iruuhihuhqfh1 Wkh vkdglqj +ru qrw, ri hdfk edu lqglfdwhv zkhwkhu +ru qrw, lwv fruuhvsrqglqj ydoxh lv lqwkh vhw v � � +818, gh�qlqj wkh er{ E1 Khuh wkh ydoxhv rffxs 5 i4> 5> 8j gh�qh wkh er{ rq wklv yduldeoh1Wkh vhqvlwlylw| ri wkh vroxwlrq er{ E wr hdfk fdwhjrulfdo ydoxh lv uh hfwhg e| wkh khljkw ri lwv eduuhodwlyh wr wkh krul}rqwdo olqh/ dqg lwv uhodwlyh vxssruw +shufhqw,1 Khuh rqh vhhv wkdw hqodujlqj wkh er{E e| lqfoxglqj dgglwlrqdo ydoxhv zrxog uhgxfh lwv phdq1 Wkh er{ phdq lv ohdvw vhqvlwlyh wr lqfoxvlrqri ydoxhv { � 5 i9> :>Pj zklfk kdyh ydoxhv ri +4916, forvhvw wr wkh er{ phdq1 Krzhyhu/ wkhvh ydoxhvfrpsulvh rqo| d vpdoo iudfwlrq ri wkh gdwd vr er{ vxssruw lv qrw vljql�fdqwo| lqfuhdvhg1 Ri wkrvh ydoxhvgh�qlqj wkh er{/ wkh h{foxvlrq ri { � @ 8 zrxog lqfuhdvh wkh er{ phdq zlwk d prghudwh uhgxfwlrq lq lwvvxssruw1 H{foxglqj hlwkhu ri wkh rwkhu wzr ydoxhv { � 5 i4> 5j zrxog vxevwdqwldoo| uhgxfh wkh vroxwlrqphdq dqg vxssruw1

Wkh rwkhu wkuhh iudphv ri Ilj14914 uhsuhvhqw wkh rughudeoh +uhdo, ydoxhg yduldeohv gh�qlqj wkhvroxwlrq er{1 Wkh olqh uhsuhvhqwv wkh vprrwk +4916, iru hdfk/ zlwk wkh srlqwv uhsuhvhqwlqj wkh +khuh,glvfuhwh ydoxhv ri hdfk fruuhvsrqglqj yduldeoh1 Wkh er{ erxqgdulhv rq hdfk uhvshfwlyh yduldeoh duh

53

Page 21: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

edu

outp

ut

2 3 4 5 6

12

34

5 M

| | | | | | | | | 0

1 2 3 4 5 6 7 8 9 M

01

23

4

occup

outp

ut

68 7 1 4 5 0 1 11 2 1

income

outp

ut

2 4 6 8 10

12

34

5

M

| | | | | | | | | 0

num < 18

outp

ut

1 2 3 4 5

12

34

5

M

| | | |7

Iljxuh 4914= Vhqvlwlylw| sorwv iru wkh �uvw er{ ri wkh iuhtxhqw |hu h{dpsoh1 Wkhvh fdq eh xvhg wr mxgjhwkh vhqvlwlylw| ri wkh er{ phdq wr wkh ydoxhv ri wkh lqgxfhg erxqgdulhv/ dv zhoo dv vxjjhvw srwhqwldoprgl�fdwlrqv wr wkh erxqgdulhv1

54

Page 22: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

lqglfdwhg e| wkh yhuwlfdo olqhv1 Wkh krul}rqwdo olqh ehwzhhq wkh erxqgdulhv rq hdfk sorw lv wkh vroxwlrqer{ E phdq1 Khuh hdfk yduldeoh kdsshqv wr gh�qh rqo| rqh erxqgdu| zlwk wkh rwkhu rqh ehlqjlpsolflwo| gh�qhg e| wkh fruuhvsrqglqj xsshu ru orzhu olplw ri wkh sorw1 Wkh �P� dw wkh uljkw ri hdfksorw uhsuhvhqwv +4916, iru plvvlqj ydoxhv dqg wkh qxpehu ehorz lw vkrzv wkh iudfwlrq ri {� @ plvvlqj+wr wkh qhduhvw 43(, iru revhuydwlrqv lq wkh hqodujhg er{ E � � 1 Wkh vpdoo yhuwlfdo �kdvk� pdunv dorqjwkh erwwrp ri hdfk sorw gholqhdwh wkh ghflohv ri wkh glvwulexwlrq ri { � iru revhuydwlrqv lq E � � 1 IurpIlj14914 rqh vhhv wkdw wkh rxwsxw phdq lq wkh vroxwlrq er{ E lv ohdvw vhqvlwlyh wr vpdoo fkdqjhv lq wkhlqfrph erxqgdu| dqg fruuhvsrqglqjo| pxfk pruh vhqvlwlyh zlwk uhvshfw wr wkh rwkhu wzr rughudeohyduldeohv +hgxfdwlrq dqg qxpehu ri fkloguhq lq wkh krxvhkrog zlwk djh ? 4;,1 Qrwh wkdw iru wkh odwwhuyduldeoh/ dssur{lpdwho| :3( ri lwv ydoxhv lq E � � duh plvvlqj +�:� lq orzhu uljkw fruqhu,1 Wkhvh prvwo|uhsuhvhqw wkh ydoxh }hur zklfk zdv frghg dv plvvlqj iru wklv yduldeoh lq wkh gdwd vhw1

Dv lqglfdwhg e| wklv h{dpsoh/ vhqvlwlylw| dqdo|vlv olnh wkdw surylghg lq Ilj14914 fdq dlg wkh xvhu lqlqwhusuhwlqj wkh vroxwlrq surgxfhg e| SULP/ dqg shukdsv dovr lq pdnlqj prgl�fdwlrqv wr lw1 Zlwk dqlqwhudfwlyh lpsohphqwdwlrq wkh xvhu frxog rswlrqdoo| prgli| wkh er{ erxqgdulhv edvhg rq lqirupdwlrqsurylghg e| wkh vhqvlwlylw| sorwv1 Fruuhvsrqglqj vhqvlwlylw| sorwv fdq wkhq eh surgxfhg iru wkh prgl�hger{/ shukdsv vxjjhvwlqj ixuwkhu fkdqjhv1 Wklv fdq eh lwhudwhg xqwlo wkh xvhu lv vdwlv�hg zlwk wkh er{erxqgdulhv rq doo yduldeohv1 Vxfk dq lpsohphqwdwlrq hvvhqwldoo| shuplwv wkh xvhu wr pdqxdoo| h{hfxwhwkh shholqj2sdvwlqj vwudwhj|/ vwduwlqj zlwk wkh SULP vroxwlrq1 Dv zlwk rwkhu dvshfwv ri wkh SULPsurfhgxuh/ wklv shuplwv wkh ixoo lqfrusrudwlrq ri xvhu mxgjhphqw dqg grpdlq nqrzohgjh149151 Uhodwlyh iuhtxhqf| sorwvWkh uxohv +818, surylgh d gh�qlwlrq ri hdfk er{ E � 1 Krzhyhu/ fduh lv uhtxluhg lq lqwhusuhwlqj hdfkri wkhvh gh�qlwlrqv dv d frpsohwh ru xqltxh ghvfulswlrq ri E � 1 Rwkhu ghvfulswlrqv/ edvhg rq gl�huhqwlqsxw yduldeohv/ pd| ohdg wr yhu| vlplodu er{hv lq whupv ri wkh dfwxdo gdwd srlqwv +4914, wkdw duhfryhuhg1 Wklv lv fdxvhg e| froolqhdulw| dprqj wkh lqsxw yduldeohv1 Dv glvfxvvhg lq Vhfwlrq 43 kljkfruuhodwlrq ehwzhhq wzr +ru pruh, lqsxw yduldeohv zlwklq d er{ fdq surgxfh dpeljxlw| lq lwv gh�qlwlrq>dssursuldwho| uhvwulfwlqj wkh udqjh ri ydoxhv ri hlwkhu yduldeoh +ru erwk, jlyhv ulvh wr vlplodu vxevhwvri revhuydwlrqv1 Rqh ri wkh jrdov ri wkh uhgxqgdqw yduldeoh holplqdwlrq vwudwhj| lv wr vhohfw wkh ehvw+prvw kljko| uhvwulfwlyh, yduldeoh zlwklq hdfk vxfk kljko| fruuhodwhg jurxs ri lqsxwv1

Iurp wkh shuvshfwlyh ri lqwhusuhwdwlrq lw lv lpsruwdqw wr eh dzduh ri srvvleoh dowhuqdwlyh gh�qlwlrqviru hdfk lqgxfhg er{ E � 1 Wkh uxoh +818, gh�qhv wkh gdwd +4914, zlwklq wkh er{1 Wklv gdwd fdq ehxvhg wr frpsduh wkh uhodwlyh iuhtxhqf| glvwulexwlrq ri ydoxhv ri hdfk lqsxw yduldeoh { � zlwklq wkh er{s � +{ � m{ 5 E � , wr wkdw ryhu wkh hqwluh gdwd vdpsoh s � +{ � ,1 Lqsxw yduldeohv +rwkhu wkdq wkrvh xvhgwr gh�qh wkh er{, iru zklfk wkh iruphu glvwulexwlrq lv pruh kljko| shdnhg wkdq wkh odwwhu/ uhsuhvhqwfdqglgdwhv iru dowhuqdwlyh gh�qlwlrqv1 Olnh wkh lqsxw yduldeohv wkdw h{solflwo| gh�qh E � +818, wkh udqjhri ydoxhv ri wkhvh rwkhu yduldeohv lv dovr uhvwulfwhg zlwklq wkh er{1

Wkhuh duh d ydulhw| ri zd|v wr frpsduh wzr glvwulexwlrqv1 Rqh srvvlelolw| lv wkurxjk wkhlu udwlru � � +{� , @ s � +{ � m{ 5 E � ,!s � +{� ,= +4917,

Qrwh wkdw wkh vxssruw ri wkh ghqrplqdwru lv dozd|v juhdwhu wkdq wkdw ri wkh qxphudwru1 D xqlirupglvwulexwlrq iru u � � +{� , lpsolhv wkdw {� lv wrwdoo| luuhohydqw wr wkh gh�qlwlrq ri E � > wkh uhodwlyh iuhtxhqf|ri lwv ydoxhv lv wkh vdph lqvlgh ru rxwvlgh ri wkh er{1 Ghsduwxuhv iurp xqliruplw| lqglfdwh dvvrfldwlrqzlwk wkh er{ gh�qlwlrq1 D kljko| shdnhg glvwulexwlrq iru u � � +{� , lpsolhv wkdw wkh lqsxw {� lv kljko|uhohydqw wr wkh gh�qlwlrq ri E � zkhwkhu ru qrw lw lv rqh ri wkh gh�qlqj yduldeohv +818,1

Iljxuh 4915 vkrzv uhodwlyh iuhtxhqf| udwlr sorwv iu ��� +{ � ,j � �� +4917, iru wkh �uvw er{E � ri wkh iuhtxhqw |hu h{dpsoh +| @ qxpehu ri ljkwv shu |hdu, ri Vhfwlrq 481 Wkh odvw elq ri hdfk sorw uhsuhvhqwv wkhydoxh { � @ plvvlqj1 Wkh rwkhu elqv vkrz wkh uhvshfwlyh u ��� +{� , iru wkh qrq0plvvlqj ydoxhv1 Wkh �uvwiudph u �� +{ � , vkrzv wkdw jhqghu lv qrw kljko| uhohydqw wr wkh gh�qlwlrq ri E � > phq +{ � @ 4, duh voljkwo|ryhu�uhsuhvhqwhg dprqj wkhvh iuhtxhqw lhuv1 Wkh vhfrqg sorw vkrzv wkdw zlgrzhg shrsoh +{ � @ 7,duh pxfk ohvv olnho| wr eh dprqj wkhvh iuhtxhqw |huv wkdw lq wkh wrwdo vdpsoh1 Wkh sorw iru { � +djh,lqglfdwhv wkdw |rxqj ^{ � � 5 +57 |hduv rog,` dqg roghu ^{ � � : +88 |hduv rog,` shrsoh duh kljko| xqghu�uhsuhvhqwhg lq E � 1 Wkh glvwulexwlrqv ri { � / { � / dqg { � +hgxfdwlrq/ rffxsdwlrq/ dqg lqfrph, uh hfw

55

Page 23: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

0.0

0.4

0.8

1.2

sex

rel.

freq

.

0.0

1.0

mar-stat re

l. fr

eq.

0.0

1.0

2.0

age

rel.

freq

.

01

23

4

edu

rel.

freq

.

0.0

1.0

2.0

occup

rel.

freq

.

01

23

45

income

rel.

freq

.

0.0

0.4

0.8

1.2

yrs-BA

rel.

freq

.

0.0

1.0

2.0

mar-dlinc

rel.

freq

.

0.0

1.0

num-hsld

rel.

freq

.

0.0

0.4

0.8

1.2

num < 18

rel.

freq

.

0.0

1.0

2.0

hsld-stat

rel.

freq

.

0.0

0.5

1.0

1.5

typ-home

rel.

freq

.

0.0

0.4

0.8

1.2

ethnic

rel.

freq

.

0.0

0.4

0.8

lang

rel.

freq

.

01

23

4

output

rel.

freq

.

Iljxuh 4915= Uhodwlyh iuhtxhqf| udwlr glvwulexwlrqv iru wkh �uvw er{ ri wkh iuhtxhqw |hu h{dpsoh1Vkrzq duh klvwrjudpv ri wkh iuhtxhqf| ri ydoxhv iru hdfk yduldeoh iru revhuydwlrqv lq wkh er{/ uhodwlyhwr wkrvh lq wkh hqwluh gdwd vhw1 Wkhvh fdq eh xvhg wr dvvhvv wkh orfdwlrq ri wkh er{ zlwk uhvshfw wr wkhydoxhv ri doo lqsxw yduldeohv/ lqfoxglqj wkrvh wkdw gr qrw dsshdu h{solflwo| lq wkh er{ gh�qlwlrq1

56

Page 24: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

wkh er{ gh�qlwlrq +Vhfwlrq 48,1 Wkh sorw ri u ��� +{ � , vkrzv wkdw { � @ 4 +rffxsdwlrq @ surihvvlrqdo2 pdqdjhuldo, duh doprvw wkuhh wlphv pruh olnho| wr eh dprqj wkh iuhtxhqw |huv ri E � wkdq lq wkhwrwdo vdpsoh1 Wkh glvwulexwlrq ri u � � +{ � , +qxpehu lq krxvhkrog, uh hfwv wkh er{ gh�qlwlrq rq qxpehuri fkloguhq lq wkh krxvh ohvv wkdq 4; |hduv rog1 Wkh sorw ri u ���� � +{ �� , +krxvhkroghu vwdwxv, uh hfwvwkdw ri u ��� +{ � , +djh,1 Wkrvh shrsoh zkr olyh zlwk sduhqwv ru idplo| +{ �� @ 6, whqg wr eh |rxqjhu+{ � � 57 |hduv rog,1 Wkh glvwulexwlrq ri u �W��� � +{ �W� , +w|sh ri krph, lqglfdwhv dq ryhu�uhsuhvhqwdwlrq rifrqgrplqlxp rzqhuv +{ �W� @ 5, dqg d gudpdwlf xqghu�uhsuhvhqwdwlrq ri preloh krph rzqhuv +{ �W� @ 7,lq E � 1 Wkh hwkqlf fodvvl�fdwlrq sorw u � ��� � +{ � � , vkrzv dq ryhu uhsuhvhqwdwlrq ri Hdvw Lqgldqv +{ � � @ 7,dqg zklwhv +{ � � @ :,/ dqg dq xqghu�uhsuhvhqwdwlrq ri Diulfdq Dphulfdqv +{ � � @ 6, dqg hvshfldoo|Klvsdqlfv +{ � � @ 8,1 Wklv lv dovr uh hfwhg lq u � ��� � +{ � � ,> shrsoh zkr vshdn Vsdqlvk dw krph +{ � � @ 5,duh ohvv olnho| wr eh dprqj wkh iuhtxhqw |huv lq E � 1

Wkh odvw iudph ri Ilj1 4915 vkrzv wkh glvwulexwlrq fruuhvsrqglqj wr +4917, iru wkh rxwsxw yduldeoh/u � � +|,1 Rqh vhhv wkdw doo ydoxhv ri | duh uhsuhvhqwhg lq E � 1 Shrsoh zkr | qlqh ru pruh wlphv shu |hdu+| @ <, duh urxjko| irxu wlphv pruh olnho| wr eh lq E � wkdq lq wkh jhqhudo srsxodwlrq/ zkhuhdv wkrvhrggv duh uhyhuvhg iru wkrvh zkr grq*w | dw doo +| @ 3,1

Wkh kljko| qrq0xqlirup glvwulexwlrqv ri u ��� +{ � , +4917, iru pdq| ri wkh lqsxwv qrw gluhfwo| gh�qlqjE � lqglfdwh wkh suhvhqfh ri vwurqj dvvrfldwlrqv dprqj pdq| ri wkh lqsxwv zlwklq wkh er{1 Wkh fkrlfhkhuh ri { � / { � / { � / dqg { � � wr h{solflwo| gh�qh E � lv edvhg rq wkh gdwd> wkh| dsshdu wr eh wkh prvwkljko| uhohydqw dv hvwlpdwhg e| wkh uhgxqgdqw yduldeoh holplqdwlrq vwudwhj| ri Vhfwlrq 431 Dv zlwk dq|hvwlpdwlrq surfhgxuh/ yduldqfh fdq fdxvh huudqw fkrlfhv dprqj forvh dowhuqdwlyhv1 Dovr lw lv srvvleoh +lixqolnho|, wkdw d kljko| uhohydqw lqsxw yduldeoh fdq eh frpsohwho| pdvnhg e| rwkhuv kljko| fruuhodwhgzlwk lw/ dqg qhyhu eh fkrvhq iru shholqj1 Dowkrxjk wkh sdwlhqw vwudwhj| +Vhfwlrq ;, dv zhoo dv pxowlsohwudmhfwrulhv +Vhfwlrq 46, khos plwljdwh wklv srvvlelolw|/ lw fdqqrw eh frpsohwho| holplqdwhg1 H{dplqlqjwkh glvwulexwlrqv ri iu � � +{� ,j � � +4917, fdq khos gldjqrvh vxfk sureohpv dv zhoo dv jlylqj dq ryhudoolpsuhvvlrq ri wkh fkdudfwhulvwlfv ri wkh lqgxfhg er{1 Ri frxuvh/ li wzr ru pruh kljko| fruuhodwhgyduldeohv dsshdu iurp wkh gdwd wr eh qhduo| htxdoo| uhohydqw/ rqo| xvhu mxgjhphqw fdq dvfhuwdlq zklfkrqh +ru erwk, dfwxdoo| duh uhodwhg wr wkh rxwsxw |1

Iru wkh pdunhwlqj gdwd +Wdeoh 4814, wkhuh duh dw prvw qlqh gl�huhqw qrq�plvvlqj ydoxhv iru dq|lqsxw yduldeoh1 Zlwk d vdpsoh vl}h ri Q @ <73< wkh ydoxh ri u� � +{ � , +4917, fdq eh uholdeo| hvwlpdwhgiru hdfk lqglylgxdo ydoxh ri {� 1 Lq pdq| gdwd vhwv uhdo�ydoxhg yduldeohv fdq dvvxph d odujh qxpehuri glvwlqfw ydoxhv1 Wkh jhrorj| gdwd glvfxvvhg lq Vhfwlrq 44 lv dq h{dpsoh1 Iru vxfk uhdo�ydoxhglqsxwv { � / sorwv ri +4917, dqdorjrxv wr Ilj1 4915 duh pdgh e| klvwrjudpplqj wkh { � 0ydoxhv xvlqj dvelq erxqgdulhv wkh �0txdqwlohv ri wkh glvwulexwlrq ri {� ryhu wkh hqwluh gdwd vhw s � +{� ,1 Wkh qxpehuri elqv +4@�, lv fkrvhq e| wkh xvhu edvhg rq vdpsoh vl}h Q dqg wkh ghjuhh ri ghwdlo uhtxluhg1

Ilqdoo|/ lw lv lpsruwdqw wr uhphpehu wkdw wkh sorwv lq Ilj1 4915 gr qrw uhsuhvhqw wkh iuhtxhqf|glvwulexwlrq ri wkh { � 0ydoxhv s � +{ � m{ 5 E � , zlwklq wkh er{ E � 1 Udwkhu/ wkh| uhsuhvhqw wkrvh iuhtxhq0flhv uhodwlyh wr wkh fruuhvsrqglqj iuhtxhqflhv s� +{� , ryhu wkh hqwluh gdwd vhw +4917,1 Iru h{dpsoh/ wkhydoxh | @ < lv qrw wkh prvw iuhtxhqw rxwsxw ydoxh lq E � +odvw iudph,1 Wklv ydoxh lv yhu| lqiuhtxhqw lqwkh hqwluh gdwd vhw1 Wkh ydoxh s � � +<, @ 7=5 lqglfdwhv wkdw wkh rxwsxw ydoxh | @ < lv 7=5 wlphv pruholnho| wr dsshdu lq E � wkdq lq wkh gdwd vhw dv d zkroh1 Sorwv ri s � +{ � m{ 5 E � , fdq dovr vhuyh dv xvhixodgglwlrqdo gldjqrvwlfv dv fdq vwdwlvwlfv rwkhu wkdq +4917, iru frpsdulqj lw wr s � +{ � ,1

49161 Lqwhuer{ glvvlplodulw|Wkh fryhulqj surfhgxuh +Vhfwlrq 9, surgxfhv d vhtxhqfh ri er{hv iE � j �

� wkdw fryhu d vxeuhjlrq U+814, ri wkh lqsxw yduldeoh vsdfh1 Qr frqvwudlqwv duh sodfhg rq wkh uhodwlrqvklsv dprqj wkh er{hv1Wkh| fdq ryhuods ru eh glvmrlqw/ eh forvh ru idu dsduw/ dv glfwdwhg e| wkh qdwxuh ri wkh wdujhw ixqf0wlrq1 Iru h{dpsoh/ li wkh wdujhw ixqfwlrq lv fkdudfwhul}hg e| d vlqjoh surplqhqw prgh/ wkh fryhulqjsurfhgxuh pljkw surgxfh d vhulhv ri qhvwhg er{hv/ hdfk vxffhvvlyho| lqgxfhg er{ frpsohwho| fryhulqjdoo ri wkrvh lqgxfhg ehiruh lw1 Dowhuqdwlyho|/ vxffhvvlyh er{hv pljkw fryhu gl�huhqw �vkrxoghuv� riwkdw prgh surgxflqj d �foxvwhu� ri forvho| uhodwhg er{hv1 Li wkhuh zhuh vhyhudo gl�huhqw surplqhqwprghv/ wkh er{hv pljkw glylgh lqwr fruuhvsrqglqj jurxsv ri qhvwhg2foxvwhuhg vhtxhqfhv1 Nqrzohgjhfrqfhuqlqj uhodwlrqvklsv dprqj wkh er{hv fdq surylgh lqirupdwlrq derxw wkh prgdo vwuxfwxuh ri wkh

57

Page 25: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

wdujhw ixqfwlrq1Uhodwlrqvklsv dprqj wkh er{hv fdq eh ghgxfhg iurp wkhlu sdluzlvh glvvlplodulwlhv1 Wkh �glvvlp0

lodulw|� G+E � > E � , ehwzhhq er{hv E � dqg E � lv gh�qhg dv wkh gl�huhqfh ehwzhhq wkh vxssruw ri wkhvpdoohvw er{ E � � wkdw fryhuv erwk ri wkhp/ dqg wkh vxssruw ri wkhlu xqlrq

G+E � > E � , @ �+E � � ,� �+E �VE � ,= +4918,

Khuh wkh vxssruw �+E,> ri d er{ E/ lv gh�qhg dv wkh iudfwlrq ri revhuydwlrqv lq wkh hqwluh gdwd vhw+414, wkdw lw fryhuv1 Wkh plqlpdo fryhulqj er{ lv gh�qhg dv

{ 5 E � � @�_

� � �^{ � 5 v � +n> o,` +4919,

zkhuh wkh ydoxh vxevhwv v � +n> o, 5 V � duh ghulyhg iurp wkrvh ri wkh uhvshfwlyh wzr er{hv iv � � j �� � �dqg iv � � j �� � � +816,1 Iru uhdo ydoxhg yduldeohv {� / v � +n> o, lv wkh vpdoohvw lqwhuydo ^w �� +n> o,> w �� +n> o,` wkdwfryhuv wkh uhvshfwlyh wzr lqwhuydov iru E � dqg E � rq {� +817,

w �� +n> o, @ plq+w �� � > w �� � ,/ w �� +n> o, @ pd{+w �� � > w �� � ,= +491:,Iru fdwhjrulfdo yduldeohv {� / wkh vxevhw v � +n> o, lv wkh xqlrq ri wkh uhvshfwlyh E � dqg E � vxevhwv

v � +n> o, @ v � �V v � � = +491;,

Wkh glvvlplodulw| phdvxuh +4918, dvvxphv ydoxhv lq wkh vhpl�rshq lqwhuydo 3 � G+E � > E � , ? 41Dowkrxjk G+E � > E � , lv qrw vwulfwo| d glvwdqfh/ lw uhwdlqv wkh hvvhqwldo surshuw| wkdw lwv ydoxh lqfuhdvhvdv wkh ydoxh vxevhwv iv � � j �� � � / iv � � j �� � � +uhdo ru fdwhjrulfdo,/ gh�qlqj wkh wzr er{hv E � > E � / lqfuhdvlqjo|gl�hu rq hdfk ri wkh lqsxw yduldeohv {� 1 Qhvwhg er{hv zloo kdyh }hur glvvlplodulw|/ dv zloo �dgmdfhqw�er{hv wkdw kdyh frqwljxrxv lqwhuydov rq rqh uhdo yduldeoh {� / dqg lghqwlfdo vxevhwv rq doo rwkhu ydul0deohv iv � � � @ v � � � j � � �� � 1 Rwkhu frq�jxudwlrqv zloo surgxfh odujhu glvvlplodulw| zlwk wkh kljkhvw ydoxhvrffxuulqj zkhq wkh wzr er{hv duh gh�qhg e| kljko| glvsdudwh ydoxh vxevhwv +818, rq rqh/ ru hvshfldoo|vhyhudo/ lqsxw yduldeohv1 Qrwh wkdw wzr er{hv fdq kdyh qrq�}hur lqwhuvhfwlrq ^�+E �

WE � , 9@ 3` dqgvwloo eh kljko| glvvlplodu/ vr orqj dv wkh| duh gh�qhg e| yhu| gl�huhqw vxevhwv rq wkh qrq�lqwhuvhfwlqjyduldeohv1 Dovr qrwh wkdw er{ glvvlplodulw| lv gh�qhg lq whupv ri wkh gdwd glvwulexwlrq1 Wzr er{hv fdqeh gh�qhg lq whupv ri yhu| gl�huhqw vhwv ri lqsxw yduldeohv dqg vwloo eh txlwh vlplodu/ li wkh wzr vhwvduh kljko| fruuhodwhg1

Glvvlplodulwlhv +4918, duh vkrzq ehwzhhq wkh wzr er{hv lqgxfhg iru hdfk pdunhwlqj gdwd h{dpsoh lqVhfwlrq 481 Iru wkh �uvw +iuhtxhqw |hu, wkh wzr er{hv duh idluo| vlplodu ^G+E � > E � , @ 3=47` lqglfdwlqjwkdw erwk er{hv uhsuhvhqw vlplodu pdunhw vhjphqwv1 Iru wkh vhfrqg h{dpsoh +kdyh d shw, wkh wzrlqgxfhg er{hv duh prghudwho| glvvlplodu ^G+E � > E � , @ 3=77`/ uhsuhvhqwlqj vrphzkdw gl�huhqw pdunhwvhjphqwv1 Wkh wzr er{hv lq wkh wklug h{dpsoh +NJR0DP, duh yhu| gl�huhqw ^G+E � > E � , @ 3=::`1 Wkh|uhsuhvhqw yhu| gl�huhqw ghprjudsklf jurxsv zkr duh lqfolqhg wr olvwhq wr NJR0DP udglr1

4:1 Vfdolqj wr yhu| odujh gdwdedvhvWkh SULP surfhgxuh lv phpru| edvhg1 Doo ri wkh gdwd +414, lv suhvxphg wr eh vwruhg lq udqgrp dffhvvphpru| +UDP,1 Zkloh wkh UDP fdsdflw| ri suhvhqw gd| frpsxwhuv lv vx!flhqw iru pdq| dssolfdwlrqv/dqg lv vwhdglo| jurzlqj/ wkhuh duh vrph gdwd plqlqj dssolfdwlrqv iru zklfk doo ri wkh uhohydqw gdwd fdquhvlgh rqo| rq vhfrqgdu| +glvn, vwrudjh1 Dowkrxjk rqh frxog hqylvlrq d glvn�edvhg lpsohphqwdwlrqri SULP/ wkhuh duh vlpsoh vwudwhjlhv wr hqdeoh lw wr sur�wdeo| xvh doo ri wklv gdwd lq d UDP edvhgfrqwh{w1

Wkh vwudwhj| hpsor|hg ghshqgv rq wkh +dssur{lpdwh, vxssruw � � +:17, uhtxluhg iru wkh lqgxfhger{hv1 Li wklv lv qrw yhu| vpdoo +� � z 3=34, wkhq d prgl�fdwlrq ri wkh expslqj vwudwhj| +Vhfwlrq 46,fdq eh hpsor|hg1 Vxffhvvlyh wudmhfwrulhv +<14, duh lqgxfhg rq lqghshqghqw udqgrpo| vhohfwhg vxevhwv riwkh hqwluh gdwd edvh1 Wkh vl}h ri hdfk vxevhw lv ghwhuplqhg e| dydlodeoh UDP1 Wkh pxowlsoh wudmhfwru|

58

Page 26: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

vwudwhj| glvfxvvhg lq Vhfwlrq 46 lv wkhq hpsor|hg wr vhohfw wkh ehvw er{ dprqj doo ri wkrvh surgxfhge| wkh lqghshqghqwo| lqgxfhg wudmhfwrulhv1 Wklv dssurdfk kdv d qdwxudo sdudooho lpsohphqwdwlrq1

Li yhu| vpdoo vxssruwv +� � � 3=34, duh ghvluhg/ wkhq d vhtxhqwldo vdpsolqj vwudwhj| fdq eh xvhg1Dv deryh/ udqgrp vxevdpsohv duh gudzq iurp wkh gdwd edvh1 Krzhyhu/ hdfk vxffhvvlyh vxevdpsoh lvuhtxluhg wr eh lqvlgh wkh er{ lqgxfhg iurp wkh suhylrxv vxevdpsoh1 SULP lv dssolhg wr hdfk vxffhvvlyhvdpsoh zlwk � � fkrvhq wr eh uhodwlyho| odujh +� � z 3=34,1 Dsso|lqj wklv surfhgxuh p wlphv surgxfhver{hv zlwk vxssruw � � 1 Iru odujh hqrxjk gdwd edvhv wklv vhtxhqwldo vwudwhj| fdq eh frxsohg zlwk wkhsdudooho rqh glvfxvvhg deryh1 Wkhvh UDP edvhg dssurdfkhv suhvxph d gdwd edvh lpsohphqwdwlrq wkdwvxssruwv udslg vhohfwlrq ri udqgrp vxevhwv1

4;1 FrpshwlwruvUxoh lqgxfwlrq kdv d orqj klvwru| lq wkh pdfklqh ohduqlqj dqg vwdwlvwlfv olwhudwxuhv1 Dowkrxjk qrwvshfl�fdoo| dlphg dw ixqfwlrq rswlpl}dwlrq/ pdq| h{lvwlqj surfhgxuhv fdq eh dssolhg lq wkdw frqwh{w/dw ohdvw lq vshfldo fdvhv1 Lq wklv vhfwlrq zh txdolwdwlyho| frpsduh SULP wr vrph ri wkh pruh srsxoduuxoh lqgxfwlrq phwkrgv1 D txdqwlwdwlyh frpsdulvrq zlwk rqh +FDUW, lv surylghg lq Vhfwlrq 4<1

4;141 Fryhulqj dojrulwkpvZkhq wkh rxwsxw yduldeoh | dvvxphv rqo| wzr ydoxhv +h1j1 | 5 i3> 4j,/ SULP fdq eh ylhzhg dv dqlqgxfwlyh ohduqlqj phwkrg zkhuh wkh wzr ydoxhv uhvshfwlyho| uhsuhvhqw qhjdwlyh dqg srvlwlyh lqvwdqfhvri d wdujhw frqfhsw wr eh ohduqhg1 Lq wklv frqwh{w SULP vkduhv pdq| ri wkh fkdudfwhulvwlfv ri rwkhupdfklqh ohduqlqj dojrulwkpv wkdw ohduq glvmxqfwlyh vhwv ri sursrvlwlrqdo uxohv wkurxjk vhtxhqwldo fry0hulqj1 Wkhvh lqfoxgh FQ5 ^Fodun dqg Qleohww +4<;<,` dqg sursrvlwlrqdo yhuvlrqv ri IRLO ^Txlqodq+4<<3,` vxfk dv ULSSHU ^Frkhq +4<<8,`1 Wkh sulqflsdo gl�huhqfh lv wkh vhdufk vwudwhj| xvhg iru h{0dplqlqj wkh vsdfh ri srvvleoh suhfrqglwlrqv wr frqvwuxfw hdfk uxoh +er{,1 Wkhvh rwkhu surfhgxuhv dooxvh yhu| juhhg| vwudwhjlhv/ hvshfldoo| zlwk fdwhjrulfdo yduldeohv1 Dv glvfxvvhg lq Vhfwlrq ;14 wklv fdqolplw shuirupdqfh lq pdq| vlwxdwlrqv dqg lv wkh sulpdu| prwlydwlrq iru wkh ghyhorsphqw ri wkh sdwlhqwvwudwhj| hpsor|hg e| SULP1 D ohvv lpsruwdqw gl�huhqfh lv lq wkh uhsuhvhqwdwlrq ri wkh lqgxfhg uxohv1Hdfk er{ lqgxfhg e| SULP +818, fdq lqyroyh lpsolflw glvmxqfwlrqv

{ � 5 v � � @ ^

��� � � ���+{� @ } � ,=

Prvw rwkhu phwkrgv surgxfh sxuho| frqmxqfwlyh uxohv lqyroylqj rqo| htxdolw| frqvwudlqwv rq wkh ydoxhvri hdfk fdwhjrulfdo yduldeoh1 Rwkhu gl�huhqfhv lqfoxgh wkh vwudwhj| iru plvvlqj ydoxhv +Vhfwlrq 45, dqgwkh xvh ri pxowlsoh wudmhfwrulhv +Vhfwlrq 46, lqvwhdg ri h{solflw edfnwudfnlqj14;151 Ghflvlrq wuhh lqgxfwlrqLq whupv ri dfwxdo dssolfdwlrq/ surfhgxuhv prvw frpshwlwlyh zlwk SULP duh olnho| wr eh ghflvlrqwuhh lqgxfwlrq whfkqltxhv vxfk dv FDUW ^Euhlpdq hw do1 +4<;7,` dqg F718 ^Txlqodq +4<<7,`1 Wkhvhphwkrgv surgxfh d vhw ri glvmrlqw uxohv wkdw froohfwlyho| fryhu wkh hqwluh lqsxw vsdfh wkurxjk uhfxuvlyhsduwlwlrqlqj1 Hdfk vxffhvvlyh sduwlwlrq +�vsolw�, lv lqgxfhg e| d frqglwlrq { � 5 v � � vr wkdw hdfk uxohlq wkh �qdo fryhulqj vhw +whuplqdo qrghv, wdnhv wkh frqmxqfwlyh irup jlyhq lq +818,1 Dowkrxjk wkh jrdori wkhvh surfhgxuhv lv dffxudf| ri dssur{lpdwlrq hyhu|zkhuh lq wkh lqsxw vsdfh/ udwkhu wkdq h{solflwrswlpl}dwlrq/ rqh fdq h{dplqh wkh uxohv dvvrfldwhg zlwk wkh kljkhvw suhglfwhg rxwsxw ydoxhv1 Wkhvhfdq eh lqwhusuhwhg lq dq dqdorjrxv pdqqhu wr wkrvh lqgxfhg e| SULP1

Wkh pdmru gl�huhqfhv ehwzhhq SULP dqg ghflvlrq wuhh lqgxfwlrq phwkrgv duh wkh xvh ri fryhulqjudwkhu wkdq sduwlwlrqlqj wr surgxfh uxoh vhwv/ dqg sdwlhqw udwkhu wkdq juhhg| vwudwhjlhv wr lqgxfh wkhlqglylgxdo uxohv1 Wkh odwwhu lvvxh lv glvfxvvhg lq Vhfwlrq ;1 Wkh uhodwlyh phulwv ri fryhulqj yhuvxv sdu0wlwlrqlqj duh lvvxhv ri ghedwh lq wkh pdfklqh ohduqlqj olwhudwxuh ^vhh Plwfkhoo +4<<:,`1 Uxohv surgxfhge| fryhulqj duh pruh �h{suhvvlyh� wkdq wkrvh surgxfhg e| sduwlwlrqlqj vlqfh wkh| duh lqgxfhg lqgh0shqghqwo| ri hdfk rwkhu1 Sduwlwlrqlqj uxohv duh irufhg wr vkduh pdq| frpprq frqmxqfwlyh vwdwhphqwv

59

Page 27: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

{ � 5 v � � 1 Wkxv/ fryhulqj whqgv wr surgxfh ihzhu uxohv/ hdfk ri d vlpsohu qdwxuh +ihzhu frqmxqfwlrqv,dqg duh wkhuhe| pruh lqwhusuhwdeoh1 Wklv kdv prwlydwhg srvw0surfhvvlqj vwudwhjlhv vxfk dv F718uxohv^Txlqodq +4<<7,/ +4<<8,` wkdw dwwhpsw wr vlpsoli| ghflvlrq wuhh uxoh vhwv1 Iru juhhg| vwudwhjlhv lw lvqrw fohdu zkhwkhu wkh lqfuhdvhg h{suhvvlyhqhvv ri fryhulqj uxohv wudqvodwhv wr kljkhu dffxudf|> wklv lvolnho| wr eh sureohp ghshqghqw1 Lq wkh fdvh ri sdwlhqw uxoh lqgxfwlrq/ fryhulqj vhhpv wr eh wkh pruhqdwxudo dowhuqdwlyh> d vwudljkwiruzdug sdwlhqw lpsohphqwdwlrq edvhg rq sduwlwlrqlqj grhv qrw dsshduwr eh reylrxv1

4;161 Xvhu lqyroyhphqwD pdmru gl�huhqfh ehwzhhq SULP dqg lwv suhghfhvvruv lv wkh lqyroyhphqw ri xvhu mxgjhphqw dv dqlqwhjudo sduw ri wkh uxoh lqgxfwlrq surfhvv +Vhfwlrqv </ 43/ 46 dqg 4914,1 Wkh sxusrvh ri wkh h{huflvhlv suhvxphg wr eh ghvfulswlyh gdwd dqdo|vlv zkhuh vshfl�f jrdov duh kljko| sureohp ghshqghqw1 WkhSULP surfhgxuh surylghv qdwxudo lqwhuidfhv iru xvhu jxlgdqfh wrzdugv lqglylgxdo jrdov/ dv zhoo dv irulqfrusrudwlqj grpdlq nqrzohgjh wr lpsuryh dffxudf|1 Wkh xvhu lv jlyhq wkh rssruwxqlw| wr fxvwrpl}hlqglylgxdo uxohv wr surgxfh wkh vpdoohvw vlpsohvw vhw frqvlvwhqw zlwk surjudppdwlf qhhgv1 Wklv lvlq frqwudvw zlwk rwkhu uxoh lqgxfwlrq phwkrgv zkhuh doo vxfk wudgh�r�v duh dxwrpdwlfdoo| pdgh e|wkuhvkroglqj pdwkhpdwlfdo fulwhuld1 Lq wkhvh +dxwrpdwlf, vhwwlqjv wkh xvhu dvvxphv d sdvvlyh uroh rivlpso| lqvshfwlqj wkh surgxfhg rxwsxw1

4<1 SULP yhuvxv FDUWDprqj wkh frpshwlwruv wr SULP wkh rqh xvlqj wkh ohdvw juhhg| vwudwhj| lv FDUW ^Euhlpdq hw do1+4<;7,`1 Rq dyhudjh/ hdfk frqmxqfwlyh frqvwudlqw { � 5 v � � gh�qlqj wkh �qdo uxoh vhw +818, +whuplqdoqrghv, vhohfwv rqh kdoi ri wkh gdwd +elqdu| vsolw, dw hdfk lwhudwlyh vwhs/ iru erwk uhdo dqg fdwhjrulfdoyduldeohv1 Wklv fdq eh ylhzhg dv d �vhpl�0juhhg| vwudwhj| zkhq frpsduhg wr prvw rwkhu phwkrgvwkdw iudjphqw wkh gdwd pxfk pruh vhyhuho|1 Dovr/ FDUW surgxfhv uxohv wkdw duh lghqwlfdo lq irup+818, wr wkrvh ri SULP/ dqg lw lv gluhfwo| htxlsshg wr kdqgoh uhdo ydoxhg rxwsxw yduldeohv +uhjuhvvlrqwuhhv,1 Lq wklv vhfwlrq zh frpsduh wkh shuirupdqfh ri FDUW dqg SULP rq vhyhudo gdwd vhwv iurp wkhshuvshfwlyh ri ixqfwlrq pd{lpl}dwlrq1 Wkh �uvw lv v|qwkhwlf gdwd gholehudwho| ghvljqhg wr looxvwudwhwkh dgydqwdjh ri wkh sdwlhqw vwudwhj| hpsor|hg e| SULP1 Wkh rwkhuv duh wkh wzr jhrorj| dqg wkuhhpdunhwlqj h{dpsohv glvfxvvhg uhvshfwlyho| lq Vhfwlrqv 44 dqg 481

Dv zlwk SULP*v rwkhu frpshwlwruv/ FDUW r�huv qr xvhu frqwuro ryhu wkh phdq�vxssruw wudgh0r�ri wkh uxohv lw surgxfhv1 Wkhuhiruh/ wr pdnh wkh uhvxowv ri wkh wzr surfhgxuhv frpsdudeoh wkh iroorzlqjsurfhgxuh zdv hpsor|hg1 FDUW zdv �uvw dssolhg wr wkh gdwd vhw1 Lwv ehvw M uxohv +whuplqdo qrghv,lq whupv ri kljkhvw suhglfwhg rxwsxw ydoxh zhuh lghqwl�hg1 SULP zdv wkhq dssolhg wr wkh vdph gdwdvhw wr vhtxhqwldoo| lqgxfh M er{hv wkurxjk wkh fryhulqj whfkqltxh +Vhfwlrq 9,1 Wkh phdq�vxssruwwudgh�r� ri hdfk er{ zdv fkrvhq wr pdwfk hlwkhu wkh phdq ru vxssruw ri wkh fruuhvsrqglqj FDUWuxoh/ ghshqglqj rq zklfk rqh frxog eh pdwfkhg prvw forvho|1 Wkh uhodwlyh srzhu ri wkh uhvshfwlyhuxohv fdq wkhq eh frpsduhg lq whupv ri wkh fruuhvsrqglqj xqpdwfkhg txdqwlw|1 Li wkh vxssruwv duhvlplodu wkhq wkh rqh zlwk wkh kljkhu phdq lv ehwwhu> li wkhlu phdqv duh vlplodu wkh rqh zlwk wkh odujhuvxssruw lv ehwwhu1 Lq doo fdvhv wkh gdwd zdv glylghg lqwr wkh vdph ohduqlqj dqg whvw vhwv +Vhfwlrq <,iru erwk surfhgxuhv1 Wkh shholqj iudfwlrq � +:18, zdv wdnhq wr eh 43( +� @ 3=4, iru SULP1

4<141 V|qwkhwlf h{dpsohWkh v|qwkhwlf gdwd frqvlvwhg ri Q @ 43333 revhuydwlrqv zlwk q @ 43 lqsxw yduldeohv udqgrpo| gudzqiurp d xqlirup glvwulexwlrq i{ � � X ^�4> 4`j � �� 1 Wkh wdujhw ixqfwlrq zdv wdnhq wr eh

| @ i+{, @�

\

� � �{ � = +4<14,

5:

Page 28: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

Wklv wdujhw kdv 5 �� � pd{lpd ri htxdo ydoxh1 Khuh zh looxvwudwh zlwk M @ 6 vr wkdw wkhuh duh 7 pd{lpd

dw dowhuqdwlqj fruqhuv ri wkh fxeh gh�qhg e| wkh �uvw wkuhh lqsxw yduldeohv1 Wdeoh 4<14 frpsduhv wkhshuirupdqfh ri FDUW dqg SULP rq wklv sureohp1

Wdeoh 4<14Shuirupdqfh rq wkh v|qwkhwlf gdwd h{dpsoh1

FDUW SULPn �| � � � +(,4 3=5< 4=75 3=57 4=66 3=4< 4=77 3=47 4=<

n �| � � � +(,4 3=5; 7=55 3=64 6=66 3=65 6=87 3=67 6=4

Fryhudjh udwlr @ 6=<

Wkh ohiw vxewdeoh vkrzv wkh phdq �| � dqg vxssruw � � +�433, iru wkh ehvw irxu FDUW uxohv dvdyhudjhg ryhu whq +udqgrpo|, uhsolfdwhg gdwd vhwv1 Wkh uljkw vxewdeoh vkrzv wkhvh txdqwlwlhv iru wkhfruuhvsrqglqj SULP uxohv dyhudjhg ryhu wkh vdph vhw ri uhsolfdwlrqv1 Wkh txdqwlw| �fryhudjh udwlr�lv lqwhqghg dv d vlqjoh vxppdu| phdvxuh ri uhodwlyh shuirupdqfh1 Lw lv wkh udwlr ri wkh �fryhudjh�

F @�

[

� � �+�| � � �|, � � � +4<15,

ri SULP wr wkdw ri FDUW1 Khuh �| lv wkh joredo phdq +41;,1 Lq wklv vhqvh +4<15, SULP*v fryhudjh lv+rq dyhudjh, doprvw irxu wlphv wkdw ri FDUW iru wklv sureohp1

Wklv h{dpsoh +4<14, lv lqwhqghg wr hpxodwh vlwxdwlrqv lq zklfk wkh lqlwldo vhw ri vsolwv +FDUW, rushhov +SULP, duh qrw zhoo gh�qhg1 Lq wklv fdvh wkh wdujhw ixqfwlrq surylghv qr lqirupdwlrq rq krzwr fkrrvh +dw plqlpxp, wkh �uvw wzr1 Vsolwv2shhov duh vhohfwhg udqgrpo| xqwlo wzr ri wkh �uvw wkuhhlqsxwv kdyh ehhq vhohfwhg1 Iurp wkdw srlqw rq wkhuh lv lqirupdwlrq wr gluhfw ixuwkhu vsolwwlqj2shholqj1Khuh SULP gudpdwlfdoo| rxw�shuirupv FDUW rzlqj wr lwv sdwlhqw vwudwhj|1 Ehfdxvh hdfk shho uhpryhvrqo| 43( ri wkh gdwd/ wkhuh whqgv wr eh pxfk pruh gdwd uhpdlqlqj zkhq wkh wdujhw ixqfwlrq vwuxfwxuhlv �uhyhdohg�/ doorzlqj vxevhtxhqw shhov wr lqfuhdvh wdujhw phdq1 Zlwk FDUW wkhuh lv w|slfdoo| pxfkohvv gdwd ohiw dw wkh fruuhvsrqglqj srlqw dorqj hdfk eudqfk ri wkh ghflvlrq wuhh1

SULP irxqg hdfk ri wkh wdujhw*v irxu pd{lpd zlwk lwv �uvw irxu uxohv rq doo whq wuldov1 Wkh uhdvrqzk| odwhu uxohv gr ehwwhu wkdq hduolhu rqhv khuh lv d frqvhtxhqfh ri wkh fryhulqj vwudwhj|1 Uhprylqjhdfk vxffhvvlyh pd{lpxp uhgxfhv wkh wdujhw v|pphwu| ryhu wkh uhpdlqlqj gdwd1 Wklv doorzv lwvvwuxfwxuh wr eh xqfryhuhg hduolhu lq wkh shholqj vhtxhqfh1

Wkh gl!fxow| ri wklv sureohp uhsuhvhqwv d olplw iru FDUW1 Pdnlqj wkh sureohp pruh gl!fxow e|lqfuhdvlqj wkh qxpehu ri lqsxwv wr q @ 53/ ru uhgxflqj wkh vdpsoh vl}h wr Q @ 4333/ ru vhwwlqj M @ 7lq +4<14, wr surgxfh hljkw pd{lpd/ fdxvhv FDUW qrw wr surgxfh d wuhh1 Lw idlov wr ghwhfw vx!flhqwvwuxfwxuh lq wkh wdujhw ixqfwlrq dqg lwv suxqlqj dojrulwkp vxevhtxhqwo| uhpryhv doo vsolwv1 Lq doo riwkhvh pruh gl!fxow fdvhv SULP frqwlqxhg wr �qg doo wkh wdujhw pd{lpd zlwk lwv �uvw 5 �

� � er{hv1

4<151 Gdwd h{dpsohvDv qrwhg/ wkh v|qwkhwlf h{dpsoh +4<14, zdv vshfl�fdoo| frqwulyhg wr h{sorlw SULP*v dgydqwdjhv1 Rqhzrxog qrw h{shfw vxfk odujh lpsuryhphqwv ryhu FDUW lq prvw vlwxdwlrqv1 Wdeohv 4<15 0 4<19 vkrzfruuhvsrqglqj uhvxowv wr wkrvh ri Wdeoh 4<14 +wkuhh uxohv rqo|, iru wkh wzr jhrorj| h{dpsohv lq Vhfwlrq44 dqg wkh wkuhh pdunhwlqj h{dpsohv lq Vhfwlrq 48 uhvshfwlyho|1

5;

Page 29: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

Wdeoh 4<15Shuirupdqfh rq wkh jhrorj| h{dpsoh/ dqflhqw vwdeoh vklhogv1

FDUW SULPn �| � � � +(,4 3=<4 44=35 3=;: 8=66 3=:< 7=6

n �| � � � +(,4 3=<7 44=35 3=;: 9=36 3=;3 9=9

Fryhudjh udwlr @ 4=53Wdeoh 4<16

Shuirupdqfh rq wkh jhrorj| h{dpsoh/ |rxqj rurjhqlf ehowv1FDUW SULPn �| � � � +(,

4 3=97 5=45 3=45 6=96 3=43 5=<

n �| � � � +(,4 3=9: 5=55 3=58 6=86 3=3< 6=7

Fryhudjh udwlr @ 4=67

Wdeoh 4<17Shuirupdqfh rq wkh pdunhwlqj h{dpsoh/ qxpehu ri ljkwv shu |hdu1

FDUW SULPn �| � � � +(,4 8=3 4=<5 6=9 5=86 5=8 5=:

n �| � � � +(,4 8=4 5=65 6=9 8=46 6=3 8=9

Fryhudjh udwlr @ 4=;:Wdeoh 4<18

Shuirupdqfh rq wkh pdunhwlqj h{dpsoh/ kdyh d shw1FDUW SULPn �| � � � +(,

4 3=;4 7=85 3=:8 49=36 3=:4 5=7

n �| � � � +(,4 3=;7 :=;5 3=:5 55=:6 3=9< 5=6

Fryhudjh udwlr @ 4=6;Wdeoh 4<19

Shuirupdqfh rq wkh pdunhwlqj h{dpsoh/ olvwhq wr NJR0DP1FDUW SULPE �| � � � (

4 3=58 9=45 3=4: :=<6 3=49 8=;

E �| � � � (4 3=56 47=35 3=4: 44=36 3=46 43=3

Fryhudjh udwlr @ 4=77

Wkh uxohv lqgxfhg e| SULP duh vhhq wr eh hlwkhu frpsdudeoh wr ru ehwwhu wkdq wkrvh ri FDUW/vrphwlphv vxevwdqwldoo| vr1 Wklv lv hvshfldoo| wkh fdvh iru wkh pdunhwlqj gdwd1 Wkh fryhudjh udwlr

5<

Page 30: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

dyhudjhg ryhu wkhvh �yh h{dpsohv lv 4=781 SULP*v fryhudjh lv rq dyhudjh 78( kljkhu wkdq wkdw riFDUW1

Lq doo exw rqh h{dpsoh +Wdeoh 4<16, FDUW surgxfhg yhu| odujh wuhhv +83 0 533 whuplqdo qrghv,1Prvw ri lwv uxohv zhuh idu pruh frpsoh{ wkdq wkrvh surgxfhg e| SULP1 Whfkqltxhv dqdorjrxv wrF718uxohv ^Txlqodq +4<<7,/ +4<<8,` pljkw khos vlpsoli| wkh FDUW uxohv lq wklv frqwh{w1 FDUW zdvxvhg zlwk doo sdudphwhu vhwwlqjv dw wkhlu ghidxow ydoxhv1 Wkhvh duh suhvxpdeo| wxqhg iru uhjuhvvlrqdqdo|vlv zkhuh wkh jrdo lv wdujhw ixqfwlrq dssur{lpdwlrq ryhu wkh hqwluh lqsxw vsdfh1 Lw lv frqfhlydeohwkdw wkhuh duh rwkhu vhwwlqjv wkdw duh pruh dssursuldwh iru rswlpl}dwlrq1 Ilqdoo|/ lw vkrxog eh qrwhgwkdw wkh uxohv vxppdul}hg lq Wdeohv 4<15 0 4<19 duh jhqhudoo| qrw wkh vdph � qru dv jrrg � dv wkhfruuhvsrqglqj rqhv suhvhqwhg lq Vhfwlrqv 44 dqg 481 Wkh phdq0vxssruw wudgh0r�v zhuh vhohfwhg khuh wrdoljq zlwk wkrvh surgxfhg e| FDUW/ udwkhu wkdq wr surgxfh wkh prvw idyrudeoh mrlqw +phdq�vxssruw,ydoxhv1

531 Vxppdu|SULP lv lqwhqghg dv dq dgglwlrq wr wkh gdwd dqdo|vw*v wrro nlw/ wr eh xvhg zkhq wkh jrdo lv hlwkhu h{solflw+Vhfwlrq 6, ru lpsolflw +Vhfwlrq 7, rswlpl}dwlrq1 Lwv glvwlqjxlvklqj fkdudfwhulvwlfv lqfoxgh sdwlhqwshholqj2sdvwlqj +Vhfwlrqv : dqg ;, frxsohg zlwk pxowlsoh wudmhfwrulhv +Vhfwlrqv 46 dqg 47, wr hqkdqfhsrzhu dqg vwdelolw|/ dqg lqwlpdwh xvhu lqyroyhphqw lq wkh prgho vhohfwlrq surfhvv +Vhfwlrqv </ 43/ 46dqg 4714,1 Lw whqgv wr surgxfh sduvlprqlrxv lqwhusuhwdeoh ghvfulswlrqv ri wkh vwuxfwxuh lw xqfryhuv+Vhfwlrqv 44/ 48/ dqg 49,/ dqg rq wkh sureohpv frqvlghuhg khuh +Vhfwlrq 4<, h{klelwv shuirupdqfhvxshulru wr frpsdudeoh surfhgxuhv vxfk dv FDUW wkdw duh lqwhqghg iru ixqfwlrq dssur{lpdwlrq1 Wkhh{whqw wr zklfk wklv shuirupdqfh jdlq jhqhudol}hv wr rwkhu vlwxdwlrqv uhpdlqv wr eh hvwdeolvkhg1 Dvzlwk doo ohduqlqj surfhgxuhv/ uhodwlyh shuirupdqfh zloo olnho| eh sureohp ghshqghqw1

541 DfnqrzohgjphqwvZh wkdqn Eloo Jul!q dqg Lpsdfw Uhvrxufhv/ Lqf1 iru surylglqj uhvshfwlyho| wkh jhrorj| dqg pdunhwlqjgdwd edvhv xvhg iru looxvwudwlrq1 Zh hvshfldoo| dfnqrzohgjh wkh frqwulexwlrq ri Duw Rzhq zkr vxjjhvwhgwkh vshfl�f pxowlsoh wudmhfwru| +�wklqqlqj�, vwudwhj| ghvfulehg lq Vhfwlrq 461

Wkh zrun ri Mhurph K1 Iulhgpdq zdv sduwldoo| vxssruwhg e| wkh Ghsduwphqw ri Hqhuj| xqghufrqwudfw GH�DF36�:9VI33848/ dqg e| wkh Qdwlrqdo Vflhqfh Irxqgdwlrq xqghu judqwv GPV�<736;37dqg GPV�<:377641 Wklv uhvhdufk zdv dovr vxssruwhg lq sduw e| wkh Dxvwudoldq Jryhuqphqw Lqwhuqd0wlrqdo Vflhqfh ) Whfkqrorj| Judqw iru uhvhdufk frooderudwlrq= Vwdwlvwlfdo Prghoolqj dqg Suhglfwlrq riwkh Vwuxfwxuh ri wkh Hduwk*v Xsshu Pdqwoh= d Iudphzrun iru Plqhudo H{sorudwlrq1

Uhihuhqfhv^4` Eduqhww/ Y1 +4<:9,1 Wkh rughulqj ri pxowlyduldwh gdwd +zlwk glvfxvvlrq,1 M1 Ur|1 Vwdwlvw1 Vrf1 D

46</ 64;06871^5` Elvkrs/ F1 P1 +4<<8,1 Qhxudo Qhwzrunv iru Sdwwhuq Uhfrjqlwlrq1 R{irug Xqlyhuvlw| Suhvv1^6` Euhlpdq/ O1 +4<<9,1 Edjjlqj suhglfwruv1 Pdfklqh Ohduqlqj 57/ 45604731^7` Euhlpdq/ O1/ Iulhgpdq/ M1 K1/ Rovkhq/ U1 D1/ ) Vwrqh/ F1 M1 +4<;7,1 Fodvvl�fdwlrq dqg Uhjuhvvlrq

Wuhhv1 Zdgvzruwk1^8` Fodun/ S1 ) Qleohww/ U1 +4<;<,1 Wkh FQ5 lqgxfwlrq dojrulwkp1 Pdfklqh Ohduqlqj 6/ 59405;71^9` Frkhq Z1 Z1 +4<<8,1 Idvw h!flhqw uxoh lqgxfwlrq1 Lq Pdfklqh Ohduqlqj= Surfhhglqjv ri wkh

Wzhoiwk Lqwhuqdwlrqdo Frqihuhqfh/ odnh Wdkrh/ FD +4480456,1 Prujdq Ndxipdqq1^:` Grqrkr/ G1 dqg Jdvnr/ P1 +4<<5,1 Euhdngrzq surshuwlhv ri orfdwlrq hvwlpdwhv edvhg rq kdoivsdfh

ghswk dqg surmhfwhg rxwo|lqjqhvv1 Dqqdov ri Vwdwlvwlfv 53/ 4;3604;5:1

63

Page 31: ExpsKxqwlqjlqKljk˘GlphqvlrqdoGdwdstatweb.stanford.edu/~jhf/ftp/prim.pdf91 Fryhulqj Iurp +516, +814, rqh vhhv wkdw wkh jrdo ri wkh rswlpl}dwlrq surfhgxuh lv wr lqgxfh d vhw ri er{hv

^;` Hiurq/ E1 ) Wlevkludql/ U1 M1 +4<<6,1 Dq Lqwurgxfwlrq wr wkh Errwvwuds1 Fkdspdq ) Kdoo1^<` Iulhgpdq/ M1 K1 +4<<:,1 Rq eldv/ yduldqfh/ 3240orvv/ dqg wkh fxuvh0ri0glphqvlrqdolw|1 Gdwd Plqlqj

dqg Nqrzohgjh Glvfryhu| 4/ 880::1^43` Juhhq/ S1 M1 +4<;4,1 Shholqj elyduldwh gdwd1 Lq Lqwhusuhwlqj Pxowlyduldwh Gdwd +Y1 Eduqhww/ hg1,

Zloh|1^44` Jul!q/ Z1 O1/ Ilvkhu/ Q1 L1/ Iulhgpdq/ M1 K1/ U|dq/ F1 J1/ dqg R*Uhloo|/ V1 +4<<:,1 Fu0S|ursh

jduqhwv lq olwkrvskhulf pdqwoh1 M1 Shwurorj| +vxeplwwhg,1^45` Kdoo/ S1 +4<;<,1 Rq surmhfwlrq sxuvxlw uhjuhvvlrq1 Dqqdov ri Vwdwlvwlfv 4:/ 8:608;;1^46` Oruhqw}/ J1 J1 +4<;9,1 Dssur{lpdwlrq ri Ixqfwlrqv1 Fkhovhd1^47` Plwfkhoo/ W1 P1 +4<<:,1 Pdfklqh Ohduqlqj1 PfJudz0Kloo1^48` Txlqodq/ M1 U1 +4<<3,1 Ohduqlqj orjlfdo gh�qlwlrqv iurp uhodwlrqv1 Pdfklqh Ohduqlqj 8/ 56<05991^49` Txlqodq/ M1 U1 +4<<7,1 F718= Surjudpv iru Pdfklqh Ohduqlqj1 Prujdq0Ndxipdqq1^4:` Txlqodq/ M1 U1 +4<<8,1 PGO dqg fdwhjrulfdo wkhrulhv +frqwlqxhg,1 Lq Pdfklqh Ohduqlqj= Surfhhg0

lqjv ri wkh Wzhoiwk Lqwhuqdwlrqdo Frqihuhqfh/ odnh Wdkrh/ FD +79707:3,1 Prujdq Ndxipdqq1^4;` Ulsoh|/ E1 G1 +4<<9, Sdwwhuq Uhfrjqlwlrq dqg Qhxudo Qhwzrunv1 Fdpeulgjh Xqlyhuvlw| Suhvv1^4<` Ulyhvw/ U1 O1 +4<;:,1 Ohduqlqj ghflvlrq olvwv1 Pdfklqh Ohduqlqj 5/ 55<05791^53` Wlevkludql/ U1 M1 dqg Nqljkw/ N1 +4<<8,1 Prgho vhdufk dqg lqihuhqfh e| errwvwuds �expslqj�1

Whfkqlfdo Uhsruw/ Xqlyhuvlw| ri Wrurqwr1^54` Ydsqln/ Y1 +4<<8,1 Wkh Qdwxuh ri Vwdwlvwlfdo Ohduqlqj Wkhru|1 Vsulqjhu1^55` Zdked/ J1 +4<<3,1 Vsolqh Prghov iru Revhuydwlrqdo Gdwd1 VLDP1

64