1

TRNG I HC KHOA HC T NHIN

CAO HC CNG NGH THNG TIN KHA 22

MY HC

BO CO:

SUPPORT VECTOR MACHINE GVHD:

TS. Trn Thi Sn

HVTH:

12 11 027 Nguyn Thanh Hng

12 11 011 Bi Th Danh

12 11 075 V Quang Trng

12 11 069 Bnh Tr Thnh

12 11 024 Phm Minh Hong

- TP. HCM 2/2013 -

2

Mc lc

1 Gii thiu ............................................................................................................... 3

2 Support Vector Classifier - SVC ............................................................................ 3

2.1 Phn lp nh phn vi SVC ............................................................................. 3

2.2 Vn d liu khng phn tch tuyn tnh ..................................................... 6

2.2.1 Soft margin ............................................................................................... 7

2.2.2 Th thut Kernel ....................................................................................... 9

2.3 Cc phng php hun luyn SVC ............................................................... 13

2.3.1 Phn on (Chunking) ............................................................................ 13

2.3.2 Phng php ca Osuna ......................................................................... 14

2.3.3 SMO Sequential minimal optimization ............................................... 14

2.4 Cc hng pht trin ..................................................................................... 15

2.4.1 Hiu qu tnh ton .................................................................................. 15

2.4.2 La chn kernel ...................................................................................... 15

2.4.3 Phn tch tng qut ................................................................................. 16

2.4.4 Hc SVM c cu trc ............................................................................. 17

3 Support Vector Regressor SVR ........................................................................ 18

3.1 Gii thiu bi ton hi quy ............................................................................ 18

3.2 Hi quy vi SVR ........................................................................................... 21

3.3 Support Vector Regression ..................................................................... 27

4 Ph lc .................................................................................................................. 30

5 Ti liu tham kho................................................................................................ 30

3

1 GII THIU

Support Vector Machine (SVM) l phng php mnh v chnh xc nht trong s cc

thut ton ni bt lnh vc khai thc d liu. SVM bao gm hai ni dung chnh l:

support vector classifier (SVC), b phn lp da theo vector h tr, v support vector

regressor (SVR), b hi quy da theo vector h tr. c pht trin u tin bi Vapnik

vo nhng nm 1990, SVM c nn tng l thuyt c xy dng trn nn mng l thuyt

xc sut thng k. N yu cu s lng mu hun luyn khng nhiu v thng khng

nhy cm vi s chiu ca d liu. Trong nhng thp nin qua, SVM pht trin nhanh

chng c v l thuyt ln thc nghim.

Trong cc phn tip theo sau y, nhm s trnh by chi tit v Support Vector

Classifier v Support Vector Regressor.

2 SUPPORT VECTOR CLASSIFIER - SVC

2.1 PHN LP NH PHN VI SVC

Xt mt v d ca bi ton phn lp nh hnh v, ta phi tm mt ng thng

sao cho bn tri n ton l cc im , bn phi n ton l cc im xanh. Bi ton m

dng ng thng phn chia ny c gi l phn lp tuyn tnh (linear

classification).

Hnh 1: Minh ha phn lp tuyn tnh

Hm tuyn tnh phn bit hai lp nh sau:

( ) (1)

Trong :

l vector trng s hay vector chun ca siu phng phn cch, T l k hiu chuyn v.

l lch

Lu rng, nu khng gian l 2 chiu th ng phn cch l ng thng, nhng

trong khng gian a chiu th gi l siu phng.

4

Tp d liu u vo gm N mu input vector {x1, x2,...,xn}, vi cc gi tr nhn

tng ng l {t1,,tn} trong * +. Gi s tp d liu c th phn tch tuyn tnh hon ton, ngha l cc mu u c phn ng lp bi ng phn cch. Khi ,

gi tr tham s w v b theo (1) lun tn ti v tha ( ) cho nhng im c nhn v ( ) cho nhng im c , v th m ( ) cho mi im d liu hun luyn.

tm ng phn cch, SVC thng qua khi nim gi l l, ng bin (margin).

L l khong cch nh nht gia im d liu gn nht n mt im bt k trn ng

phn cch, xem hnh HNH 2.

Hnh 2: Minh ha margin (l)

Theo SVC, ng phn cch tt nht l ng c margin ln nht. iu ny c ngha

l tn ti rt nhiu ng phn cch xoay theo cc phng khc nhau, v khi phng

php s chn ra ng phn cch m c margin ln nht.

Hnh 3: Minh ha ng phn cch ti u

Khong cch t im d liu n ng phn cch nh sau:

| ( )|

(2)

Khng mt tnh tng qut, Vapnik xp x bi ton thnh:

5

{

(3)

Cc im d liu lm cho du = xy ra trong biu thc trn c gi l cc vector

h tr (support vector). Chng cng chnh l cc im d liu gn ng phn cch ti

u nht. Theo , khong cch t cc support vector n mt phn cch ti u s l:

( )

{

(4)

Khi , l phn cch gia hai lp l

(5)

tm c ng phn cch ti u, SVC c gng cc i theo w v b:

( )

(6)

iu ny tng ng vi:

( )

(7)

y c xem l bi ton c s (primal problem). gii quyt bi ton ny, ngi

ta dng phng php nhn t Lagrange (Lagrange multiplier). Hm Lagrange tng ng

cho (7) l:

( )

, (

) -

(8)

Ly o hm L theo hai bin w v b, ta c

{

( )

( )

(9)

Suy ra:

6

{

(10)

Th vo hm Lagrange, thu c:

( )

(11)

iu kin b sung Karush-Kuhn-Tucker (KKT) l:

, ( ) - (12)

Theo , ch nhng support vector (xi, yi) mi c i tng ng khc khng, nhng

im d liu cn li c i bng 0. Support vector chnh l ci m ta quan tm trong qu

trnh hun luyn ca SVM. Vic phn lp cho mt im d liu mi s ch ph thuc vo

cc support vector.

Bi ton kp (dual problem) (11) l lp bi ton ti u quy hoch bc 2 li (convex

quadratic programming optimization) tiu biu. Trong nhiu trng hp, n c th t ti

u ton cc khi p dng cc thut ton ti u ph hp, v d SMO (sequential minimal

optimization). Chi tit SMO s c trnh by phn sau.

Sau khi tm c cc nhn t Lagrange ti u i th chng ta c th tnh w v b ti u

theo cng thc bn di. Lu vi b th ch cn ly mt vector h tr dng (tc t = +1)

l c, nhng m bo tnh n nh ca b, chng ta c th tnh bng cch ly gi tr

trung bnh da trn cc support vector.

(13)

2.2 VN D LIU KHNG PHN TCH TUYN TNH

Vic yu cu d liu phi phn tch tuyn tnh hon ton l nghim ngt v khng

ph hp vi cc bi ton thc t, c bit l cc trng hp phn lp phi tuyn phc tp.

Trong khi , cc mu khng phn tch tuyn tnh hon ton dn n vic khng th gii

quyt cc bi ton ti u tm w v b tng ng. gii quyt vn ny, c hai cch

tip cn chnh:

7

Soft-margin Th thut Kernel.

2.2.1 SOFT MARGIN

V nhiu l do, do bn cht hoc do sai st trong qu trnh thu thp d liu, tn ti mt

s im thuc lp ny ln ln vo lp kia, iu ny s lm ph v s phn tch tuyn

tnh. Nu ta c tnh phn tch hon ton s lm cho m hnh d on qu khp. chng

li s qu khp, ngi ta m rng SVC n chp nhn mt vi im phn lp sai. K

thut ny gi l soft margin.

Hnh 4: Minh ha trng hp d liu nhiu;

lm iu ny, mt bin (gi l slack variable) i c thm vo biu thc cn ti

u nhm cho php m hnh phn lp thc hin phn lp sai mc chp nhn c:

( )

(14)

Tham s C dng cn bng gia phc tp tnh ton v s lng im khng th

phn tch. N c gi l tham s chun ha (regularization parameter). Gi tr C c

th c lng nh thc nghim hoc phn tch d liu.

Cc bin i c thm cho tng im d liu, cho bit s sai lch khi phn lp vi

thc t. C th:

cho nhng im nm trn l hoc pha trong ca l.

( ) cho nhng im cn li.

8

Theo , nhng im nm trn ng phn cch ( ) s c v nhng im phn lp sai s c . Theo Lagrange ta vit li:

( )

* ( ) +

(15)

Trong * + v * + l cc nhn t Lagrange.

Cc iu kin KKT cn tha l:

( )

( ( ) )

vi i = 1,,n

Ly o hm (15) theo w, b v { }:

Th tt c vo (15) ta c:

( )

(16)

Suy ra cng thc cho bi ton kp vi soft margin nh sau:

9

( )

(17)

iu kin KKT tng ng l:

, ( ) -

(18)

Nh trc , tp cc im c khng c ng gp g cho vic d on im d liu mi. Nhng im cn li to thnh cc support vector. Nhng im c v theo (18) tha:

( )

(19)

Nu th , suy ra . l nhng im nm trn l.

Nhng im c th , c th l nhng im phn lp ng nm gia l v ng phn cch nu hoc c th l phn lp sai nu

xc nh tham s b, chng ta dng nhng support vector m , tng ng vi . Ln na, m bo tnh n nh ca b ta nn tnh theo trung bnh.

2.2.2 TH THUT KERNEL

Theo nh l Cover v s phn tch mu, mt bi ton phn lp mu phc tp m

chuyn sang khng gian c s chiu cao bng php chuyn i phi tuyn th c kh nng

phn tch tuyn tnh cao hn khi chuyn sang khng gian c s chiu thp. Nh vy, ta

c th gii quyt vn khng phn tch tuyn tnh bng cch thc hin php chuyn i

phi tuyn d liu u vo sang khng gian c s chiu cao hn (thm ch l v cng).

Tuy nhin, s chuyn i nh vy s lm tng phc tp tnh ton v xc nh s chiu

ph hp l vn khng d dng. Th thut kernel c a ra gii quyt vn ny.

Mu cht ca th thut kernel l xc nh hm kernel ph hp tnh c tch v hng

ca mu d liu sau khi thc hin chuyn i m khng cn phi quan tm s chiu l

bao nhiu.

10

Hnh 5. Bin i khng gian d liu sang khng gian c trng.

Gi : X H l php bin i phi tuyn t khng gian u vo m chiu X vo khng

gian c trng H m cc mu c th phn tch tuyn tnh. Khi ng phn cch

ti u c nh ngha nh sau:

( )

(20)

Khng mt tnh tng qut, gn b = 0 v cng thc n gin thnh:

( )

(21)

Lm tng t nh phn trn th vector trng s ti u l trong khng gian c trng mi s l:

( )

(22)

Theo siu mt phng ti u trong khng gian c trng mi l:

( ) ( )

(23)

Trong ( ) ( ) l tch v hng ca hai vector (x) v (xi). T y, chng ta c th p dng hm kernel tch v hng.

nh ngha (Kernel tch v hng): Kernel l mt hm K(x, x), sao cho vi mi x,

x thuc X, X l tp con ca khng gian m chiu Rm tha iu kin sau:

( ) ( ) ( ) (24) Trong l php bin i khng gian u vo X sang khng gian c trng H.

Theo , hm xc nh siu phng ti u s thnh:

11

( )

(25)

u im ca kernel l c th xy dng ng phn cch ti u m khng phi quan

tm chi tit n dng hm ca php bin i . Nh vy, hm kernel lm cho thut ton

khng nhy cm vi s chiu, trnh c cc tnh ton phc tp khi tnh tch v hng

cng nh thit k b phn lp. nh l Mercer ch ra cc thuc tnh m hm kernel

K(x,x) cn phi c.

nh l Mercer: Cho K(x, x) l mt hm i xng lin tc c nh ngha trn min

gi tr ng v tng t cho x. Hm K(x, x) l kernel nu c th m rng theo dy nh sau:

( ) ( ) ( )

(26)

Trong , i l h s dng vi mi i. s m rng ni trn l hp l v hi t th

cn iu kin sau:

( ) ( ) ( )

(27)

ng vi mi (.) m

( )

(28)

Din gii nh l Mercer: c im hu ch nht cn ch khi xy dng kernel l bt

k mt tp con hu hn ngu nhin trong khng gian u vo X th ma trn xy dng

tng ng vi hm kernel K(x, x) l ma trn i xng v bn xc nh, cn gi l ma

trn Gram.

( ( ))

K l ma trn bn xc nh nu tt c cc tr ring ca n l khng m. Mt s bi ton,

rng buc iu kin ma trn l xc nh dng, ngha l cc tr ring phi ln hn 0

m bo rng bi ton s hi t v gii php l duy nht.

Theo rng buc ni trn th vic chn hm kernel vn kh phng khong, Cc loi

hm kernel c th s dng nh:

Tuyn tnh:

12

( ) (29)

a thc:

( ) ( ) (30)

Gaussian (radial basis function):

( ) (

)

(31)

Vic iu chnh cc tham s trong hm cho ph hp vi bi ton l mt cng vic kh

vt v. Hm a thc v radial basis function l hai hm c s dng ph bin nht. Tuy

nhin, chn c hm ph hp chng ta nn da theo thng tin m ta mun rt trch ra

t d liu. Chng hn,

Hm a thc cho php m hnh ha s lin kt ca cc c trng (tng theo bc ca a thc)

Radial basis function cho php m hnh ha nhng c trng c th phn tch d liu phn b theo hnh trn (hoc siu cu)

Hin nay, c mt s nghin cu h tr hm kernel mt cch t ng, chng hn nh

nghin cu ca Tom Howley v Michael MaddenError! Reference source not found..

So snh th thut kernel vi soft margin c th thy hai hng tip cn theo nhng

cch khc hn nhau.

Margin mm Th thut kernel

Thm rng buc ni lng v cho php li

Khng hiu qu khi m d liu khng th phn tch tuyn tnh nhiu, chng hn phn lp phi tuyn, v li phn lp nhm qu cao

Chuyn d liu sang khng gian c chiu cao hn mt cch khng tng mnh nh hm kernel lm cho bi ton c th phn tch tuyn tnh.

Khng th m bo rng bi ton c th phn tch tuyn tnh tuyt i trn tt c d liu, c bit l nhng bi ton phc tp.

Da trn u v khuyt im ca hai phng php, chng ta c th kt hp tng

chnh xc. Khi , bi ton ti u ha c rng buc s c dng nh sau:

( )

( )

(32)

13

Theo phng php nhn t Lagrange, ta thu c hm phn lp ti u nh sau:

( ) ( )

(33)

Trong , b* c th tnh bi cc vector h tr dng,

( )

(34)

2.3 CC PHNG PHP HUN LUYN SVC

Trng tm hun luyn SVC l hun luyn cc vector h tr t d liu u vo. Nh

cp trn, cc nhn t Lagrange cho php chng ta xc nh c cc vector h tr.

Tm cc nhn t Lagrange ti u cho tng im d liu thuc dng bi ton quy hoch

bc 2 (quadratic programming), vit tt l QP. Kch thc d liu hun luyn rt ln, lm

cho bi ton QP trong SVM tr nn kh gii vi cc k thut gii QP chun. C th l ma

trn lu tr d liu cn thit chy thut ton c kch thc bng bnh phng s lng

mu, vi 4000 mu hun luyn th ma trn b nh 128MB l khng .

T kh khn , nhiu tc gi xut cc phng php hun luyn SVC vi kch

thc d liu ln. Trong phn ny, chng ti cp 3 phng php tiu biu theo th t

thi gian ra i.

2.3.1 PHN ON (CHUNKING)

Phng php phn on c xut bi Vapnik, da trn thc t l kt qu thu c

ca bi ton QP trn ma trn ban u v trn ma trn b i cc dng v ct ca d liu

c nhn t Lagrange bng 0 l nh nhau.

Theo , tc gi chia bi ton QP ban u thnh cc bi ton QP nh hn, vi mc

tiu l xc nh tt c cc nhn t Lagrange khc khng v loi b cc nhn t Lagrange

bng 0. mi bc lp, phng php gii bi ton QP gm cc mu sau:

Cc mu tng ng vi nhn t Lagrange khc 0 bc lp trc.

M mu t nht vi phm iu kin KKT, vi M l gi tr cho trc. Nu c t

hn M th ly ht cc mu vi phm.

14

Mi bi ton con QP c khi to theo kt qu ca bi ton con trc . bc

cui cng, tt c cc nhn t Lagrange khc 0 s c xc nh v bi ton c gii.

Phng php phn on gim kch thc ca bi ton xung cn xp x bnh phng

cc mu hun luyn c nhn t Lagrange khc 0. Tuy nhin, kch thc ma trn vn cn

ln v khng th lu trong b nh.

2.3.2 PHNG PHP CA OSUNA

Nm 1997, Osuna v ng tc gi chng minh thnh cng nh l lin quan n

cc thut ton QP. nh l ny ni rng, bi ton QP ln c th chia nh thnh tp cc bi

ton QP nh hn. Ch khi t nht mt mu vi phm iu kin KKT c thm vo tp

mu cho bi ton con trc , mi bc s gim hm mc tiu v duy tr kh nng tha

mn tt c cc rng buc. Do , mt dy cc bi ton con QP lun lun thm t nht mt

mu vi phm s m bo hi t. Thut ton phn on trn cng tun th cc iu kin

ca nh l nn n hi t

Osuna v cc ng s xut gi cho kch thc ca ma trn lun c nh trong tt

c cc bi ton con. iu ny ng ngha vi vic s lng mu thm vo v xa i l

bng nhau. S dng ma trn c kch thc c nh cho php hun luyn trn tp d liu

ln. Trn l thuyt, Osuna ngh thm vo mt mu v ly ra mt mu mi bc. Tuy

nhin, trong thc t tng tnh hiu qu cc nh nghin cu thao tc trn nhiu mu,

vic chn mu da trn heuristic.

Tuy nhin, cc nghin cu vn s dng phng php s hc gii cc bi ton QP

con. Nhng phng php s hc th rt kh thu c kt qu ng do sai s.

2.3.3 SMO SEQUENTIAL MINIMAL OPTIMIZATION

SMO l phng php n gin c th gii quyt nhanh bi ton QP trong SVM m

khng yu cu b nh lu tr thm cng nh khng s dng phng php s hc mi

bc.

SMO cng phn r thnh cc bi ton nh hn, nhng khng ging cc phng php

trc, n chn bi ton ti u nh nht c th mi bc. C th hn, l bi ton ti u

ch lin quan n hai nhn t Lagrange. mi bc, SMO chn ra hai nhn t Lagrange

ti u ha ng thi, v cp nht SVC phn nh cc thay i.

Mt u im ca SMO l v ch lm vic vi hai nhn t Lagrange nn c th s dng

phng php gii tch, trnh c vn sai s ca phng php s hc. Ngoi ra, v

khng yu cu thm b nh nn SMO ph hp vi mi bi ton hun luyn SVC ln trn

my tnh c nhn hoc workstation.

15

SMO gm 2 thnh phn: thut ton phn tch tm gi tr ca nhn t Lagrange v

mt heuristic chn nhn t ti u mi bc.

Hnh 6. Cc thnh phn ca SMO.

Chi tit cc thnh phn c th tham kho trong ti liu Error! Reference source not

found..

2.4 CC HNG PHT TRIN

Trong thp k trc, SVM c pht trin nhanh c v l thuyt ln thc hnh.

Hin nay, nhiu nghin cu vn tip tc v vn ny. Trong phn ny, chng ta s lit

k mt s hng nghin cu chnh ang c thc hin v nhiu vn nghin cu m.

2.4.1 HIU QU TNH TON

Mt trong nhng hn ch ca SVM trc y l phc tp tnh ton cao trong giai

on hun luyn, dn ti khng th p dng thut ton i vi CSDL ln. Mt cch tip

cn mi l chia vn ti u ha thnh chui cc vn nh hn, trong mi vn

ch lin quan ti mt cp gi tr c chn k lng vic ti u ha c th c thc

hin mt cch hiu qu. Tin trnh lp li cho ti khi cc vn ti u ha c phn tch

v gii quyt thnh cng.

Mt cch tip cn gn y l coi vn hc SVM nh l vic tm xp x nh nht ca

tp ng. Cc mu ny mt khi a vo khng gian N chiu s biu din mt tp im

c s dng xy dng mt xp x nh nht. Gii quyt cc vn hc SVM trn cc

tp ct li ny c th to ra cc li gii xp x tt vi tc tt. V d nh my vector li

[18] c th hc SVM cho hng triu d liu trong mt giy.

2.4.2 LA CHN KERNEL

Trong SVM kernel, vic la chn hm kernel phi p ng nh l Mercer

(http://en.wikipedia.org/wiki/Mercer's_theorem). Bi vy, cc hm kernel ph bin bao

16

gm 3 loi: sigmod, a thc v radial basis function. iu ny s gii hn kh nng p

dng kernel trick. Gn y, Pekalska v cng s cung cp mt cch xem xt mi l

thit k mt hm kernel da trn i snh quan h xp x tng qut. Hm kernel mi

khng cn phi p ng cc iu kin ca Mercer cng khng b gii hn ch trong khng

gian c trng v cho kt qu phn lp tt hn kt qu thc nghim kernel Mercer thng

thng. Tuy nhin, c s l thuyt ca kernel tng qut ca Pekalska cn thc hin cc

nghin cu xa hn trong tng lai.

Hn na, mt cch tip cn ph bin khc l hc nhiu kernel. Thng qua vic kt ni

cc kernel, h thng c th cho kt qu tt hn. iu ny cng tng t vic s dng

ng b cc kernel. Bng vic thit lp cc hm mc tiu ring, s la chn cc tham s

kernel tt hn c th c thc hin cho php ha trn kernel.

2.4.3 PHN TCH TNG QUT

Chng ta quen vi vic s dng kch thc VC c lng gii hn li tng qut

ca cc my kernel. Tuy nhin, gii hn i hi phc tp phi c nh, khng ph

thuc vo d liu hun luyn v dn ti khng th thc hin phn lp hiu qu. gii

quyt vn ny, phc tp ca Rademacher c gii thiu nh mt gii php thay

th nh gi phc tp ca b phn lp thay cho kch thc VC truyn thng, kh

nng phn lp ph hp vi d liu ngu nhin. N c nh ngha nh sau:

nh ngha 5.1 ( phc tp Rademacher). i vi mu S = {x1, x2, , xn} c

to ra bi phn b D trn tp d liu X v lp hm gi tr thc F vi min X, phc tp

Rademacher thc nghim ca F l gi tr ngu nhin

1 2

1

2 ( ) [ sup | ( ) ||x , x , ... , x ]n

n i i nf F i

R F E f xn

Trong , = {1, 2, , n} l cc gi tr {1} gi tr ca bin ngu nhin

(Rademacher). phc tp Rademacher ca F l:

1

2( ) [R (F)]=E [ sup | ( )|]n

n S n S i if F i

R F E f xn

Phn sup bn trong biu thc tnh k vng l o tng quan tt nht c th tm

c gia hm cc lp v nhn ngu nhin. Hn na, trong cc my kernel chng ta c

th t c mt gii hn trn cho phc tp Rademacher:

nh l 5.2 Phn tch phc tp. Nu k: X x X -> R l mt kernel v S = {x1, x2,

, xn} l mt mu cc im t X, th phc tp Rademacher thc nghim ca b phn

lp FB tha mn:

17

1

2 2 ( ) ( , ) ( )n

n B i i

i

B BR F k x x tr K

n n

Trong B l gii hn ca cc trng s w trong b phn lp.

Mt iu ng ch l phc tp Rademacher ch bao gm ma trn kernel tng

ng c xc nh thng qua d liu hun luyn c th. N kh thi hn so vi s dng

kch thc VC truyn thng ti u phc tp b phn lp.

2.4.4 HC SVM C CU TRC

Gn y, mt s thut ton quan tm ti thng tin cu trc c pht trin nhiu hn

SVM truyn thng. H cung cp mt quan im mi khi thit k mt b phn lp, ni m

mt b phn lp c th nhy cm vi cu trc phn b ca d liu. Cc thut ton ny

thng c 2 cch tip cn chnh. Cch tip cn th nht l hc nhiu fold. Gi s d liu

c chia thnh cc fold nh trong khng gian u vo v thut ton in hnh nht l

my h tr vector Laplacian (LapSVM). u tin, chng ta c th xy dng LapSVM

thng qua th Laplacian trong mi lp. Sau , chng ta a cu trc nhiu fold ca

d liu vi cc ma trn Laplacian tng ng vo cc framework truyn thng ca SVM

nh ton hng b sung.

Cch tip cn th hai l bng cch tm hiu cc thut ton gom cm bng cch gi s

rng d liu s cha mt s cm da vo thng tin phn b ban u. Gi s ny dng

nh tng qut hn so vi trng hp hc nhiu fold, iu ny dn ti xut hin mt s

margin machine ln ph bin. Cch tip cn gn y c bit ti l margin machine ln

c cu trc (SLMM). SLMM p dng cc k thut gom cm nm bt thng tin cu

trc trong cc lp. Sau , n s dng khong cch Mahalanobis nh mt o khong

cch t cc mu ti cc mt siu phng quyt nh thay th cho khong cc Euclidean

truyn thng, gii thiu thng tin cu trc lin quan theo mt quy nh no . Mt s

my margin ln ni ting nh SVM MPM (minimax probability machine), M4 (maxi-min

margin machine) c xem nh l cc trng hp c bit ca SLMM. Thc nghim cho

thy SLMM cho kt qu phn lp tt hn. Tuy nhin, khi vn ti u ha ca SLMM

c thnh lp cng thc nh SOCP (second order cone programming) hn l QP trong

SVM th SLMM c ch ph tnh ton cao hn nhiu so vi SVM truyn thng khi hun

luyn. Hn na, n khng tng qut i vi cc quy m d liu v trong trng hp

nhiu lp. Do , my h tr vector c cu trc mi (SSVM) c pht trin khm

ph framework c in ca SVM hn nhiu so vi cc hn ch trong SLMM. Vn ti

u ha tng ng c th vn c gii quyt bi QP nh trong SVM v gi cho li gii

khng ch tha tht m cn kh nng m rng. Hn na, SSVM c ch ra ni chung

l tt hn v mt l thuyt v thc nghim hn so vi SVM v SLMM.

18

3 SUPPORT VECTOR REGRESSOR SVR

3.1 GII THIU BI TON HI QUY

Gi s rng chng ta a ra mt tp hun luyn bao gm N = 10 phn t (x1, , xn),

quan st gi tr u vo ca x cng vi cc gi tr mc tiu tng ng (t1, , tn).

ng mu xanh l cy c to ra t hm f(x) = sin(2x) v 10 im mu xanh

(blue) c to ra t hm sin cng thm 1 sai s nht nh (iu ny nhm m phng

trong thc t khi thu thp d liu s c mt sai s nht nh). Mc tiu l ta cn tm 1

ng hi quy gn nh tng t vi ng sin t tp d liu m khng h bit g v

ng mu xanh l cy.

Nhn xt: iu ny bn cht l mt vn kh khn khi chng ta phi khi qut ha

mt d liu hu hn. Hn na d liu quan st b hng vi nhiu. V vy cho mt gi tr x

khng chc chn l thu c gi tr t chnh xc.

Chng ta kho st phng trnh ng cong bc M nh sau:

Trong : M l bc ca a thc, xj l th hin ca x, wj l h s chung biu th bng

cc vector. Lu , mc d hm a thc y(x,w) l mt hm phi tuyn theo x v tuyn tnh

theo w.

Cc gi tr ca cc h s s c xc nh bng cch a vo cc a thc trn tp d

liu hun luyn. iu ny c th c thc hin bng cch gim thiu chc nng li gia

cc hm y(x,w). Mt cch n gin chn la hm li c s dng rng ri l hm li

c cho bi tng ca bnh phng sai gia cc tin on y(xn,w) cho mi im d liu

xn v tng ng gi tr mc tiu tn, v vy chng ta c hm li ti thiu:

19

Gi tr 1/2 c thm vo nhm thun tin cho vic tnh ton sau ny cho sau ny.

Tip theo chng ta s cp n vic la chn hm li nh th no l tt nht.

Chng ta c th gii quyt vn qu khp bng cch chn gi tr ca w hm E(w)

nh nht c th. Bi v hm li l mt hm bc 2 theo tham s w, cc dn xut ca n i

vi cc h s s l tuyn tnh trong cc phn t ca w, v v s ti thiu ca hm li c

1 gii php duy nht, th hin bi w*, c th tm thy trong cng thc cui cng. Kt qu

a thc c mang li bi hm y(x, w*).

Nhn xt: Vi M=0, chng ta nhn thy rng ng hi quy khng i. M=1, a thc

cho kt qu km khp vo cc d liu. M=3, a thc cho kt qu khp tt nht. V cui

cng vi M=9, a thc cho kt qu qu khp trong hun luyn d liu.

Cho mi ln chn M, sau chng ta c th nh gi cc gi tr cn li ca E(w*)

c mang li bi E(w) cho vic hun luyn d liu.

i khi n l thun tin hn s dng hm li ERMS (RMS=root-mean-square):

Trong phn chia bi N cho php chng ta so snh kch c cc b d liu khc nhau

mt cch bnh ng, v cn bc m bo rng ERMS c o trn cng mt t l (v trong

cng mt n v) nh l cc bin mc tiu t.

20

Sau khi p dng hm li ERMS ta c th training v test nh sau:

Ta nhn thy gi tr M nm trong khong 3

21

Tuy nhin, nu chng ta s dng qu ln th kt qu s km ph hp. Ta c bng

kim tra gi tr (w*)

Sau khi p dng hnh pht li mi vi ta c th training v test nh sau:

3.2 HI QUY VI SVR

tng c bn ca SVR l nh x khng gian u vo (m nu ta p dng trc tip

hi qui tuyn tnh th khng hiu qu) sang mt khng gian c trng nhiu chiu m

, ta c th p dng c hi qui tuyn tnh.

c im ca SVR l cho ta mt gii php tha (sparse solution); ngha l xy

dng c hm hi qui, ta khng cn phi s dng ht tt c cc im d liu trong b

22

hun luyn. Nhng im c ng gp vo vic xy dng hm hi qui c gi l nhng

Support Vector.

Hm hi qui cn tm c dng:

( ) ( ) (35) Trong , l vc t trng s, T l k hiu chuyn v, l hng s,

l vc t u vo, ( ) l vc t c trng, lm hm nh x t khng gian u vo sang khng gian c trng.

Nh vy, mc tiu ca vic hun luyn SVR l tm ra c w v b.

Cho tp hun luyn {(x1, t1), (x2, t2), , (xN, tN)} . Vi bi ton hi qui n

gin, tm w v b ta phi ti thiu ha hm li chun ha:

* +

(36)

Vi l hng s chun ha.

c c mt gii php tha, ta s thay hm li trn bng hm li -insensitive. c

im ca hm li ny l nu tr tuyt i ca s sai khc gia gi tr d on y(x) v gi

tr ch nh hn (vi > 0) th n coi nh li bng 0.

( ( ) ) { | ( ) |

| ( ) | (37)

Hnh 7: Minh ha hm li thng thng v hm li -insensitive

Hnh trn so snh gia hm li bc 2 thng thng (ng cong mu xanh) v hm

li -insensitive (ng gp khc mu .) Nh ta c th thy khi s sai khc gia gi tr

d on v gi tr ch nm trong [-, ] th hm li -insensitive c gi tr bng 0.

Nh vy by gi, ta phi ti thiu ha hm li chun ha sau:

( ( ) )

(38)

23

Vi ( ) ( ) , C l hng s chun ha ging nh nhng c nhn

vi hm li thay v vi .

cho php mt s im nm ngoi ng , ta s a thm cc bin lng (slack

variable) vo. i vi mi im d liu xn, ta cn hai bin lng v , trong

ng vi im m ( ) (nm ngoi v pha trn ng) v ng vi im m ( ) (nm ngoi v pha di ng.) Hnh di minh ha cho cc bin lng ny.

Hnh 8: Minh ha SVR vi cc bin lng

iu kin mt im ch nm trong ng l: vi yn = y(xn). Vi vic s dng cc bin lng, ta cho php cc cc im ch nm ngoi ng (ng vi

cc bin lng > 0) v nh th th iu kin by gi s l:

(39)

(40) Nh vy, ta c hm li cho SVR:

( )

(41)

Mc tiu ca ta l ti thiu ha hm li ny vi cc rng buc:

Gi y l vn ti u ha A.

C ngay hm Lagrange:

24

( )

( )

( )

( )

(42)

Vi ( ) ( )

Ly o hm theo w, b, , v cho bng 0, ta c:

( ) ( )

(43)

( )

(44)

(45)

(46)

Dng 4 kt qu ny th vo (42), ta s loi b c w, b, , , , :

( )

( )( ) ( )

( )

( )

(47)

Vi k l hm nhn: ( ) ( ) ( ). Bt k mt hm no tha iu kin Mercer th u c th c dng lm hm nhn. Hm nhn c s dng ph bin nht l

hm Gaussian:

( ) ( ) (48)

Nh vy, ta chuyn t vn ti u ha A sang vn ti u ha tng ng B:

Ti a ha (47) vi cc rng buc:

25

( )

Li ch chnh ca vic chuyn i t vn ti u ha A sang vn ti u ha B l

vn ti u ha B c s dng hm nhn. iu ny s gip cho vic tnh ton trong

khng gian nhiu chiu tr nn rt hiu qu.

Th (43) vo (35), ta c s d on cho mt mu mi x:

( ) ( )

( ) (49)

Theo iu kin KKT (KarushKuhnTucker), c ngay:

( ) (50)

( ) (51) ( ) (52) ( ) (53)

T y, ta c th rt c nhng thng tin quan trng nh sau:

Nu th : im nm bin trn ca ng ( ) hoc nm ngoi v pha trn ca ng ( )

Nu th : im nm bin di ca ng ( )

hoc nm ngoi v pha di ca ng ( )

v khng th cng dng v nu vy th ta c: v

, cng li ta s thy ngay v tri lun dng, trong khi v phi bng 0: v l!

Nhng im Support Vector l nhng im ng gp vo hm d on (49), ngha

l nhng im c hoc : nhng im nm trn bin ng hoc nm ngoi ng.

Nhng im nm trong ng s c v do khng ng gp g vo qu trnh d on.

Thy ngay d tnh c b bng cch xt mt im xn c . T (52) ta c . T (50) ta c . Kt hp vi (49) c ngay:

( )

( ) (54)

Ta cng s c kt qu tng t nu xt im c .

vng chc hn, ta nn ly trung bnh ca tt c cc gi tr ca b li.

26

Vi SVR s dng hm li -insensitive v hm nhn Gaussian ta c 3 tham s cn

phi xc nh: h s C, tham s ca hm nhn Gaussian v rng ca ng .

C 3 tham s ny u nh hng n chnh xc d on ca m hnh v cn phi

chn la k cng (do ngi dng truyn vo trc khi hun luyn, c th dng cc thut

ton nh GA, GridSearch tm cc tham s ny).

Nu C qu ln th s u tin vo phn li hun luyn, dn n m hnh phc

tp, d b qu khp. Cn nu C qu nh th li u tin vo phn phc tp m

hnh, dn n m hnh qu n gin, gim chnh xc d on.

ngha ca cng tng t C. Nu qu ln th c t vect h tr, lm cho m

hnh qu n gin. Ngc li, nu qu nh th c nhiu vect h tr, dn n m

hnh phc tp, d b qu khp.

Tham s phn nh mi tng quan gia cc vc t h tr nn cng nh hng n chnh xc d on ca m hnh.

Mt vi nhn xt gia SVC v SVR:

SVC c l c nh bng 1, vi mc ch tch lp, tc l y cc im d liu ra 2

pha ng phn cch, iu ny th hin r phn iu kin ca (7).

SVR c l do ngi dng truyn vo, vi mc ch hi quy, tc l ht cc im d

liu vo trong ng , iu ny th hin r phn iu kin ca (41).

Vy v sao vi SVC l bng 1, nhng SVR phi la chn ? Do mc tiu ca SVC

l tm ng phn cch nhng vi tiu ch phi c l ln nht, ta s thy c rng

khi ta c ng phn cch c th, th vic xc nh l l d dng. Do l l im

gn nht n ng phn cch, im ny l xc nh c khi c ng phn cch

c th nn n gin ta cho l bng 1 (iu ny lm c do ch cn scale w v b

trong cng thc khong cch m khng nh hng tnh cht vn ).

Nhng vi SVR l l cha c xc nh bi mt im c bit no c, v rng

ca l c nh hng n cht lng ca hm hi qui nn cn xem nh tham s

truyn vo.

Khi dng hm Kernel tng s chiu c phi cng ln cng tt? iu ny l khng

m bo, c th ta c th xt v d mc 3.1, khi m qu ln d dn n qu khp.

Do thng thng khi tng s chiu d liu ta khng nn tng thm qu s

chiu ban u.

27

3.3 SUPPORT VECTOR REGRESSION

Chnh sa t -SVR vi:

c xc nh trong qu trnh tnh ton thay v chn t u.

v khng m.

i vi -SVR ta cn cc tiu ha hm.

v-SVR b sung thm s hng v, vo hm cc tiu ha

Lp phng trnh Lagrange:

Tnh o hm tng ng vi cc bin ta c:

Gii h 4 phng trnh trn ta s dn n bi ton ti u ha hm

Vi cc rng buc i km l

28

v cn trn ca t l li (fraction of errors)

v cn di ca t l SV(fraction of SVs)

v xp x bng 2 t l trn

v-SVR i tm li gii vi cc bin , w,b trong khi -SVR i tm li gii vi cho

trc v cc bin l w, b.

S dng SVR th nhng thay i nh cc b gi tr ch (target value) ca nhng

im nm ngoi ng ng khng lm nh hng n kt qu xy dng ng hi quy.

xc nh c khng ph thuc vo s mu d liu

So snh v nhn xt:

29

Vi 2 mc nhiu khc nhau (0 v 1) th phng php v-SVR lun hiu chnh c

( 0 v 1.19 ) tt v ph hp vi tp d liu.

v SVR : gim khi v tng, v >1 th = 0. Do nn chn v: 0 v 1

Nu v < 1 th thng > 0, vn c th bng 0 nu d liu khng nhiu.

Chn v khc nhau ch lm s ln nhy gi tr ca khc nhau, cui khng i.

v SVR : tng khi s nhiu tng

M hnh tham s trong v-SVR

c s dng trong trng hp c nhiu ph thuc vo mu d liu x

Gi l tp 2p hm trong khng gian u vo. Vi , ta cc

tiu ha hm:

Tha rng buc:

30

Tnh ton tng t nh cc phn trn duy ch mt s rng buc thay i nh sau:

4 PH LC

L thuyt nhn t Lagrange

Vn cc i hm f(x) tha iu kin ( ) s c vit li di dng ti u ca hm Lagrange nh sau:

( ) ( ) ( )

Trong x v phi tha iu kin Karush-Kuhn-Tucker (KKT) nh sau:

( )

( )

Nu l cc tiu hm f(x) th hm Lagrange s l

( ) ( ) ( )

5 TI LIU THAM KHO

[1] Bernhard Schlkopf, Alexander J. Smola, Learning with Kernels: Support Vector

Machines, Regularization, Optimization, and Beyond, The MIT Press, 2001, pp.

251-277

[2] Bishop C. M., Pattern Recognition and Machine Learning, Springer, 2006

[3] John C. Platt, Sequential Minimal Optimization, A Fast Algorithm for Training

SVM, 1998

[4] Tom Howley, Michael Madden, The Genetic Kernel Support Vector Machine,

Kluwer Academic, 2005

[5] Xindong Wu, Vipin Kumar, The Top Ten Algorithms in Data Mining, Chapman

and Hall/CRC, 2009, pp. 37-59