Khai thc d liu Gom cm vn bn

Khai thc d liu v ng dng ti: Gom cm vn bn Ting AnhSVTH: L Th Duyn1042023Phm Th Nng1042075

LOGONi dung thc hinTng quan v gom cm1 Phng php gii quyt bi ton M hnh nh gi cht lngTi liu tham kho2341. Tng quan v gom cmS pht trin mnh m ca internet lm cho con ngi cng trn ngp trong khi d liu khng lLm sao gim thi gian bin mc ti liu trong cc th vin in t?Lm sao gim thi gian tm kim ti liu tin cy t ngun thng tin to ln ny?Lm th no gim thiu s lng i tng cn thit trong vic phn loi ti liu ?

=> Cn thit xy dng cng c gom cm d liu cho ngun thng tin ny.

31. Tng quan v gom cmGii thiu: Gom cm vn bn l bi ton quan trng trong KTDL, cho php gom cc vn bn c ni dung tng t nhau vo cng mt cm

iu ny gii quyt c nhng vn gim bt chi ph, tm kim,

1. Tng quan v gom cmV c bn, m hnh gom cm cn xc nh cc yu t sau: Tin x l d liuPhng php tnh tng t gia cc vn bnThut ton dng gom cmK-MeansHACnh gi cht lng m hnh gom cmPrecisionRecallEntropy

1. Tng quan v gom cmVai tr:H tr giai on tin x l d liu (Data Preprocessing)M t s phn b d liu/i tng (Data Distribution)Gom cm ti liu (Document Clustering)Nhn dng mu (Pattern Recognition)

1. Tng quan v gom cmMc ch:Gim thiu chi ph Tit kim thi gian tm kimXc nh c bn cht ca vic nhm cc i tng trong mt tp d liu khng c nhn

1. Tng quan v gom cmPhn loi cc phng php gom cm:Phng php phn hoch (Partitioning)Phng php phn cp (Hierarchical)Da trn mt (Density Based)Da trn li (Grid Based)

2. Phng php gii quyt bi tonV c bn, m hnh gom cm cn xc nh c hai yu t:Tnh tng t v ni dung gia hai vn bn ( tng t cng cao th c xu hng thuc v mt cm)Thut ton gom cm: dng gom cc i tng gn nhau li thnh mt nhm

2. Phng php gii quyt bi tonMt s hng tip cn tnh tng tDa trn m hnh khng gian vectorDa trn m hnh thDa trn tng t cu

2. Phng php gii quyt bi tonTnh tng t gia 2 vn bnHng tip cn

M hnh khng gian vector

M hnh th

tng t cu

2. Phng php gii quyt bi tonM hnh khng gian vectorMi vn bn c biu din bng mt vectorMi chiu ca vector tng ng vi mt t (chui t) quan trng (vector c trng)Khi , tng t gia hai vn bn chnh l tng t gia hai vector i din

2. Phng php gii quyt bi tonMt s h s dng tnh tng t gia hai vector

EuclideDice JacardCoefficientH sCosine2. Phng php gii quyt bi ton tng t Cosine: L mt trong nhng o tng t ph bin nht c p dng vi vn bn dng textGi s c hai vn bn vi 2 vector i din l t1 v t2, tng t Cosine:

2. Phng php gii quyt bi ton

2. Phng php gii quyt bi ton

2. Phng php gii quyt bi tonH s tnh trng lng ca t: tf-idfVd: Trong mt vn bn bao gm 100 t, trong t learn xut hin 3 ln. Gi s ta c 10 triu vn bn trong tp d liu v t learn xut hin trong 1000 vn bn trong s . Vy trng s ca t learn theo tf-idf l? Cch tnh: tf ca t learn l: (3/100) = 0.03idf ca t learn c tnh l:idf = log(10 000 000/1000) =4Vy trng s ca t learn theo tf-idf l: 0.03 x 4 = 0.12

2. Phng php gii quyt bi tonThut ton gom cm: Chn thut ton K-Means ci t gii quyt bi tonCc bc thc hin thut ton:Bc 1: Chn ngu nhin k tm cho k cmBc 2: Tnh khong cch gia cc i tng n k tm (Euclide)Bc 3: Nhm cc i tng vo nhm gn nht

2. Phng php gii quyt bi tonBc 4: Xc nh li tm mi cho cc nhmBc 5: Thc hin li bc 2 cho n khi khng c s thay i nhm no ca cc i tng

2. Phng php gii quyt bi tonu v nhc im ca thut ton:u im: n gin, d hiu, d ci tNhc im: Hiu qu ca thut ton ph thuc vo vic chn s nhmChi ph thc hin vng lp tnh khong cch kh lnVic chn k ngu nhin s cho nhng kt qu khc nhau

2. Phng php gii quyt bi tonTh vin/Cng c h tr:S dng cng c tch cu, tch t:Sharp NLP: http://sharpnlp.codeplex.com/ OpenNLP: http://incubator.apache.org/opennlp/Cng c phc hi t gc:Themis: http://code.google.com/p/ir-themis/ Porter stemmer: http://tartarus.org/~martin/PorterStemmer/

3. M hnh nh gi cht lngPhng php gom nhm tt nht l phng php s to ra cc nhm c cht lng. l s ging nhau gia cc i tng trong cng mt nhm hoc gia cc nhm c s ging nhau thpLm sao bit c iu ny???

3. M hnh nh gi cht lng nh gi cht lng kt qu gom nhm ngi ta dng mt s o cht lng:Bnh phng sai (Sum of Squared Error SSE) hn lon (Entropy) chnh xc (Precision) bao ph (Recall)3. M hnh nh gi cht lng o Entropy:Gi s: P l kt qu phn chia ca thut ton bao gm m cm. Vi mi cm j trong P, ta cn tnh Pi,j (Pi,j l kh nng ca 1 thnh vin cm j thuc vo lp i). Entropy ca mi cm j c tnh nh sau:

Ej = - log(Pi,j)

3. M hnh nh gi cht lngng dng ca m hnh gom cm:Phn cm ti liu WebPhn vng Biology (sinh hc)Libraries (th vin)Insuranse (bo him)City planning (quy hoch ha th)Earthquake (nghin cu a chn)

4. Ti liu tham khoTi liu tham kho:[1] Anna Huang, Similarity measures for Text Document Clustering, NZCSRSC 2008, April 2008. Christchurch, NewZealan [2] M. Steinbach, G. Karypis, V. Kumar, A Comparison of Document Clustering Techniques, KDD-2000 Workshop on Text Mining, 2000.

26Q & A

HT GROUPThank You !LOGO