Upload
sandra-gilmore
View
213
Download
1
Embed Size (px)
Citation preview
Web NoisesDetection and Elimination
PengBoDec 3, 2010
What are Web Noises?
主题 Topic
导航 NavGuide
广告 Adv
Call them Noises
虽然这些信息对于人浏览 Web 有用,但常常对自动 Web 信息处理带来负面影响,比如 Web page clustering, classification, information retrieval and information extraction.
hamper automated information gathering and Web data mining,
“Template Detection via Data Mining and its Applications”
Non-Relevant Data on the Web
A fundamental problem on the Web:
“non-relevant” – not directly related to the main topic / functionality of the page
Local (intra-page) noise Irrelevant items within a Web page. E.g., banner ads, navigational guides
Many pages contain lots of non-relevant data
Duplicate data on the Web
Another problem on the Web:
Mirrors , News copy, etc, Global noise
Redundant objects Larger than individual page E.g., mirror sites, duplicated Web pages
There are much duplicate or near duplicate data
Why it influences?
Hypertext IR Principles--principles of all link based IR tools: Relevant Linkage Principle
p links to q q is relevant to p Topical Unity Principle
q1 and q2 are co-cited in p q1 and q2 are related to each other
Lexical Affinity Principle The closer the links to q1 and q2 are the
stronger the relation between them.
Violations of Relevant Linkage Principle
Navigational links http://www.ibm.com/
Download links http://www.beethoven.com/
Advertisement links http://www.yahoo.com/
Endorsement links http://www.ebay.com/
Spam links
Violations of Topical Unity Principle
Violations of the Relevant Linkage Principle
Bookmark pages http://
bookmark.yinsha.com/ 网上书签
General resource lists http://sewm.pku.edu.cn/IR-
Guide.txt IR Guide Personal homepages
http://www.cse.iitb.ac.in/~soumen/ Soumen’s Home Page
Violations of Lexical Affinity Principle
Alphabetical index lists Computer and Communication Companies ("M" entries)
HTML representation Adjacent cells in the same column are far from each
other in the HTML text
IR Tool Problems
Generalization Search for “Frequency Division Multiplexing”
and get back general Electrical Engineering sites
Topic drift Search for “Finite Model Theory” and get SF
49’ers fan web sites Irrelevance
Get “Yahoo” as a result regardless of the query Bias
Search for “computing companies” and get Microspy highly ranked
Hypertext Improvement Problem
remove violations of the Hypertext IR principles process quickly millions of pages
Develop hypertext processing techniques that:
• automatically improve hypertext data
• are efficient and scalable
Main Goal
Hypertext Cleaning
Web
Crawler
Hypertext Cleaner
IR Tool
Template detection
DOM TreeDOM Tree
模版 Template模版 Template
Templates
Templates Detection
Semantic Definition: A template is a master HTML
shell page that is used as a basis for composing new pages
Content of new pages plugged into template shell
All pages share common look & feel Usually controlled by a central
authority Not necessarily confined to a
single site May include variety of data
Navigational bars Advertisements Company info and policies
Search pagelet
Navigation pagelet
Services pagelet
Company info pagelet
Ad pagelet
Pagelets
Semantic Definition: A pagelet is a maximal region of a page
that has a single topic or functionality Not too large
has only one topic / functionality Not too small
any larger region that contains it has other topics / functionalities
IR with Pagelets
Use pagelets rather than pages as atomic units for information retrieval
Main Idea 1
Main Idea 2
Eliminate pagelets belonging to templates
Pagelets: Syntactic Definition
A pagelet is a node in the HTML parse tree of a page satisfying the following: Its HTML tag is one of the
following: <TABLE>, <OL>, <UL>,
<AREA>, <P>, <DL>, … None of it’s children
contains more than k hyperlinks
None of its ancestor is a pagelet
Templates: Syntactic Definition
A template is a collection T = (p1,…,pk) of pagelets satisfying:
Similarity:p1,…,pk are identical or almost identical
Connectivity Every two pages owning pagelets in T are
reachable from each other (undirectedely) through other pages owning pagelets in T.p1
p3
p5
p2
p4
Template Recognition Problem: Given a set of pages S find all the templates in S.
Template Recognition in Large Sets
Cluster pagelets in S according to shingle
Calculate shingle(p) for each pagelet pS
Discard clusters of size 1
For each remaining cluster C:
Construct graph Gc of pages that own pagelets in C
Find undirected connected components of Gc
Output components of size > 1
Evaluation
Question:
How to evaluate the performance/effectiveness of this cleaning algorithm?
Benefits of template detection
Cleaning via feature weighting
Cleaning via feature weightingCleaning via feature weighting
In a given Web site Noisy blocks — Share
common contents or presentation styles
Meaningful (or main) blocks — diverse in contents and presentation style
Weighting features makes cleaning automatic (nothing is eliminated)“Eliminating noisy
information in Web pages for data mining”
DOM treesDOM trees
<BODY bgcolor=WHITE> <TABLE width=800 height=200 > … </TABLE> <IMG src="image.gif" width=800> <TABLE bgcolor=RED> … </TABLE></BODY>
bc=red
bc=white
IMG TABLE
BODY
root
width=800 height=20
0TABLE
width=800
Build Site style tree (SST)
commoncommon
SST
Style Node S = (ELEMENTs, n) ELEMENTs — a sequence of element nodes n — number of pages that has this style
Element Node E = (Tag, Attr, STYLEs) Tag — tag name. E.g., TABLE, IMG; Attr — display attributes of Tag. E.g., bgcolor=RED STYLEs — style nodes below E
Quantify the importance
Inner NodeInner Node
Leaf NodeLeaf Node
Weighting policy
Inner Node Importance
(1)
l = |E.STYLEs| m = number of pages containing E, |E.parent.n| pi — percentage of tag nodes (in E.parent.n)
using the i-th presentation style Inner NodeImp(E) — diversity of presentation
styles
1
1
1
log)(
1 mif
mifppENodeImp
l
iimi
NodeImp(Body) = -1log1001 = 0NodeImp(Table)
= -(0.35log1000.35 + 2*0.25log1000.25+ 0.15log1000.15) = 0.29 >0
1
1
1
log)(
1 mif
mifppENodeImp
l
iimi
1
1
1
log)(
1 mif
mifppENodeImp
l
iimi
Weighting policy
Features( terms) of Leaf Node Importance of Leaf Node’s Features
(3)
m = number of pages containing E, |E.parent.n| pij — probability of ai appears in E of page j HE(ai) — information entropy of ai
the higher HE(ai), the less important ai
1
1log
0)(
1mif
mifppaH
m
jijmij
iE
Weighting policy
Leaf Node Importance
(2)
N — number of features in E ai — a feature of content in E (1-HE(ai)) — information contained in ai Leaf NodeImp(E) —content diversity of E
N
aH
N
aHENodeImp
N
iiE
N
iiE
11
)(1
))(1()(
Et1:PCMag,samsung
t2:PCMag,epson
t3:PCMag,canon
TABLE
Ep
SST:
IMG
root
3
m = 3N = |{PCMag, samsung,
epson, canon}| = 4HE(PCMag) =
-3 * (1/3log31/3) = 1
HE(samsung)=HE(epson)
=HE(canon) =
-(0+0+1log31) = 0NodeImp(E) = ((1-1) + 3*(1-0))/4
= 0.75
1
1log
0)(
1mif
mifppaH
m
jijmij
iE
1
1log
0)(
1mif
mifppaH
m
jijmij
iE
Transitive Weighting policy
Composite ImportanceComposite Importance
0
0.290
0.75
Page nosie
noisy element node For an element node E in the SST, if all of its
descendents and itself have composite importance less than a specified threshold t, then we say element node E is noisy.
Maximal noisy element node meaningful element node :
If an element node E in the SST does not contain any noisy descendent, we say that E is meaningful.
Maximal meaningful element node
Web page cleaning via block elimination
We can use SST (site style tree) to identify & eliminate noise content blocks in a page. Build SST by sample pages crawled from a site. Computing an importance value for each block,
using a specified threshold t to decide noisy or not noisy
Matching to noisy blocks and not noisy blocks in the tree, given a new page.
Noise Detection and Elimination
Table Img Table
Body
Table
TrTr
root
Text
Text
AP
P P P A P Img
AImg
A A A AA
After simplification
Table Img Table
Body
Table
TrTr
root
Text
Summary of the technique
Evaluate Common and Diversity of content and styles DOM trees SST Information Entropy Based Evaluation
Node Importance Composite Importance
Noise detection and automatic matching
Near duplicate detection
Syntactic clustering of the web contentsWWW6,1997
Syntactic clustering of the web contentsWWW6,1997
Document Representation
How to represent a document? Represent document content by a feature
set , preparing the computations of resemblance or similarity.
For document D, extract it’s feature set as S(D)
Defining similarity of documents
How to express the concept “roughly the same” precisely?
Quantity Definition: resemblance The resemblance fo two documents A and B is a
number between 0 and 1.
Defining similarity of documents(cont’d)
Resemblance
Symmetric, reflexive, not transitive, not a metric
Note r (A,A) = 1 But r (A,B)=1 does not mean A and B are identical!
Forgives any number of occurrences and any permutations of the terms.
Resemblance distance
)()(
)()(),(
BSAS
BSASBAr
),(1),( BArBAd
Jaccard coefficientJaccard coefficient
Feature Selection
Assume: we have converted page into a sequence of tokens Eliminate punctuation, HTML markup, lower
case, etc How to do feature selection, S(D)=?
Document level Character/word level Shingle level
Shingling
A contiguous subsequence contained in D is called a shingle.
Given a document D we define its w-shingling S(D, w) as the set of all unique shingles of size w contained in D. D = (a,rose,is,a,rose,is,a,rose) S(D,4) = {(a,rose,is,a),(rose,is,a,rose),
(is,a,rose,is)} “a rose is a rose is a rose” => a_rose_is_a rose_is_a_rose is_a_rose_is
Why shingling? S(D,4) .vs. S(D,1)What is a good
value for w?
Sketches
Set of all shingles is large Bigger than the original
document Can we create a document
sketch by sampling only a few shingles?
Requirement Sketch resemblance should be
a good estimate of document resemblance
Choosing a sketch
Random sampling E.g., suppose we have identical documents A &
B each with n shingles M(A) = set of s shingles from A, chosen
uniformly at random; similarly M(B) Does it work?
For s=1: E[|M(A) M(B)|] = 1/n But r(A,B) = 1 So the sketch overlap is an underestimate
Verify that this is true for any value of s
Choosing a sketch
Improvements: Random sampling + compare “special” item Random permutations + compare “smallest”
shingle Random permutation
Let be a set (1..N e.g.) Pick a permutation : uniformly at random
={3,7,1,4,6,2,5} A={2,3,6} MIN((A))=?
Estimating Jaccard Coefficient
Theorem : If permutations are picked uniformly at
random from the n! possible permutations,
Choosing a sketch
Create a “sketch vector” (e.g., of size 200) for each document Documents which share more than t (say 80%)
corresponding vector elements are similar For doc d, sketchd[i] is computed as follows:
Let f map all shingles in the universe to 0..2m
Let i be a specific random permutation on 0..2m
Pick MIN i (f(s)) over all shingles s in d
Computing Sketch[i] for Doc1
Document 1
264
264
264
264
Start with 64 bit shingles
Permute on the number line
with i
Pick the min value
Test if Doc1.Sketch[i] = Doc2.Sketch[i]
Test for 200 random permutations: , ,… 200
Are these equal?
Document 1
264
264
264
264A
Document 2
264
264
264
264A
However…
A = B iff the shingle with the MIN value in the union of Doc1 and Doc2 is common to both (I.e., lies in the intersection)This happens with probability: Size_of_intersection / Size_of_union
BA
Document 1
264
264
264
264A
Document 2
264
264
264
264A
Finding all near-duplicates
Naïve implementation makes O(N^2) sketch comparisons Suppose N=100 million
How can you do it faster?How can you do it faster?
本次课小结
Web Noises Hypertext IR Principles
Template Detection Semantic and Syntactic
Definition Information Entropy of
Features Weighting SST
Near duplicates detection
Jaccard similarity Shingling sketch
Document 1
264
264
264
26
4
A
Document 2
264
264
264
264A
Table
Img Table
Body
Table
TrTr
root
Text
References
[1] B.-Y. Ziv and R. Sridhar, "Template detection via data mining and its applications," in Proceedings of the 11th international conference on World Wide Web. Honolulu, Hawaii, USA: ACM Press, 2002.
[2] Y. Lan, L. Bing, and L. Xiaoli, "Eliminating noisy information in Web pages for data mining," in Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining. Washington, D.C.: ACM Press, 2003.
[3] G. David, P. Kunal, and T. Andrew, "The volume and evolution of web page templates," in Special interest tracks and posters of the 14th international conference on World Wide Web. Chiba, Japan: ACM Press, 2005.
[4] Z. B. Andrei, C. G. Steven, S. M. Mark, and Z. Geoffrey, "Syntactic clustering of the Web," in Selected papers from the sixth international conference on World Wide Web. Santa Clara, California, United States: Elsevier Science Publishers Ltd., 1997.
[5] N. Shivakumar and H. Garca-Molina, "Finding near-replicas of documents on the web," presented at Proceedings of Workshop on Web Databases (WebDB'98), Mar, 1998.
Related Resources
Html-tidy Code http://code.google.com/p/html-tidy/
Shingle Codehttp://research.microsoft.com/research/downloads/Details/4e0d0535-ff4c-4259-99fa-ab34f3f57d67/Details.aspx?0sr=d
Thank You!
Q&A
阅读材料
[1] IIR Chapter 19.6 [2] G. Salton and C. Buckley, "Term-
weighting approaches in automatic text retrieval," Inf. Process. Manage., vol. 24, pp. 523, 1988.
DOM Tree
W3C Document Object Model allow programs and scripts to dynamically access
and update the content, structure and style of documents. The document can be further processed and the results of that processing can be incorporated back into the presented page.
Information Entropy
In information theory, entropy is a measure of the uncertainty associated with a random variable. The term by itself in this context usually refers to the Shannon entropy, which quantifies, in the sense of an expected value, the information contained in a message, usually in units such as bits.
Estimating algorithm
1. Generate a set of m random permutations
2. for each do3. compute and 4. check if5. end for6. if equality was observed in k cases,
estimate
m
kddr ),(' 21
))((min))((min 21 dSdS ))(( 1dS ))(( 2dS
Some other approaches
For set W of shingles, let MINs(W) = set of s smallest shingles in W
Assume documents have at least s shingles Define
M(A) = MINs(S(A)) M(AB) = MINs(M(A) M(B)) r’(A,B) = |M(AB) M(A) M(B)| / s
By increasing sample size (s) we can make it very unlikely r’(A,B) is significantly different from r(A,B)
100-200 shingles is sufficient in practice Compute a fingerprint f for each shingle (e.g.,
Rabin fingerprint) 40 bits is usually enough to keep estimates reasonably
accurate Fingerprint also eliminates need for random permutation
Finding all near-duplicates
Naïve implementation makes O(N^2) sketch comparisons Suppose N=100 million
Divide-Compute-Merge (DCM) Divide data into batches that fit in memory Operate on individual batch and write out
partial results in sorted order Merge partial results
Generalization of external sorting
doc1: s11,s12,…,s1kdoc2: s21,s22,…,s2k…
DCM Steps
s11,doc1s12,doc1…s1k,doc1s21,doc2…
Invertt1,doc1t1,docX…t2,doc1t2,docY…
sort onshingle_fp
doc1,docX,1doc1,docZ,1…doc1,docY,1…
Invert and pair
doc1,docX,1doc1,docX,1…doc1,docY,1…
sort on<docid1,docid2>
doc1,docX,2doc1,docY,10…
Merge
Finding all near-duplicates
1. Calculate a sketch for each document2. For each document, write out the pairs <shingle_fp,
docId>3. Sort by shingle_fp (DCM)4. In a sequential scan, generate triplets of the form
<docId1,docId2,1> for pairs of docs that share a shingle (DCM)
5. Sort on <docId1,docId2> (DCM)6. Merge the triplets with common docids to generate
triplets of the form <docId1,docId2,count> (DCM)7. Output document pairs whose resemblance exceeds
the threshold
DCM algorithm
1. for each random permutation do2. create a file3. for each document d do4. write out to 5. end for6. sort using key s -- this results in contiguous blocks with
fixed s containing all associated
7. create a file8. for each pair within a run of having a given s do9. write out a document-pair record to10. end for11. sort on key 12. end for13. merge for all in order, counting the number of
entries
),( 21 dd
sd
ddSs )),((min
f
f
f
g
f
),( 21 dd
g ),( 21 dd
g ),( 21 dd ),( 21 dd
g
Some optimizations
“Invert and Pair” is the most expensive step
We can speed it up eliminating very common shingles Common headers, footers, etc. Do it as a preprocessing step
Also, eliminate exact duplicates up front Probabilistic Counting [5]
Detecting duplicate pages
State of the art Technology
Volume and Evolution of Page Templates
Our results show that 40–50% of the content on the web is template content.
Over the last eight years, the fraction of template content has doubled, and the growth shows no sign of abating.
Text, links, and total HTML bytes within templates are all growing as a fraction of total content at a rate of between 6 and 8% per year. Question:
how to design the experiment to reach these conclusions?
Question: how to design the experiment to
reach these conclusions?