Εύρεση ∆ιαχείριση Πληροφορίαςστον...

1

Εύρεση & ∆ιαχείρισηΠληροφορίας στονΠαγκόσµιο Ιστό

∆ιδάσκων –∆ηµήτριος Κατσαρός, Ph.D.

@ Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύωνΠανεπιστήµιο Θεσσαλίας

∆ιάλεξη 7η: 17/04/2007

2

Web characteristics

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 3

Search use …(iProspect Survey, 4/04, http://www.iprospect.com/premiumPDFs/iProspectSurveyComplete.pdf)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 4

Without search engines the web wouldn’t scale

1. No incentive in creating content unless it can be easily found –other finding methods haven’t kept pace (taxonomies, bookmarks, etc)

2. The web is both a technology artifact and a social environment • “The Web has become the “new normal” in the American way of

life; those who don’t go online constitute an ever-shrinking minority.” – [Pew Foundation report, January 2005]

3. Search engines make aggregation of interest possible: • Create incentives for very specialized niche players

• Economical – specialized stores, providers, etc • Social – narrow interests, specialized communities, etc

4. The acceptance of search interaction makes “unlimited selection”stores possible:• Amazon, Netflix, etc

5. Search turned out to be the best mechanism for advertising on the web, a $15+ B industry. • Growing very fast but entire US advertising industry $250B –

huge room to grow• Sponsored search marketing is about $10B

5

Classical IR vs. Web IR

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 6

Basic assumptions of Classical Information Retrieval

• Corpus: Fixed document collection• Goal: Retrieve documents with information

content that is relevant to user’s information need

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 7

Classic IR Goal

•Classic relevance• For each query Q and stored document D in a given corpus

assume there exists relevance Score(Q, D)• Score is average over users U and contexts C

• Optimize Score(Q, D) as opposed to Score(Q, D, U, C)• That is, usually:

• Context ignored• Individuals ignored• Corpus predetermined

Bad assumptionsin the web context

8

Web IR

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 9

The coarse-level dynamics

Content creators Content aggregators

Feeds

Crawls

Content consumers

Adv

ertis

emen

tEd

itoria

l

Subs

crip

tion

Tran

sact

ion

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 10

Brief (non-technical) history• Early keyword-based engines

• Altavista, Excite, Infoseek, Inktomi, ca. 1995-1997• Paid placement ranking: Goto.com (morphed into

Overture.com → Yahoo!)• Your search ranking depended on how much you paid• Auction for keywords: casino was expensive!

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 11

Brief (non-technical) history• 1998+: Link-based ranking pioneered by Google

• Blew away all early engines save Inktomi• Great user experience in search of a business model• Meanwhile Goto/Overture’s annual revenues were

nearing $1 billion• Result: Google added paid-placement “ads” to the

side, independent of search results• Yahoo follows suit, acquiring Overture (for paid

placement) and Inktomi (for search)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 12

Algorithmic results.

Ads

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 13

Ads vs. search results• Google has maintained that ads

(based on vendors bidding for keywords) do not affect vendors’rankings in search results

Web Results 1 - 10 of about 7,310,000 for miele. (0.12 seconds)

Miele, Inc -- Anything else is a compromise At the heart of your home, Appliances by Miele. ... USA. to miele.com. Residential Appliances. Vacuum Cleaners. Dishwashers. Cooking Appliances. Steam Oven. Coffee System ... www.miele.com/ - 20k - Cached - Similar pages

Miele Welcome to Miele, the home of the very best appliances and kitchens in the world. www.miele.co.uk/ - 3k - Cached - Similar pages

Miele - Deutscher Hersteller von Einbaugeräten, Hausgeräten ... - [ Translate this page ] Das Portal zum Thema Essen & Geniessen online unter www.zu-tisch.de. Miele weltweit ...ein Leben lang. ... Wählen Sie die Miele Vertretung Ihres Landes. www.miele.de/ - 10k - Cached - Similar pages

Herzlich willkommen bei Miele Österreich - [ Translate this page ] Herzlich willkommen bei Miele Österreich Wenn Sie nicht automatisch weitergeleitet werden, klicken Sie bitte hier! HAUSHALTSGERÄTE ... www.miele.at/ - 3k - Cached - Similar pages

Sponsored Links

CG Appliance Express Discount Appliances (650) 756-3931Same Day Certified Installation www.cgappliance.com San Francisco-Oakland-San Jose, CA Miele Vacuum Cleaners Miele Vacuums- Complete SelectionFree Shipping! www.vacuums.com Miele Vacuum Cleaners Miele-Free Air shipping! All models. Helpful advice. www.best-vacuum.com

Search =miele

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 14

Ads vs. search results

• Other vendors (Yahoo, MSN) have made similar statements from time to time• Any of them can change anytime

• We will focus primarily on search results independent of paid placement ads• Although the latter is a fascinating technical

subject in itself

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 15

Web search basics

The Web

Ad indexes

Web Results 1 - 10 of about 7,310,000 for miele. (0.12 seconds)

Miele, Inc -- Anything else is a compromise At the heart of your home, Appliances by Miele. ... USA. to miele.com. Residential Appliances. Vacuum Cleaners. Dishwashers. Cooking Appliances. Steam Oven. Coffee System ... www.miele.com/ - 20k - Cached - Similar pages

Miele Welcome to Miele, the home of the very best appliances and kitchens in the world. www.miele.co.uk/ - 3k - Cached - Similar pages

Miele - Deutscher Hersteller von Einbaugeräten, Hausgeräten ... - [ Translate this page ] Das Portal zum Thema Essen & Geniessen online unter www.zu-tisch.de. Miele weltweit ...ein Leben lang. ... Wählen Sie die Miele Vertretung Ihres Landes. www.miele.de/ - 10k - Cached - Similar pages

Herzlich willkommen bei Miele Österreich - [ Translate this page ] Herzlich willkommen bei Miele Österreich Wenn Sie nicht automatisch weitergeleitet werden, klicken Sie bitte hier! HAUSHALTSGERÄTE ... www.miele.at/ - 3k - Cached - Similar pages

Sponsored Links

CG Appliance Express Discount Appliances (650) 756-3931 Same Day Certified Installation www.cgappliance.com San Francisco-Oakland-San Jose, CA Miele Vacuum Cleaners Miele Vacuums- Complete Selection Free Shipping! www.vacuums.com Miele Vacuum Cleaners Miele-Free Air shipping! All models. Helpful advice. www.best-vacuum.com

Web spider

Indexer

Indexes

Search

User

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 16

User Needs

• Need [Brod02, RL04]• Informational – want to learn about something (~40% / 65%)

• Navigational – want to go to that page (~25% / 15%)

• Transactional – want to do something (web-mediated) (~35% / 20%)• Access a service• Downloads

• Shop• Gray areas

• Find a good hub• Exploratory search “see what’s there”

Low hemoglobin

United Airlines

Seattle weatherMars surface images

Canon S410

Car rental Brasil

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 17

Web search users• Make ill defined

queries• Short

• AV 2001: 2.54 terms avg, 80% < 3 words)

• AV 1998: 2.35 terms avg, 88% < 3 words [Silv98]

• Imprecise terms• Sub-optimal syntax (most

queries without operator)• Low effort

• Wide variance in• Needs• Expectations• Knowledge• Bandwidth

• Specific behavior• 85% look over one

result screen only (mostly above the fold)

• 78% of queries are not modified (one query/session)

• Follow links –“the scent of

information” ...

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 18

Query Distribution

Power law: few popular broad queries, many rare specific queries

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 19

How far do people look for results?

(Source: iprospect.com WhitePaper_2006_SearchEngineUserBehavior.pdf)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 20

True example*

Corpus

TASK

Info Need

Query

Verbal form

Results

SEARCHENGINE

QueryRefinement

Noisy building fan in courtyard

Info about EPA regulations

What are the EPA rules about noise pollution

EPA sound pollution

Mis-conception

Mis-translation

Mis-formulation

PolysemySynonimy

* To Google or to GOTO, Business Week Online, September 28, 2001

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 21

Users’ empirical evaluation of results

• Quality of pages varies widely• Relevance is not enough• Other desirable qualities (non IR!!)

• Content: Trustworthy, new info, non-duplicates, well maintained,• Web readability: display correctly & fast• No annoyances: pop-ups, etc

• Precision vs. recall• On the web, recall seldom matters

• What matters• Precision at 1? Precision above the fold?• Comprehensiveness – must be able to deal with obscure queries

• Recall matters when the number of matches is very small• User perceptions may be unscientific, but are

significant over a large aggregate

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 22

Users’ empirical evaluation of engines• Relevance and validity of results• UI – Simple, no clutter, error tolerant• Trust – Results are objective• Coverage of topics for poly-semic queries• Pre/Post process tools provided

• Mitigate user errors (auto spell check, syntax errors,…)• Explicit: Search within results, more like this, refine ...• Anticipative: related searches

• Deal with idiosyncrasies• Web specific vocabulary

• Impact on stemming, spell-check, etc• Web addresses typed in the search box• …

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 23

Loyalty to a given search engine(iProspect Survey, 4/04)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 24

The Web corpus

• No design/co-ordination• Distributed content creation, linking,

democratization of publishing• Content includes truth, lies, obsolete

information, contradictions …• Unstructured (text, html, …), semi-

structured (XML, annotated photos), structured (Databases)…

• Scale much larger than previous text corpora … but corporate records are catching up.

• Growth – slowed down from initial “volume doubling every few months”but still expanding

• Content can be dynamically generatedThe Web

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 25

The Web: Dynamic content

• A page without a static html version• E.g., current status of flight AA129• Current availability of rooms at a hotel

• Usually, assembled at the time of a request from a browser• Typically, URL has a ‘?’ character in it

Application server

Browser

AA129

Back-enddatabases

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 26

Dynamic content

• Most dynamic content is ignored by web spiders• Many reasons including malicious spider traps

• Some dynamic content (news stories from subscriptions) are sometimes delivered as dynamic content• Application-specific spidering

• Spiders commonly view web pages just as Lynx (a text browser) would

• Note: even “static” pages are typically assembled on the fly (e.g., headers are common)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 27

The web: size

• What is being measured?• Number of hosts• Number of (static) html pages

• Volume of data

• Number of hosts – netcraft survey• http://news.netcraft.com/archives/web_server_survey.html• Monthly report on how many web hosts & servers are out there

• Number of pages – numerous estimates (will discuss later)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 28

Netcraft Web Server Surveyhttp://news.netcraft.com/archives/web_server_survey.html

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 29

The web: evolution

• All of these numbers keep changing• Relatively few scientific studies of the

evolution of the web [Fetterly & al, 2003]• http://research.microsoft.com/research/sv/sv-pubs/p97-

fetterly/p97-fetterly.pdf

• Sometimes possible to extrapolate from small samples (fractal models) [Dill & al, 2001]• http://www.vldb.org/conf/2001/P069.pdf

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 30

Rate of change• [Cho00] 720K pages from 270 popular sites sampled

daily from Feb 17 – Jun 14, 1999 • Any changes: 40% weekly, 23% daily

• [Fett02] Massive study 151M pages checked over few months• Significant changed -- 7% weekly• Small changes – 25% weekly

• [Ntul04] 154 large sites re-crawled from scratch weekly• 8% new pages/week • 8% die• 5% new content• 25% new links/week

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 31

Static pages: rate of change• Fetterly et al. study (2002): several views of data, 150 million pages

over 11 weekly crawls• Bucketed into 85 groups by extent of change

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 32

Other characteristics

• Significant duplication• Syntactic – 30%-40% (near) duplicates [Brod97, Shiv99b, etc.]• Semantic – ???

• High linkage • More than 8 links/page in the average

• Complex graph topology• Not a small world; bow-tie structure [Brod00]

• Spam• Billions of pages

33

Spam

Search Engine Optimization

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 34

The trouble with paid placement…

• It costs money. What’s the alternative?• Search Engine Optimization:

• “Tuning” your web page to rank highly in the search results for select keywords

• Alternative to paying for placement• Thus, intrinsically a marketing function

• Performed by companies, webmasters and consultants (“Search engine optimizers”) for their clients

• Some perfectly legitimate, some very shady

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 35

Simplest forms• First generation engines relied heavily on tf/idf

• The top-ranked pages for the query maui resort were the ones containing the most maui’s and resort’s

• SEOs responded with dense repetitions of chosen terms• e.g., maui resort maui resort maui resort• Often, the repetitions would be in the same color as the

background of the web page• Repeated terms got indexed by crawlers• But not visible to humans on browsers

Pure word density cannot be trusted as an IR signal

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 36

Variants of keyword stuffing• Misleading meta-tags, excessive repetition• Hidden text with colors, style sheet tricks,

etc.

Meta-Tags = “… London hotels, hotel, holiday inn, hilton, discount, booking, reservation, sex, mp3, britney spears, viagra, …”

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 37

Search engine optimization (Spam)

• Motives• Commercial, political, religious, lobbies• Promotion funded by advertising budget

• Operators• Contractors (Search Engine Optimizers) for lobbies, companies• Web masters• Hosting services

• Forums• E.g., Web master world ( www.webmasterworld.com )

• Search engine specific tricks • Discussions about academic papers ☺

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 38

Cloaking• Serve fake content to search engine spider• DNS cloaking: Switch IP address. Impersonate

Is this a SearchEngine spider?

Y

N

SPAM

RealDocCloaking

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 39

The spam industry

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 40

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 41

More spam techniques

• Doorway pages• Pages optimized for a single keyword that re-direct to the

real target page• Link spamming

• Mutual admiration societies, hidden links, awards – more on these later

• Domain flooding: numerous domains that point or re-direct to a target page

• Robots• Fake query stream – rank checking programs

• “Curve-fit” ranking programs of search engines• Millions of submissions via Add-Url

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 42

The war against spam

• Quality signals - Prefer authoritative pages based on:

• Votes from authors (linkage signals)

• Votes from users (usage signals)• Policing of URL

submissions• Anti robot test

• Limits on meta-keywords• Robust link analysis

• Ignore statistically implausible linkage (or text)

• Use link analysis to detect spammers (guilt by association)

• Spam recognition by machine learning

• Training set based on known spam

• Family friendly filters• Linguistic analysis, general

classification techniques, etc.• For images: flesh tone

detectors, source text analysis, etc.

• Editorial intervention• Blacklists• Top queries audited• Complaints addressed• Suspect pattern detection

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 43

More on spam• Web search engines have policies on SEO practices

they tolerate/block• http://help.yahoo.com/help/us/ysearch/index.html• http://www.google.com/intl/en/webmasters/

• Adversarial IR: the unending (technical) battle between SEO’s and web search engines

• Research http://airweb.cse.lehigh.edu/

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 44

Answering “the need behind the query”

• Semantic analysis• Query language determination

• Auto filtering • Different ranking (if query in Japanese do not return English)

• Hard & soft (partial) matches• Personalities (triggered on names)• Cities (travel info, maps)• Medical info (triggered on names and/or results)• Stock quotes, news (triggered on stock symbol)• Company info• Etc.

• Natural Language reformulation• Integration of Search and Text Analysis

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 45

The spatial context -- geo-search• Two aspects• Geo-coding -- encode geographic coordinates to make search

effective• Geo-parsing -- the process of identifying geographic context.

• Geo-coding• Geometrical hierarchy (squares)• Natural hierarchy (country, state, county, city, zip-codes, etc)• Geo-parsing• Pages (infer from phone nos, zip, etc). About 10% can be parsed.• Queries (use dictionary of place names) • Users

• Explicit (tell me your location -- used by NL, registration, from ISP)• From IP data

• Mobile phones • In its infancy, many issues (display size, privacy, etc)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 46

Yahoo!: britney spears

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 47

Ask Jeeves: las vegas

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 48

Yahoo!: salvador hotels

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 49

Yahoo shortcuts• Various types of queries that are “understood”

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 50

Google andrei broder new york

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 51

Answering “the need behind the query”: Context

• Context determination • spatial (user location/target location)• query stream (previous queries)• personal (user profile) • explicit (user choice of a vertical search, ) • implicit (use Google from France, use google.fr)

• Context use• Result restriction

• Kill inappropriate results• Ranking modulation

• Use a “rough” generic ranking, but personalize later

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 52

Google: dentists bronx

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 53

Yahoo!: dentists (bronx)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 54

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 55

Query expansion

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 56

Context transfer

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 57

No transfer

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 58

Context transfer

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 59

Transfer from search results

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 60

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 61

Resources• IIR Chapter 19

62

Web characteristics II:Web size measurementNear-duplicate detection

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 63

Today’s topics

• Estimating web size and search engine index size

• Near-duplicate document detection

64

Size of the web

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 65

What is the size of the web ?• Issues

• The web is really infinite • Dynamic content, e.g., calendar • Soft 404: www.yahoo.com/<anything> is a valid page

• Static web contains syntactic duplication, mostly due to mirroring (~30%)

• Some servers are seldom connected• Who cares?

• Media, and consequently the user• Engine design• Engine crawl policy. Impact on recall.

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 66

What can we attempt to measure?•The relative sizes of search engines • The notion of a page being indexed is still reasonably well

defined.• Already there are problems

• Document extension: e.g. engines index pages not yet crawled, by indexing anchortext.

• Document restriction: All engines restrict what is indexed (firstn words, only relevant words, etc.)

•The coverage of a search engine relative to another particular crawling process.

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 67

New definition?

• (IQ is whatever the IQ tests measure.)• The statically indexable web is whatever

search engines index.• Different engines have different preferences

• max url depth, max count/host, anti-spam rules, priority rules, etc.

• Different engines index different things under the same URL:• frames, meta-keywords, document restrictions, document

extensions, ...

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 68

Statistical methods• Random queries

• Random searches

• Random IP addresses

• Random walks

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 69

A ∩ B = (1/2) * Size A

A ∩ B = (1/6) * Size B

(1/2)*Size A = (1/6)*Size B

∴ Size A / Size B =

(1/6)/(1/2) = 1/3

Sample URLs randomly from A

Check if contained in B

and vice versa

A ∩ B

Each test involves: (i) Sampling (ii) Checking

Relative Size from Overlap [Bharat & Broder, 98]

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 70

Sampling URLs

• Ideal strategy: Generate a random URL and check for containment in each index.

• Problem: Random URLs are hard to find! Enough to generate a random URL contained in a given Engine.

Key lesson from this lecture.

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 71

Random URLs from random queries [Bharat & B, 98]

• Generate random query: how?• Lexicon: 400,000+ words from a crawl of Yahoo!• Conjunctive Queries: w1 and w2

e.g., vocalists AND rsi

• Get 100 result URLs from the source engine• Choose a random URL as the candidate to

check for presence in other engines.• This distribution induces a probability weight

W(p) for each page. • Conjecture:

W(SE1) / W(SE2) ~ |SE1| / |SE2|

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 72

Query Based Checking

• Strong Query to check for a document D:• Download document. Get list of words. • Use 8 low frequency words as AND query

• Check if D is present in result set.• Problems:

• Near duplicates• Frames• Redirects• Engine time-outs• Might be better to use e.g. 5 distinct conjunctive

queries of 6 words each.

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 73

Computing Relative Sizes and Total Coverage [BB98]

• Six pair-wise overlapsfah* a - fha* h = ε1fai* a - fia* i = ε2fae* a - fea* e = ε3fhi* h - fih* i = ε4fhe* h - feh* e = ε5fei* e - fie* i = ε6

• Arbitrarily, let a = 1.

• We have 6 equations and 3 unknowns.

• Solve for e, h and i to minimize Σ εi2

• Compute engine overlaps.

• Re-normalize so that the total joint coverage is 100%

a = AltaVista, e = Excite, h = HotBot, i = Infoseek

fxy = fraction of x in y

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 74

Advantages & disadvantages

• Statistically sound under the induced weight.• Biases induced by random query

• Query Bias: Favors content-rich pages in the language(s) of the lexicon

• Ranking Bias: Solution: Use conjunctive queries & fetch all• Checking Bias: Duplicates, impoverished pages omitted• Document or query restriction bias: engine might not deal

properly with 8 words conjunctive query• Malicious Bias: Sabotage by engine• Operational Problems: Time-outs, failures, engine

inconsistencies, index modification.

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 75

Random searches• Choose random searches extracted from a local

log [Lawrence & Giles 97] or build “random searches” [Notess]• Use only queries with small results sets. • Count normalized URLs in result sets.• Use ratio statistics

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 76

Advantages & disadvantages

• Advantage• Might be a better reflection of the human perception of

coverage

• Issues• Samples are correlated with source of log• Duplicates• Technical statistical problems (must have non-zero

results, ratio average, use harmonic mean?)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 77

Random searches [Lawr98, Lawr99]

• 575 & 1050 queries from the NEC RI employee logs• 6 Engines in 1998, 11 in 1999• Implementation:

• Restricted to queries with < 600 results in total• Counted URLs from each engine after verifying query match• Computed size ratio & overlap for individual queries • Estimated index size ratio & overlap by averaging over all

queries

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 78

• adaptive access control • neighborhood preservation

topographic • hamiltonian structures • right linear grammar • pulse width modulation

neural • unbalanced prior probabilities • ranked assignment method • internet explorer favourites

importing • karvel thornber• zili liu

Queries from Lawrence and Giles study

• softmax activation function • bose multidimensional system

theory • gamma mlp• dvi2pdf • john oliensis• rieke spikes exploring neural • video watermarking • counterpropagation network • fat shattering dimension • abelson amorphous computing

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 79

Random IP addresses [Lawrence & Giles ‘99]• Generate random IP addresses• Find a web server at the given address

• If there’s one• Collect all pages from server.• Method first used by O’Neill, McClain, & Lavoie,

“A Methodology for Sampling the World Wide Web”, 1997. http://digitalarchive.oclc.org/da/ViewObject.jsp?objid=0000003447

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 80

Random IP addresses [ONei97, Lawr99]

• HTTP requests to random IP addresses • Ignored: empty or authorization required or excluded• [Lawr99] Estimated 2.8 million IP addresses running

crawlable web servers (16 million total) from observing 2500 servers.

• OCLC using IP sampling found 8.7 M hosts in 2001• Netcraft [Netc02] accessed 37.2 million hosts in July 2002

• [Lawr99] exhaustively crawled 2500 servers and extrapolated• Estimated size of the web to be 800 million• Estimated use of metadata descriptors:

• Meta tags (keywords, description) in 34% of home pages, Dublin core metadata in 0.3%

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 81

Advantages & disadvantages

• Advantages• Clean statistics• Independent of crawling strategies

• Disadvantages• Doesn’t deal with duplication • Many hosts might share one IP, or not accept requests• No guarantee all pages are linked to root page.

• Eg: employee pages • Power law for # pages/hosts generates bias towards sites

with few pages.• But bias can be accurately quantified IF underlying

distribution understood• Potentially influenced by spamming (multiple IP’s for

same server to avoid IP block)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 82

Random walks[Henzinger et al WWW9]

• View the Web as a directed graph• Build a random walk on this graph

• Includes various “jump” rules back to visited sites• Does not get stuck in spider traps!• Can follow all links!

• Converges to a stationary distribution• Must assume graph is finite and independent of the walk. • Conditions are not satisfied (cookie crumbs, flooding)• Time to convergence not really known

• Sample from stationary distribution of walk• Use the “strong query” method to check coverage by SE

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 83

Dependence on seed list• How well connected is the graph? [Broder et al.,

WWW9]

•

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 84

Advantages & disadvantages• Advantages

• “Statistically clean” method at least in theory!• Could work even for infinite web (assuming convergence)

under certain metrics.• Disadvantages

• List of seeds is a problem.• Practical approximation might not be valid.• Non-uniform distribution

• Subject to link spamming

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 85

Conclusions• No sampling solution is perfect. • Lots of new ideas ...• ....but the problem is getting harder• Quantitative studies are fascinating and a good

research problem

86

Duplicate detection

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 87

Duplicate documents

• The web is full of duplicated content• Strict duplicate detection = exact match

• Not as common• But many, many cases of near duplicates

• E.g., Last modified date the only difference

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 88

Duplicate/Near-Duplicate Detection

• Duplication: Exact match can be detected with fingerprints

• Near-Duplication: Approximate match• Overview

• Compute syntactic similarity with an edit-distance measure• Use similarity threshold to detect near-duplicates

• E.g., Similarity > 80% => Documents are “near duplicates”• Not transitive though sometimes used transitively

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 89

Computing Similarity• Features:

• Segments of a document (natural or artificial breakpoints)• Shingles (Word N-Grams)• a rose is a rose is a rose →

a_rose_is_arose_is_a_rose

is_a_rose_is • Similarity Measure between two docs (= sets of shingles)

• Set intersection [Brod98](Specifically, Size_of_Intersection / Size_of_Union )

Jaccard measure

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 90

Shingles + Set Intersection

• Computing exact set intersection of shingles between all pairs of documents is expensive/intractable• Approximate using a cleverly chosen subset of shingles from

each (a sketch)• Estimate (size_of_intersection / size_of_union)based on a short sketch

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 91

Sketch of a document

• Create a “sketch vector” (of size ~200) for each document• Documents that share ≥ t (say 80%)

corresponding vector elements are near duplicates

• For doc D, sketchD[ i ] is as follows:• Let f map all shingles in the universe to 0..2m (e.g., f =

fingerprinting)• Let πi be a random permutation on 0..2m

• Pick MIN {πi(f(s))} over all shingles s in D

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 92

Computing Sketch[i] for Doc1

Document 1

264

264

264

264

Start with 64-bit f(shingles)

Permute on the number line

with πi

Pick the min value

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 93

Test if Doc1.Sketch[i] = Doc2.Sketch[i]

Document 1 Document 2

264

264

264

264

264

264

264

264

Are these equal?

Test for 200 random permutations: π1, π2,… π200

A B

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 94

However…

Document 1 Document 2

264

264

264

264

264

264

264

264

A = B iff the shingle with the MIN value in the union of Doc1 and Doc2 is common to both (I.e., lies in the intersection)

This happens with probability:Size_of_intersection / Size_of_union

BA

Why?

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 95

Set Similarity• Set Similarity (Jaccard measure)

• View sets as columns of a matrix; one row for each element in the universe. aij = 1 indicates presence of item i in set j

• Example

ji

jiji CC

CC)C,Jaccard(C

U

I=

C1 C2

0 11 0

1 1 Jaccard(C1,C2) = 2/5 = 0.40 01 10 1

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 96

Key Observation• For columns Ci, Cj, four types of rows

Ci Cj

A 1 1B 1 0C 0 1D 0 0

• Overload notation: A = # of rows of type A• Claim

CBAA)C,Jaccard(C ji ++

=

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 97

“Min” Hashing• Randomly permute rows• Hash h(Ci) = index of first row with 1 in column Ci• Surprising Property

• Why?• Both are A/(A+B+C)• Look down columns Ci, Cj until first non-Type-D row• h(Ci) = h(Cj) type A row

[ ] ( )jiji C,CJaccard )h(C)h(C P ==

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 98

Min-Hash sketches• Pick P random row permutations • MinHash sketch

SketchD = list of P indexes of first rows with 1 in column C

• Similarity of signatures • Let sim[sketch(Ci),sketch(Cj)] = fraction of permutations

where MinHash values agree • Observe E[sim(sig(Ci),sig(Cj))] = Jaccard(Ci,Cj)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 99

Example

C1 C2 C3R1 1 0 1R2 0 1 1R3 1 0 0R4 1 0 1R5 0 1 0

SignaturesS1 S2 S3

Perm 1 = (12345) 1 2 1Perm 2 = (54321) 4 5 4Perm 3 = (34512) 3 5 4

Similarities1-2 1-3 2-3

Col-Col 0.00 0.50 0.25Sig-Sig 0.00 0.67 0.00

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 100

Implementation Trick• Permuting rows even once is prohibitive• Row Hashing

• Pick P hash functions hk: {1,…,n} {1,…,O(n)}• Ordering under hk gives random row permutation

• One-pass Implementation• For each Ci and hk, keep “slot” for min-hash value• Initialize all slot(Ci,hk) to infinity• Scan rows in arbitrary order looking for 1’s

• Suppose row Rj has 1 in column Ci• For each hk,

• if hk(j) < slot(Ci,hk), then slot(Ci,hk) hk(j)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 101

Example

C1 C2R1 1 0R2 0 1R3 1 1R4 1 0R5 0 1

h(x) = x mod 5g(x) = 2x+1 mod 5

h(1) = 1 1 -g(1) = 3 3 -

h(2) = 2 1 2g(2) = 0 3 0

h(3) = 3 1 2g(3) = 2 2 0

h(4) = 4 1 2g(4) = 4 2 0

h(5) = 0 1 0g(5) = 1 2 0

C1 slots C2 slots

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 102

Comparing Signatures• Signature Matrix S

• Rows = Hash Functions• Columns = Columns• Entries = Signatures

• Compute – Pair-wise similarity of signature columns

• Problem• MinHash fits column signatures in memory• But comparing signature-pairs takes too much time

• Technique to limit candidate pairs?• Locality Sensitive Hashing (LSH)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 103

Resources

• IIR 19• See also

• Phelps & Wilensky. Robust Hyperlinks & Locations, 2002

• Ziv Bar-Yossef and Maxim Gurevich. Random Sampling from a Search Engine’s Index, WWW 2006.

• Broder et al. Estimating corpus size via queries. CIKM 2006.

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 104

More resources• Related papers:

• [Bar Yossef & al, VLDB 2000], [Rusmevichientong & al, 2001], [Bar Yossef & al, 2003]

105

Crawling and web indexes

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 106

Today’s lecture• Crawling• Connectivity servers

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 107

Basic crawler operation• Begin with known “seed” pages• Fetch and parse them

•Extract URLs they point to•Place the extracted URLs on a

queue• Fetch each URL on the queue and

repeat

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 108

Crawling picture

Web

URLs crawledand parsed

URLs frontier

Unseen Web

Seedpages

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 109

Simple picture – complications• Web crawling isn’t feasible with one machine

• All of the above steps distributed• Even non-malicious pages pose challenges

• Latency/bandwidth to remote servers vary• Webmasters’ stipulations

• How “deep” should you crawl a site’s URL hierarchy?• Site mirrors and duplicate pages

• Malicious pages• Spam pages • Spider traps – incl dynamically generated

• Politeness – don’t hit a server too often

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 110

What any crawler must do

• Be Polite: Respect implicit and explicit politeness considerations for a website• Only crawl pages you’re allowed to• Respect robots.txt (more on this shortly)

• Be Robust: Be immune to spider traps and other malicious behavior from web servers

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 111

What any crawler should do

• Be capable of distributed operation: designed to run on multiple distributed machines

• Be scalable: designed to increase the crawl rate by adding more machines

• Performance/efficiency: permit full use of available processing and network resources

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 112

What any crawler should do

• Fetch pages of “higher quality” first• Continuous operation: Continue

fetching fresh copies of a previously fetched page

• Extensible: Adapt to new data formats, protocols

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 113

Updated crawling picture

URLs crawledand parsed

Unseen Web

SeedPages

URL frontier

Crawling thread

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 114

URL frontier

• Can include multiple pages from the same host

• Must avoid trying to fetch them all at the same time

• Must try to keep all crawling threads busy

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 115

Explicit and implicit politeness

• Explicit politeness: specifications from webmasters on what portions of site can be crawled• robots.txt

• Implicit politeness: even with no specification, avoid hitting any site too often

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 116

Robots.txt

• Protocol for giving spiders (“robots”) limited access to a website, originally from 1994• www.robotstxt.org/wc/norobots.html

• Website announces its request on what can(not) be crawled• For a URL, create a file URL/robots.txt• This file specifies access restrictions

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 117

Robots.txt example• No robot should visit any URL starting with

"/yoursite/temp/", except the robot called “searchengine":

User-agent: *Disallow: /yoursite/temp/

User-agent: searchengine

Disallow:

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 118

Processing steps in crawling• Pick a URL from the frontier• Fetch the document at the URL• Parse the URL

• Extract links from it to other docs (URLs)• Check if URL has content already seen

• If not, add to indexes• For each extracted URL

• Ensure it passes certain URL filter tests• Check if it is already in the frontier (duplicate URL

elimination)

E.g., only crawl .edu, obey robots.txt, etc.

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 119

Basic crawl architecture

WWW

Fetch

DNS

Parse

Contentseen?

URLfilter

DupURLelim

DocFP’s

URLset

URL Frontier

robotsfilters

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 120

DNS (Domain Name Server)• A lookup service on the internet

• Given a URL, retrieve its IP address• Service provided by a distributed set of servers – thus,

lookup latencies can be high (even seconds)• Common OS implementations of DNS lookup are

blocking: only one outstanding request at a time• Solutions

• DNS caching• Batch DNS resolver – collects requests and sends them

out together

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 121

Parsing: URL normalization• When a fetched document is parsed, some of

the extracted links are relative URLs• E.g., at

http://en.wikipedia.org/wiki/Main_Pagewe have a relative link to

/wiki/Wikipedia:General_disclaimer which is the same as the absolute URLhttp://en.wikipedia.org/wiki/Wikipedia:General_disclaimer

• During parsing, must normalize (expand) such relative URLs

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 122

Content seen?

• Duplication is widespread on the web

• If the page just fetched is already in the index, do not further process it

• This is verified using document fingerprints or shingles

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 123

Filters and robots.txt• Filters – regular expressions for URL’s

to be crawled/not• Once a robots.txt file is fetched from a

site, need not fetch it repeatedly• Doing so burns bandwidth, hits web

server• Cache robots.txt files

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 124

Duplicate URL elimination

• For a non-continuous (one-shot) crawl, test to see if an extracted+filtered URL has already been passed to the frontier

• For a continuous crawl – see details of frontier implementation

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 125

Distributing the crawler

• Run multiple crawl threads, under different processes – potentially at different nodes• Geographically distributed nodes

• Partition hosts being crawled into nodes• Hash used for partition

• How do these nodes communicate?

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 126

Communication between nodes• The output of the URL filter at each node is sent

to the Duplicate URL Eliminator at all nodes

WWW

Fetch

DNS

ParseContentseen?

URLfilter

DupURLelim

DocFP’s

URLset

URL Frontier

robotsfilters

Hostsplitter

Tootherhosts

Fromotherhosts

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 127

URL frontier: two main considerations

• Politeness: do not hit a web server too frequently

• Freshness: crawl some pages more often than others• E.g., pages (such as News sites) whose content

changes oftenThese goals may conflict each other.(E.g., simple priority queue fails – many

links out of a page go to its own site, creating a burst of accesses to that site.)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 128

Politeness – challenges

• Even if we restrict only one thread to fetch from a host, can hit it repeatedly

• Common heuristic: insert time gap between successive requests to a host that is >> time for most recent fetch from that host

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 129

URL frontier: Mercator scheme

Prioritizer

Biased front queue selectorBack queue router

Back queue selector

K front queues

B back queuesSingle host on each

URLs

Crawl thread requesting URL

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 130

Mercator URL frontier

• URL’s flow in from the top into the frontier• Front queues manage prioritization• Back queues enforce politeness• Each queue is FIFO

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 131

Front queues

Prioritizer

1 K

Biased front queue selectorBack queue router

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 132

Front queues

• Prioritizer assigns to URL an integer priority between 1 and K• Appends URL to corresponding queue

• Heuristics for assigning priority• Refresh rate sampled from previous crawls• Application-specific (e.g., “crawl news sites

more often”)

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 133

Biased front queue selector

• When a back queue requests a URL (in a sequence to be described): picks a front queue from which to pull a URL

• This choice can be round robin biased to queues of higher priority, or some more sophisticated variant• Can be randomized

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 134

Back queues

Biased front queue selectorBack queue router

Back queue selector

1 B

Heap

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 135

Back queue invariants

• Each back queue is kept non-empty while the crawl is in progress

• Each back queue only contains URLs from a single host• Maintain a table from hosts to back queues

B

1

3…

Back queueHost name

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 136

Back queue heap

• One entry for each back queue• The entry is the earliest time te at which

the host corresponding to the back queue can be hit again

• This earliest time is determined from• Last access to that host• Any time buffer heuristic we choose

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 137

Back queue processing• A crawler thread seeking a URL to crawl:• Extracts the root of the heap• Fetches URL at head of corresponding back queue

q (look up from table)• Checks if queue q is now empty – if so, pulls a

URL v from front queues• If there’s already a back queue for v’s host, append v to q

and pull another URL from front queues, repeat• Else add v to q

• When q is non-empty, create heap entry for it

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 138

Number of back queues B

• Keep all threads busy while respecting politeness

• Mercator recommendation: three times as many back queues as crawler threads

139

Connectivity servers

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 140

Connectivity Server[CS1: Bhar98b, CS2 & 3: Rand01]

• Support for fast queries on the web graph• Which URLs point to a given URL?• Which URLs does a given URL point to?

Stores mappings in memory from• URL to outlinks, URL to inlinks

• Applications• Crawl control• Web graph analysis

• Connectivity, crawl optimization• Link analysis

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 141

Most recent published work

• Boldi and Vigna• http://www2004.org/proceedings/docs/1p595.pdf

• Webgraph – set of algorithms and a java implementation

• Fundamental goal – maintain node adjacency lists in memory• For this, compressing the adjacency lists is the

critical component

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 142

Adjacency lists

• The set of neighbors of a node• Assume each URL represented by an

integer• E.g., for a 4 billion page web, need 32 bits

per node• Naively, this demands 64 bits to represent

each hyperlink

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 143

Adjaceny list compression

• Properties exploited in compression:• Similarity (between lists)• Locality (many links from a page go to

“nearby” pages)• Use gap encodings in sorted lists• Distribution of gap values

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 144

Storage

• Boldi/Vigna get down to an average of ~3 bits/link• (URL to URL edge)• For a 118M node web graph

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 145

Main ideas of Boldi/Vigna• Consider lexicographically ordered list of all URLs,

e.g., • www.stanford.edu/alchemy• www.stanford.edu/biology• www.stanford.edu/biology/plant• www.stanford.edu/biology/plant/copyright• www.stanford.edu/biology/plant/people• www.stanford.edu/chemistry

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 146

Boldi/Vigna• Each of these URLs has an adjacency list• Main thesis: because of templates, the adjacency

list of a node is similar to one of the 7 preceding URLs in the lexicographic ordering

• Express adjacency list in terms of one of these• E.g., consider these adjacency lists

• 1, 2, 4, 8, 16, 32, 64• 1, 4, 9, 16, 25, 36, 49, 64• 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144• 1, 4, 8, 16, 25, 36, 49, 64

Encode as (-2), remove 9, add 8

Τµ. Μηχανικών Η/Υ, Τηλεπικοινωνιών & ∆ικτύων, Πανεπιστήµιο Θεσσαλίας 147

Resources• IIR Chapter 20• http://research.microsoft.com/~najork/mercator.pdf• www.robotstxt.org/wc/norobots.html• http://www.www2004.org/proceedings/docs/1p595.pdf