IR System Evaluation

Farhad Oroumchian

IR System Evaluation

• System-centered strategy – Given documents, queries, and relevance judgments– Try several variations on the retrieval system– Measure which gets more good docs near the top

• User-centered strategy– Given several users, and at least 2 retrieval systems– Have each user try the same task on both systems– Measure which system works the “best”

Which is the Best Rank Order?

R

R

R

R

R

R

R

R

R

R

RRRRRR

RR

R

R

R

RRR

R

R

R

R

R

RRRRR

R

RRRR

R

R

a b c d e f g h

Measures of Effectiveness

• Good measures of effectiveness should– Capture some aspect of what the user wants– Have predictive value for other situations

• Different queries, different document collection– Be easily replicated by other researchers– Be expressed as a single number

• Allows two systems to be easily compared

• No measures of effectiveness are that good!

Some Assumptions

• Unchanging, known queries– The same queries are used by each system

• Binary relevance– Every document is either relevant or it is not

• Unchanging, known relevance– The relevance of each doc to each query is known

• But only used for evaluation, not retrieval!

• Focus on effectiveness, not efficiency

Exact Match MOE

• Precision– How much of what was found is relevant?

• Often of interest, particularly for interactive searching

• Recall– How much of what is relevant was found?

• Particularly important for law and patent searches

• Fallout– How much of what was irrelevant was rejected?

• Useful when different size collections are compared

The Contingency Table

Relevant Retrieved

Irrelevant Retrieved Irrelevant Rejected

Relevant RejectedRelevant

Not relevant

Retrieved Not RetrievedDocAction

Precision

Relevant RetrievedRetrieved

RecallRelevant Retrieved

Relevant

FalloutIrrelevant Rejected

Not Relevant

MOE for Ranked Retrieval

• Start with the first relevant doc on the list– Compute recall and precision at that point

• Move down the list, 1 relevant doc at a time– Computing recall and precision at each point

• Plot precision for every value of recall– Interpolate with a nonincreasing step function

• Repeat for several queries• Average the plots at every point

The Precision-Recall CurveRR

R

R

Relevant Retrieved

Irrelevant Retrieved Irrelevant Rejected

Relevant RejectedRelevant=4

Not relevant=6

Retrieved Not RetrievedDoc=10

Action

00.10.20.30.40.50.60.70.80.91

1 2 3 4 5 6 7 8 9 10

PrecisionRecall 0

0.2

0.4

0.6

0.8

1

0 0.5 1

Recall

Precision

0

0.2

0.4

0.6

0.8

1

0 0.5 1

Recall

InterpolatedPrecision

Single-Number MOE• Precision at a fixed number of documents

– Precision at 10 docs is the “AltaVista measure”• Precision at a given level of recall

– Adjusts for the total number of relevant docs• Average precision

– Average of precision at recall=0.0, 0.1, …, 1.0– Area under the precision/recall curve

• Breakeven point– Point where precision = recall

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1

Recall

Single-Number MOEPrecision at recall=0.1

Precision at 10 docs

Average Precision

Breakeven Point

Single-Number MOE Weaknesses• Precision at 10 documents

– Pays no attention to recall• Precision at constant recall

– A specific recall fraction is rarely the user’s goal• Breakeven point

– Nobody ever searches at the breakeven point• Average precision

– Users typically operate near an extreme of the curve• So the average is not very informative

Why Choose Average Precision?

• It is easy to trade between recall and precision– Adding related query terms improves recall

• But naive query expansion techniques kill precision– Limiting matches by part-of-speech helps precision

• But it almost always hurts recall

• Comparisons should give some weight to both– Average precision is a principled way to do this

• Rewards improvements in either factor

How Much is Enough?

• The maximum average precision is 1.0– But inter-rater reliability is 0.8 or less– So 0.8 is a practical upper bound at every point

• Precision 0.8 is sometimes seen at low recall

• Two goals– Achieve a meaningful amount of improvement

• This is a judgment call, and depends on the application– Achieve that improvement reliably across queries

• This can be verified using statistical tests

Statistical Significance Tests• How sure can you be that an observed difference doesn’t

simply result from the particular queries you chose?

System A0.200.210.220.190.170.200.21

System B0.400.410.420.390.370.400.41

Experiment 1Query

1234567

Average 0.20 0.40

System A0.020.390.160.580.040.090.12

System B0.760.070.370.210.020.910.46

Experiment 1Query

1234567

Average 0.20 0.40

The Sign Test

• Compare the average precision for each query– Note which system produces the bigger value

• Assume that either system is equally likely to produce the bigger value for any query

• Compute the probability of the outcome you got– Any statistics package contains the formula for this

• Probabilities<0.05 are “statistically significant”– But they still need to pass the “meaningful” test!

The Students T-Test• More powerful than the sign test

– If the assumptions are satisfied• Compute the average precision difference

– On a query by query basis for enough queries to approximate a normal distribution

• Assume that the queries are independent• Compute the probability of the outcome you got

– Again, any statistics package can be used• A probability>0.05 is “statistically significant”

Obtaining Relevance Judgments

• Exhaustive assessment can be too expensive– TREC has 50 queries for >1 million docs each year

• Random sampling won’t work either– If relevant docs are rare, none may be found!

• IR systems can help focus the sample– Each system finds some relevant documents– Different systems find different relevant documents– Together, enough systems will find most of them

Pooled Assessment Methodology• Each system submits top 1000 documents• Top 100 documents for each are judged

– All are placed in a single pool– Duplicates are eliminated– Placed in an arbitrary order to avoid bias

• Evaluated by the person that wrote the query• Assume unevaluated documents not relevant

– Overlap evaluation shows diminishing returns• Compute average precision over all 1000 docs

TREC Overview

• Documents are typically distributed in April• Topics are distributed June 1

– Queries are formed from topics using standard rules• Top 1000 selections are due August 15• Special interest track results due 2 weeks later

– Cross-language IR, Spoken Document Retrieval, …• Relevance judgments available in October• Results presented in late November each year

Concerns About Precision/Recall

• Statistical significance may be meaningless• Average precision won’t reveal curve shape

– Averaging over recall washes out information• How can you know the quality of the pool?• How to extrapolate to other collections?

Project Test Collection

مواد‌قانون‌از‌سيستم�‌قوانين‌داده‌پردازي•• 41‌queryاز‌سيستم‌قوانين‌

• Relevance judgments keyed to ItemID– Relevance is in scale 0-4

• بطور‌كلي‌نامربوط -=0• نامربوط =1• كمي‌مربوط =2• مربوط =3• كامال‌مربوط =4

– ‌نامربوط‌تلقي‌مي‌شوند‌0،1،2مربوط‌و‌4و‌3معموال‌ ‌

Project Overview• Install or write the software the software• Choose the parameters

– Stopwords, stemming, term weights, etc.• Index the document collection

– This may require some format-specific tweaking• Run the 20 queries• Compute average precision and other measures• Test query length effect for statistical significance

Team Project User Studies

• Measure value of some part of the interface– e.g., selection interface with and without titles

• Choose a dependent variable to measure– e.g., number of documents examined

• Run a pilot study with users from your team– Fine tune your experimental procedure

• Run the experiment with at least 3 subjects– From outside your team (may be in the class)