11

Shades of Grey: On the effectiveness of reputation-based “blacklists”

Reporter: 林佳宜Email: M98570015@mail.ntou.edu.tw2010/8/16

ReferencesS. Sinha, M. Bailey, and F. Jahanian.

Shades Of Grey: On the effectiveness of reputation based blacklists. In International Conference on Malicious and Unwanted Software (Malware 2008), October 2008.

2

3

OutlineIntroductionBlacklistsApproachEvaluationConclusion

IntroductionMalicious code, or malware, executed on

compromised hostsThe host-based anti-virus software

falling woefully behind–with detection rates as low as 40%

Defenders have turned to coarse-grained, reputation-based techniques

real time blacklists

Used to block unsolicited email, or spam

Investigate a number of possible causes for this low accuracy

4

Several contributionsAn investigation of email, spam, and

spam tool behavior in the context of a large academic network.

An analysis of the accuracy of four prevalent spam blacklists

A preliminary study of the causes of inaccuracy and a discussion of the issues as they relate to reputation-based services

5

Blacklists and ToolCurrently a large number of organizations

provide services for spam detection NJABL SORBS SpamHaus SpamCop

Using a spam detector for spam detection called SpamAssassin

These techniques have gained prominence, little is known about their effectiveness or potential

draw backs

6

SpamAssassin evaluationFalse positive rate for

SpamAssassin is small close to 0.5% for a threshold of 5.0

7

Experience Data Identify the spam received by a

large academic network consisting

7,000 unique hosts total of 1,074,508 emails millions of email messages, over a period 10

days in June of 2008

8

Number of mails per hour observedOn an average

8,000 SMTP connections per hour

9

Email characteristicsTotal of 1,074,508 emails were

successfully delivered

10

Blacklists effectivenessEvaluate the false positive and false

negative rates of four blacklists NJABL, SORBS, SpamCop, SpamHaus

False negative NJABL had a false negative rate of 98% SpamCop had a false negative rate of 35%

False positive The NJABL has the least false positives The SORBS has the most false positives

11

False positive rateThe NJABL has the least false

positives followed by SpamHausSORBS has an overall false positive

rate of 9.5%

12

Exploring blacklist false positivesFalse positive rates for SORBS were

significantly higher. Two possible reasons:

SpamAssassin is itself wrong and the blacklists are correctly pointing out the spam.

it is likely that prominent mail servers shared by legitimate and illegitimate people are getting blacklisted

SpamAssassin we found that SpamAssassin has around 5% of false negatives

13

False negative rateNJABL had a very few false

positives, it has a huge false negative

SpamCop has the smallest false negative rate

The blacklists seem to have significantly higher false

negative

14

Exploring blacklist false negativesIt is difficult to come up with reasons

We do not know have access to the spamtrap deployment

We do not know the precise algorithm used for blacklisting

Two possible causes: lack of visibility and the possibility of low volume or low rate

spammers.

15

Low volume/short lived spammersReason that a blacklist may miss

spam Visibility, low volume or short lived 80% of these sources were observed just for

a second

16

Evaluate the coverage of different blacklistsNJABL has been omitted because

its low detection rate

17

18

ConclusionPresented a preliminary evaluation of

four popular blacklists on an academic network

Found that the blacklists have significant false negative rates and a higher than expected false positive rate

Not be able to detect low volume spammers

Questions

19