Prediction Market Science & Technology at Yahoo!

ResearchResearch

Prediction Market Science &Technology at Yahoo!David M. PennockMike Dooley, Tej Kasturi, Bernard Mangold, Havi Hoffman

Yiling Chen, Chao-Hsien Chu, Sandip Debnath, Rael Dornfest,Joan Fiegenbaum, Gary Flake, Lance Fortnow, Brian Galebach,Lee Giles, Joe Kilian, Steve Lawrence, Tracy Mullen, Rahul Sami,Emile Servan-Schreiber, Michael Wellman, Justin Wolfers

ResearchResearch

Prediction markets• Futures market designed to elicit a forecast

about some future event• Leverages “wisdom of crowds”, extracting and

combining information from distributed sources• Have been used successfully to

• Predict election outcomes [IEM, 1988-]• Predict corporate metrics (sales, product release

times, …) [HP, 2001-] [MSFT, Eli Lily, Intel,Siemens] [GOOG, 2005-]

• Predict movie box office returns [HSX], news[NewsFutures], scientific conjectures [FX], sportingevents, judicial nominations, economic numbers, …[inTrade], real estate [HedgeStreet], many others

ResearchResearch

Prediction markets research @ Y!2002-2005

• Computational aspects & mechanism design• n events, 2n combinations, 22n poss. bets!• Algorithms & computational complexity

• Leverage independence (“compact markets”)• “Betting boolean-style”: Generic bidding language

• New exchange mechanism: dynamic pari-mutuel market;Cross btw stock market and horse race betting;Ideal for huge numbers of futures and low liquiditycommon in derivatives trading and gambling

• Empirical analyses of real-$/play-$ markets;Does money matter?

• Academic: 6 pubs; 4 patents; 2 workshops• Practical: Search futures & Tech Buzz Game

2

1

3

4

ResearchResearch

Search Futures & Tech Buzz Game

ResearchResearch

Search futures• Search data: What people worldwide are thinking about today• Search futures: What people will be thinking about tomorrow• Billions of advertising/business dollars ride on answers• Research questions

• Is search buzz predictable? If so, what factors promote accuracy?(trading mechanism, currency [real/fantasy], subsidies, dividendrates, noise bots, #traders, competence of traders, ... )

• What types of search concepts? What types of trends?(sudden, cyclical, growth, burst)

• How to handle huge numbers of (combinatorial/conditional)markets and low liquidity

• Which traders do well? (buzz traders, buy/hold traders, daytraders, noise traders, affinity traders, cheaters)

• Can machine learning post-processing boost accuracy?• How to combat search spam, manipulation• How to hedge marketing/business risks with real-$ search/PPC

futures

ResearchResearch

• Yahoo!,O’Reilly launched Buzz Game 3/05 @ETech• Research testbed for investigating search futures• Buy “stock” in hundreds of technologies

• Earn dividends based on actual search “buzz”

• API interface• Exchange mechanism is new Yahoo!R invention

Cross btw stock market and horse race betting

http://buzz.research.yahoo.com

ResearchResearchExchange InterfaceDynamic Parimutuel “Market Maker”

ResearchResearch

Technology forecasts• iPod phone • What’s next?

Google Calendar?

• Another Apple unveiling10/12; iPod Video?

searchbuzz

price

9/8-9/18: searchesfor iPod phone soar;early buyers profit

8/29: Appleinvites pressto “secret”unveiling

8/28: buzz gamersbegin biddingup iPod phone

9/7: Appleannounces

Rokr

9am 10/5

ResearchResearch

Forecastaccuracy• Average forecast error

across 352 stocks• Market closing deadline

focuses traders• Dividend levels matter• Intelligent strategies work

• Randomized bots lostmoney to real traders

• Contest winner followedoptimal buzz tradingstrategy (prices ∝ √buzz);Went from 4th to 1st place infinal days

• Forecast error doesdecrease over time

end of phase 1contest period

forecast errorrapidly declinesas traders zero in

on correctpredictions

Early lessonslearned

ResearchResearch







2

1

3

4

ResearchResearch

Does Money Matter?

ResearchResearch

Real markets vs. market gamesHSX IEM

averagelog

score

arbitrageclosure

ResearchResearch

1 2 5 10 20 50 100

estimate

1

2

5

10

20

50

100

actual

Real markets vs. market gamesHSX FX, F1P6

probabilisticforecasts

expectedvalue

forecasts

489 movies

forecast source avg log scoreF1P6 linear scoring -1.84F1P6 F1-style scoring -1.82betting odds -1.86F1P6 flat scoring -2.03F1P6 winner scoring -2.32

ResearchResearch

Does money matter?Play vs real, head to headExperiment• 2003 NFL Season• ProbabilitySports.com

Online football forecastingcompetition

• Contestants assessprobabilities for each game

• Quadratic scoring rule• ~2,000 “experts”, plus:• NewsFutures (play $)• Tradesports (real $)

• Used “last trade” prices

Results:• Play money and real

money performedsimilarly• 6th and 8th respectively

• Markets beat most ofthe ~2,000 contestants• Average of experts

came 39th (caveat)

Electronic Markets, Emile Servan-Schreiber, Justin Wolfers, DavidPennock and Brian Galebach

ResearchResearch

0

25

50

75

100

Tra

deS

po

rts

Pri

ces

0 20 40 60 80 100

NewsFutures Prices

Fitted Value: Linear regression

45 degree line

n=416 over 208 NFL games.Correlation between TradeSports and NewsFutures prices = 0.97

Prices: TradeSports and NewsFutures

Prediction Performance of Markets

Relative to Individual Experts

020406080100120140160180200220240260280300

1 2 3 4 5 6 7 8 9 10 11 12 13 14

Week into the NFL season

Ra

nk NewsFutures

Tradesports

0

10

20

30

40

50

60

70

80

90

100

Ob

serv

ed F

req

uen

cy o

f V

icto

ry

0 10 20 30 40 50 60 70 80 90 100

Trading Price Prior to Game

TradeSports: Correlation=0.96

NewsFutures: Correlation=0.94

Data are grouped so that prices are rounded to the nearest ten percentage points; n=416 teams in 208 games

Market Forecast Winning Probability and Actual Winning Probability

Prediction Accuracy

ResearchResearch

Does money matter?Play vs real, head to head

Probability-

Football Avg

TradeSports

(real-money)

NewsFutures

(play-money)

Difference

TS - NF

Mean Absolute Error

= lose_price

[lower is better]

0.443

(0.012)

0.439

(0.011)

0.436

(0.012)

0.003

(0.016)

Root Mean Squared Error

= ?Average( lose_price2 )

[lower is better]

0.476

(0.025)

0.468

(0.023)

0.467

(0.024)

0.001

(0.033)

Average Quadratic Score

= 100 - 400*( lose_price2 )

[higher is better]

9.323

(4.75)

12.410

(4.37)

12.427

(4.57)

-0.017

(6.32)

Average Logarithmic Score

= Log(win_price)

[higher (less negative) is better]

-0.649

(0.027)

-0.631

(0.024)

-0.631

(0.025)

0.000

(0.035)

Statistically:TS ~ NFNF >> AvgTS > Avg

ResearchResearch







2

1

3

4

ResearchResearch

Dynamic Parimutuel Market

ResearchResearch

What is a pari-mutuel market?

• E.g. horse racetrack style wagering• Two outcomes: A B• Wagers:

AA BB

ResearchResearch



AA BB

√

ResearchResearch



AA BB

√

ResearchResearch

What is a pari-mutuel market?• Before outcome is revealed, “odds” are

reported, or the amount you would win perdollar if the betting ended now• Horse A: $1.2 for $1; Horse B: $25 for $1; … etc.

• Strong incentive to wait• payoff determined by final odds; every $ is same• Should wait for best info on outcome, odds• ⇒ No continuous information aggregation• ⇒ No notion of “buy low, sell high” ; no cash-out

Dynamic pari-mutuelmarket

Standard PM: Every $1 bet is the same DPM: Value of each $1 bet varies

depending on the status of wagering at thetime of the bet

Encode dynamic value with a price– price is $ to buy 1 share of payoff– price of A is lower when less is bet on A– as shares are bought, price rises; price is for an

infinitesimal share; cost is integral

ResearchResearch

1

1

1

11

1

11

1

11

1

Pari-mutuel marketBasic idea

ResearchResearch

0.9

0.40.2

32.5

2

1.6

1.31.11

1

Dynamic pari-mutuel marketBasic idea

ResearchResearch

How are prices set?

• A price function pi(n) gives theinstantaneous price of aninfinitesimal additional share beyondthe nth

• Cost of buying n shares: ∫0n

pi(n) dn• Different reasonable assumptions

lead to different price functions

ResearchResearch

Price functions

Closed formcost() & shares()

pi/pj = Si/SjAll money

Closed formshares() ;Numeric cost()

pi/pj = Mi/MjAll money


pi/pj = Mi/MjLosing money


p1= P2

p2= P1

Losing money

ResultConstraint/Assumption

Share type

ResearchResearch







2

1

3

4

ResearchResearch

Computational Aspects:Complex Betting

Market combinatorics

ProbabilityBush wins each state

90 to 100 (14)

80 to 90 (6)

70 to 80 (3)

60 to 70 (6)

50 to 60 (2)

40 to 50 (5)

30 to 40 (2)

20 to 30 (2)

10 to 20 (7)

0 to 10 (4)

Source: www.Tradesports.com; 3/26/2004.

[Thanks: Wolfers, Fortnow]

ProbabilityBush wins each state

90 to 100 (14)

80 to 90 (6)

70 to 80 (3)

60 to 70 (6)

50 to 60 (2)

40 to 50 (5)

30 to 40 (2)

20 to 30 (2)

10 to 20 (7)

0 to 10 (4)

Source: www.Tradesports.com; 3/26/2004.

[Thanks: Wolfers, Fortnow]

Compound markets Doubly exponential Pr(CA ^ AZ) ?

Pr(Elec | FL) ?Pr((IL^NJ)∨(¬IL^¬NJ)) ?

Not derivable as alinear combinations ofbase securities

2250 possible functions “Only” 250 securities

needed to span space

Market combinatorics Combinatorial markets Singly exponential Buy/sell multiple

securitiessimultaneously

Buy 2 TX & sell 4 CT,pay $10

Exactly analogous tocombinatorial auctions

250 possible “bundles”of securities

Compound markets I:Brute force

In principle, markets in all possible combinations will getyou everything you want

In practice, this is infeasible It’s also unnatural

$1 if A1&A2&…&AnI am entitled to:








Compound markets II:Leverage independence

Structure market according to unanimouslyagreed-upon independencies

E2

E5

E3

E6

E4

E1

$1 if E6|E3E5

$1 if E6|Ê3Ê5

$1 if E6|E3Ê5

$1 if E6|Ê3E5

[Pennock & Wellman 2000]

CARA & Markov indep ⇒ risk-neutral indep If all agents have CARA, then market structured as

TRIANGULATE[∪ni=1 MORALIZE(Di)] is op complete

Can still yield exponential savings (“compact sec. markets”) This example: 19 vs. 63

Compound markets II:Leverage independence

$1 if E6|E3E5

$1 if E6|E3Ê5 $1 if E6|Ê3Ê5

$1 if E6|Ê3E5

E2

E5

E3

E6

E4

E1

[Pennock & Wellman 2000]

Compound markets III:High-level bidding language A bidding language: write your own security

For example

Offer to buy/sell q units of it at price p Let everyone else do the same Auctioneer must decide who trades with whom at

what price… How? (next) More concise/expressive; more natural

$1 if Boolean_fn | Boolean_fnI am entitled to:

$1 if A1 | A2I am entitled to:

$1 if (A1&A7)||A13 | (A2||A5)&A9I am entitled to:

$1 if A1&A7I am entitled to:

The matching problem There are many possible matching rules for the

auctioneer A natural one: maximize trade subject to

no-risk constraint Example:

– buy 1 of for $0.40– sell 1 of for $0.10– sell 1 of for $0.20

No matter what happens,auctioneer cannot losemoney

$1 if A1

$1 if A1&A2$1 if A1&A2

trader gets $$ in state:A1A2 A1A2 A1A2 A1A2

0.60 0.60 -0.40 -0.40-0.90 0.10 0.10 0.10 0.20 -0.80 0.20 0.20

-0.10 -0.10 -0.10 -0.10

The matching problem Another way to look at it:

Logical reduction|

| Example:

– buy 1 of for $0.40– sell 1 of for $0.10– sell 1 of for $0.20

||

Clear match btw buy and sell|

$1 if A1

$1 if A1&A2$1 if A1&A2

$1 if A1= sell for $0.3

The matching problem Divisible orders: will accept any q* ≤ q Indivisible: will accept all or nothing Let Ω=all possible combinations; |Ω|=2n

Let αi be fraction of order i filled Let Υi

ω be payoff for order i in state ω Div. MP: Does ∃αi∈[0,1], ∀ω∈Ω, -∑αiΥi

ω≥0 Indiv. MP: Does ∃αi∈{0,1}, ∀ω∈Ω, -∑αiΥi

ω≥0 Optimizations

– max trade; max percent orders filled– max auctioneer utility subject to no-risk– max auctioneer utility -- with risk (“market maker”)

(at least 1 αi > 0)

Divisible vs. indivisible Sell 1 of A1 at $0.50 Buy 1 of (A1&A2) | (A1 || A2) at $0.50 Buy 1 of A1|A2 at $0.40


-0.50 -0.50 0.50 0.50 0.50 -0.50 -0.50 0 0 0.60 0 -0.40

0 -0.40 0 0.10



-0.50 -0.50 0.50 0.50 0.50 -0.50 -0.50 0 0 0.60 0 -0.40

0 -1 0 0.50



-0.50 -0.50 0.50 0.50 0.50 -0.50 -0.50 0 0 0.60 0 -0.40

-0.50 0.10 0.50 0.10



-0.50 -0.50 0.50 0.50 0.50 -0.50 -0.50 0 0 0.60 0 -0.40

0.50 0.10 -0.50 -0.40



-0.50 -0.50 0.50 0.50 0.50 -0.50 -0.50 0 0 0.60 0 -0.40

0 0 0 -0.10

3/5 x3/5 x 1 x

divisiblematch!

Complexity results Divisible orders: will accept any q* ≤ q Indivisible: will accept all or nothing

Natural algorithms– divisible: linear programming– indivisible: integer programming;

logical reduction?

# events divisible indivisibleO(log n) polynomial NP-completeO(n) co-NP-complete Σ2

p complete

reduction from SAT

reduction from X3C

reduction from T∃∀BF

Fortnow; Kilian; Sami

LP

Open questions Other matching rules

– maximize utility subject to no-risk– maximize utility (market maker)

What to do with the surplus– can be in cash and “leftover” securities– auctioneer keeps surplus– surplus is shared back among traders, auctioneer; how?

Trader optimization problem– how to choose securities, p’s, q’s, subject to limits/penalties for

number, complexity of bids– ultimately a game-theoretic question

Approximate algorithms, heuristics Incentive properties

ResearchResearch







2

1

3

4

Documents

Prediction Market Science & Technology at Yahoo!