Upload
webshopwednesday
View
104
Download
1
Embed Size (px)
DESCRIPTION
The basics you should take a look at before starting A/B testing.
Citation preview
Best Practices and Lessons Learned from
100,000 A/B TestsA/B TestsMartijn Geerlings
100,000
Thursday, November 28, 13
Is de (klant) reis...
belangrijker dan de bestemming?
Thursday, November 28, 13
2009! 2010! 2012!2011! 2013!
Thursday, November 28, 13
Lessons Learned
Thursday, November 28, 13
Lesson #1:
Define success .
Thursday, November 28, 13
Thursday, November 28, 13
Lesson #1:
Define success .
Thursday, November 28, 13
Lesson #1:
Define quantifiable success metrics.
Thursday, November 28, 13
Thursday, November 28, 13
Visitvisitors per day
Thursday, November 28, 13
Visitvisitors per day
Emailsign up
rate
Thursday, November 28, 13
Visitvisitors per day
Emailsign up
rate$
$ per recipient
Thursday, November 28, 13
Visit Email $sign up
rate $ per recipientvisitors per day
Thursday, November 28, 13
Media
Button
Thursday, November 28, 13
Button:
1.Sign Up
2.Learn More
3. Join Us Now
4.Sign Up Now
Media:
1.Get Involved Image
2.Family Image
3.Change Image
4.Barack’s Video
5.Springfield Video
6.Sam’s Video
Splash page experimentVariations:
Thursday, November 28, 13
Button: “Sign Up”Thursday, November 28, 13
Button: “Learn More”Thursday, November 28, 13
Button: “Join Us Now”Thursday, November 28, 13
Button: “Sign Up Now”Thursday, November 28, 13
Button:
1.Sign Up
2.Learn More
3. Join Us Now
4.Sign Up Now
Media:
1.Get Involved Image
2.Family Image
3.Change Image
4.Barack’s Video
5.Springfield Video
6.Sam’s Video
Variations:Splash page experiment
Thursday, November 28, 13
Media: “Get Involved”Thursday, November 28, 13
Media: “Family”Thursday, November 28, 13
Media: “Change”Thursday, November 28, 13
Media: “Barack’s Video”Thursday, November 28, 13
Media: “Springfield Video”Thursday, November 28, 13
Media: “Sam’s Video”Thursday, November 28, 13
Button:
1.Sign Up
2.Learn More
3. Join Us Now
4.Sign Up Now
Media:
1.Get Involved Image
2.Family Image
3.Change Image
4.Barack’s Video
5.Springfield Video
6.Sam’s Video
Variations:Splash page experiment
Thursday, November 28, 13
Button:
1.Sign Up
2.Learn More
3. Join Us Now
4.Sign Up Now
Media:
1.Get Involved Image
2.Family Image
3.Change Image
4.Barack’s Video
5.Springfield Video
6.Sam’s Video
Variations:Splash page experiment
Thursday, November 28, 13
Button:
1.Sign Up
2.Learn More
3. Join Us Now
4.Sign Up Now
Media:
1.Get Involved Image
2.Family Image
3.Change Image
4.Barack’s Video
5.Springfield Video
6.Sam’s Video
Variations:Splash page experiment
Thursday, November 28, 13
Splash page experiment
Thursday, November 28, 13
Splash page experiment
Thursday, November 28, 13
Splash page experiment
Thursday, November 28, 13
Email Subscriptions Volunteers Amount Raised
Original: 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 +288,000 +$57,000,000
New: 10,000,000 1,000,000 $200,000,000
Splash page experiment
Thursday, November 28, 13
Email Subscriptions Volunteers Amount Raised
Original: 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 +288,000 +$57,000,000
New: 10,000,000 1,000,000 $200,000,000
Splash page experiment
Thursday, November 28, 13
Email Subscriptions Volunteers Amount Raised
Original: 7,120,000 712,000 $143,000,000
+40.6% +2,880,000 +288,000 +$57,000,000
New: 10,000,000 1,000,000 $200,000,000
Splash page experiment
Thursday, November 28, 13
Lesson #1:
Define quantifiable success metrics.
Thursday, November 28, 13
Lesson #2:
Less is more. Reduce choices.
Thursday, November 28, 13
vs.
Thursday, November 28, 13
vs.
+8% engagement
Thursday, November 28, 13
+43.8% Purchases
Thursday, November 28, 13
vs.
Thursday, November 28, 13
vs.
+14.8% conversion rate
Thursday, November 28, 13
Lesson #2:
Less is more. Reduce choices.
Thursday, November 28, 13
Lesson #3:
Words matter. Focus on your call to action.
Thursday, November 28, 13
Donation button experiment
Variations
Thursday, November 28, 13
Donation button experiment
VariationsNever
Signed UpSigned Up,
Never DonatedPreviously Donated
0.0% 0.0% 0.0%+2.3% +27.8% +16.3%-27.8% N/A N/A+15.2% -24.6% +11.9%+8.5% +2.9% +18.4%
Thursday, November 28, 13
vs.
Thursday, November 28, 13
vs.
Thursday, November 28, 13
vs.
+14.6% clicks
Thursday, November 28, 13
Lesson #3:
Words matter. Focus on your call to action.
Thursday, November 28, 13
Lesson #4:
Fail fast.
Thursday, November 28, 13
vs.
Thursday, November 28, 13
vs.
-10% conversions
Thursday, November 28, 13
vs.
Thursday, November 28, 13
vs.
-10% conversions
Thursday, November 28, 13
Lesson #4:
Fail fast.
Thursday, November 28, 13
Lesson #5:
Seek the Global Maximum.
Thursday, November 28, 13
Thursday, November 28, 13
Thursday, November 28, 13
Thursday, November 28, 13
Thursday, November 28, 13
Thursday, November 28, 13
+0.5% Retention +1.4% Engagement
Thursday, November 28, 13
Lesson #5:
Seek the Global Maximum.
Thursday, November 28, 13
1. Define quantifiable success metrics.2. Less is more. Reduce choices.3. Words matter. Focus on your call to action.4. Fail fast.5. Seek the Global Maximum.
Lessons Learned
Thursday, November 28, 13
Thursday, November 28, 13
One more thing...
Thursday, November 28, 13
6000+ happy testers
Start Today!
Thursday, November 28, 13