Embed Size (px)
Citation preview
Lars Lofgren and Will Kurt
Keep Your Gains from A/B Tests Without Killing Them Later May 2014
@larslofgren
Hit me up
1 The limits of A/B tests
We’ll cover…
2 The standard solutions
3 Simulations! Woohoo!
#KISSwebinar
4 The 3 strategies of A/B testing that work
5 How we A/B test at KISSmetrics
WATCH WEBINAR RECORDING NOW
Limits of A/B Tests
A/B tests don’t give you perfect decisions.
No ma!er what you do, you’re never 100% certain
If we’re not careful, winners aren’t really winners
Your conversions go up… and then they come back down
The Standard Solution
Run your test until you hit 95% statistical significance.
1 Pick the minimal improvement
Scientific A/B testing:
2 Determine your sample size
3 Determine degree of certainty (95%)
#KISSwebinar
4 Start test but don’t check it early
5 If results aren’t significant, keep control
This gives us the best data but not necessarily the
best ROI.
So how far do we take this?
Simulation Time!
We modeled several A/B testing strategies.
Using Monte Carlo simulations, we tested different strategies over 1
million observations (people).
Will Kurt gets full credit for all this.
@willkurt
1 Pick the minimal improvement
The Scientist:
2 Determine your sample size
3 Determine degree of certainty (95%)
#KISSwebinar
4 Start test but don’t check it early
5 If results aren’t significant, keep control
Results for the Scientist:
1 Waits until 80% significance
The Reckless Marketer:
#KISSwebinar
2 Calls a winner as soon as 80% gets hit
Results for the Reckless Marketer:
1 Waits for 95% significance
The Impatient Marketer:
#KISSwebinar
2 Moves on to the next test a#er 500 people
Results for the Impatient Marketer:
The Realist
#KISSwebinar
1 Waits for 99% significance
2 Moves on to the next test a#er 2,000 people
Results for the Realist:
The Persistent Realist
#KISSwebinar
1 Waits for 99% significance
2 Moves on to the next test a#er 20,000 people
Results for the Persistent Realist:
The Blitz Realist
#KISSwebinar
1 Waits for 99% significance
2 Moves on to the next test a#er 200 people
Results for the Blitz Realist:
Let’s compare them using the area under the curve.
A/B Strategy Scores
Strategy Conditions Score
Scientist Stats like a pro 67759
Reckless Marketer 80% 57649
Impatient Marketer 95% and 500 people 60532
Realist 99% and 2,000 people 67896
Persistent Realist 99% and 20,000 people 68346
Blitz Realist 99% and 200 people 62836
No Testing Testing? NOPE! 50000
Each score is the area under the curve from the simulation. The higher the score, the more
conversions you received.
0
17500
35000
52500
70000
Persistent Realist Realist Scientist Blitz Realist Impatient Reckless No Testing
50,000
57,64960,532
62,83667,75967,89668,346
A/B Strategy Scores
LOG IN WITH GOOGLE
Start Your Free KISSmetrics Trial
3 Strategies
Don’t make decisions at less than 95% significance.
You’ll waste all the time you spend testing
1 Be a scientist at 95%
We have 3 viable strategies for making this work:
2 Only make changes at 99%
3 Sloppy 95% but make it up in volume
#KISSwebinar
1 Pick the minimal improvement
Be a scientist when you have lots of data and resources
2 Determine your sample size
3 Determine degree of certainty (95%)
#KISSwebinar
4 Start test but don’t check it early
5 If results aren’t significant, keep control
If you don’t have the data or resources to be a
scientist, go fast at 99%.
And if you still want to play at 95% without being
a scientist, never stop testing.
How We A/B Test
First, get volume to 4000+ people/month.
Only make changes at 99% significance.
Let the test run at least 1 week before checking
results.
If not at 99% a#er two weeks, launch the next
test.
If the next test isn’t ready, let it keep running while you build the next one.
The KISSmetrics A/B Testing Strategy
1 Get to 4,000 people/month for test
2 Only change the control if you reach 99%
3 Check results a#er 1 week
4 Launch the next test at 2 weeks
5 Let old tests run if you’re still building
This strategy isn’t perfect. It’s a balance between good data and speed.
1 Be a scientist at 95%
Remember the 3 strategies:
2 Only make changes at 99%
3 Sloppy 95% but make it up in volume
#KISSwebinar
