Embed Size (px)
DESCRIPTION
A/B Testing Beyond Headlines and Button Colors -- ideas for tests (particularly for B2B SaaS), common pitfalls in organizations, and how to overcome them.
Citation preview
Conversion Optimization: The World Beyond
Headlines and Button Color
Patrick McKenzieFounder, Kalzumeus Software
Low hanging fruit tastes great,but the rest is worth the work, too.
Test Idea #1Quotas / Tier Levels / etc
@user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")
What Did We Learn From That Test?
• ~60% increase in sales going from 15 -> 8 cards• Synergistic effects can occasionally actually happen.• You don’t know what you don’t know until you test
for it.
Key Points With Testing Quotas
• Grandfather in existing users.• Set quota levels intelligently.• Be ready to tolerate small uptick in support load.• Your product is still too cheap.
Test Idea #2Payment Methods / Checkout Flow
Threw This Together In An Hour (Oops)
Rethought Design & UX
What We Learned From This Test
• A single A/B test can help your bottom line more than a product launch.• Dealing with synergistic effects in testing can
occasionally be maddening.• The data is never wrong, but sometimes the data
isn’t telling you what you think it is.
Key Points With Testing Checkout Flow
• Make absolutely sure you don’t break this!• Have UX-focused team (product / marketing / etc)
go over every step of checkout funnel with zero tolerance for bad UX.• Checkout flows go “stale” at many successful
companies. Schedule a deep dive on them twice a year, at least.• Upsells and cross selling are classics for a reason!
Test Idea #3Offline Business Processes
Testing degree of sales which maximizes conversion of leads
What We Learned From This Test
• Important to communicate with sales/CS.• Lead to resource-intensive but hugely successful
lead-scoring efforts.• Justified very expensive interventions like e.g.
onboarding help for everybody.
Key Points With Testing Offline Processes
• Your software/workflow has to close the loop back to the testing software. This can be tricky.• Virtually no CRM or A/B testing software accurately
models enterprise decisionmaking process. You might be forced to optimize for proxy conversions.• Volumes required for statistical significance are
really punishing, so sometimes we “proxy and pray.”
Test Idea #4Product On-boarding / FRE
What Engineers Will Ship If You Let Them
Key Goals of Onboarding Process
• Get user to a concrete success as soon as possible.• For more complex sales, lay groundwork for future
success by e.g. inviting in team or getting permission to follow-up.• For B2C, especially in social, you live and die on
virality during the onboarding experience.
In-app Tour Introducing Core Use
Cheat Code for B2B SaaS Sales
What We Learned From This Test
• Users who start onboarding tour 25% to 100%+ more likely to purchase software• Users sometimes have visceral reaction to elements
of the onboarding experience.• Engineers prefer in-app onboarding but it is
expensive as heck to build and does not always justify itself versus e.g. a Youtube video.
Key Points With Testing Offline Processes
• Your software/workflow has to close the loop back to the testing software. This can be tricky.• Virtually no CRM or A/B testing software accurately
models enterprise decisionmaking process. You might be forced to optimize for proxy conversions.• Volumes required for statistical significance are
really punishing, so sometimes we “proxy and pray.”
Test Idea #5Core Product Interactions
Too Complicated
80% of Users Use 20% of Features
Try Radically Simplifying
Improved Presentation and UX Of Core Workflow
What We Learned From This Test
• Just hiding the advanced features increased task success by 10%, which directly improved sales• Further jump of 15% when we did the redesign– Absolutely no increase in sales…– … but we saw in the funnel that this was solely due to
less people hitting the trial limits …– … which we were able to profitably exploit with the
quota limit test described earlier
Key Points With Testing Core Interactions
• Test the entire funnel. Sometimes task success doesn’t (unfortunately) help bottom line metrics.• A good understanding of where people spend their
time/energy is vital – test the 90% interaction, not the 2%.
ROI on ~$500 A Month Testing Budget
Febru
ary, 2
010
March, 2
010
April, 2010
May, 2
010
June, 2010
July, 2010
August,
2010
Septem
ber, 2010
October,
2010
November,
2010
December,
2010
January
, 2011
Febru
ary, 2
011
March, 2
011
April, 2011
May, 2
011
June, 2011
July, 2011
August,
2011
Septem
ber, 2011
October,
2011
November,
2011
December,
2011
January
, 2012
Febru
ary, 2
012
March, 2
012
April, 2012
May, 2
012
June, 2012
July, 2012
August,
2012
Septem
ber, 2012
October,
2012
November,
2012
December,
2012
January
, 2013
Febru
ary, 2
013
March, 2
013
April, 2013
May, 2
013
June, 2013
July, 2013
August,
2013
Septem
ber, 2013
October,
2013
November,
2013
December,
2013
January
, 2014
Febru
ary, 2
0140.00%
0.50%
1.00%
1.50%
2.00%
2.50%
3.00%
3.50%
Let’s Talk Testing Challenges
A/B Testing Across System Boundaries
Not A Hard Problem
Our Marketing Site
A/B
Gets A Wee Bit Trickier With 2 Sites
Our Marketing Site
A/B
The Web Application
Total Freaking Madness
Our Marketing Site
A/B
The Web Application
iPhone app
Offline processesAndroid app
Partner site
Content minisite
Offsite checkoutJapanese subsidiary
Sanity Restored
Consumer ApplicationsIdentity & A/B Manager
Lessons Learned With This
• Think about strategy for transitioning people between systems – easier to build into design earlier than later• Cookies with unique identities are a stopgap, not a
solution. You need to understand the sales process.• This is a good opportunity to standardize on an A/B
testing tool and workflow… or you’ll hate your life• This is a “high class problem” which can be deferred
until $50M+ in revenue, but when you solve it, it is going to cost serious time/money.
Increasing A/B Testing Adoption In Engineering and Marketing Teams
Single Most Important Lesson: One Line of Code
@user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")
Other Lessons Learned
• Organization should learn rather than guess better.– Best investment ever: organization-wide wiki.– Next best: weekly “What we tried this week” email.
• Engineers love A/B testing when you describe it like a puzzle or a video game, with bragging rights.• Marketers love A/B testing when it’s positioned as a
way to increase their status in the organization rather than a replacement for their talents.
Things To Never Be Without
• CS should be able to force-override a user’s participation in an A/B test (and, relatedly, “see what users is seeing” is great if you can get it).• A/B tests deeper than surface level need unit/etc
tests. Make sure your framework supports it.• Test switcher UI in development/staging is a huge
win.• URL query parameter to flip tests a huge win.
Thanks for Listening!
• I love talking about this and you cannot possibly waste my time: @patio11 or [email protected]• Free email mini-course on CRO in software
companies: http://bit.ly/patrick-optimizely• Liked this presentation? I deliver something like
this twice a month, free, to my email list. http://training.kalzumeus.com to sign up
