Patrick McKenzie Opticon 2014: Advanced A/B Testing

Conversion Optimization: The World Beyond

Headlines and Button Color

Patrick McKenzieFounder, Kalzumeus Software

Low hanging fruit tastes great,but the rest is worth the work, too.

Test Idea #1Quotas / Tier Levels / etc

@user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")

What Did We Learn From That Test?

• ~60% increase in sales going from 15 -> 8 cards• Synergistic effects can occasionally actually happen.• You don’t know what you don’t know until you test

for it.

Key Points With Testing Quotas

• Grandfather in existing users.• Set quota levels intelligently.• Be ready to tolerate small uptick in support load.• Your product is still too cheap.

Test Idea #2Payment Methods / Checkout Flow

Threw This Together In An Hour (Oops)

Rethought Design & UX

What We Learned From This Test

• A single A/B test can help your bottom line more than a product launch.• Dealing with synergistic effects in testing can

occasionally be maddening.• The data is never wrong, but sometimes the data

isn’t telling you what you think it is.

Key Points With Testing Checkout Flow

• Make absolutely sure you don’t break this!• Have UX-focused team (product / marketing / etc)

go over every step of checkout funnel with zero tolerance for bad UX.• Checkout flows go “stale” at many successful

companies. Schedule a deep dive on them twice a year, at least.• Upsells and cross selling are classics for a reason!

Test Idea #3Offline Business Processes

Testing degree of sales which maximizes conversion of leads

What We Learned From This Test

• Important to communicate with sales/CS.• Lead to resource-intensive but hugely successful

lead-scoring efforts.• Justified very expensive interventions like e.g.

onboarding help for everybody.

Key Points With Testing Offline Processes

• Your software/workflow has to close the loop back to the testing software. This can be tricky.• Virtually no CRM or A/B testing software accurately

models enterprise decisionmaking process. You might be forced to optimize for proxy conversions.• Volumes required for statistical significance are

really punishing, so sometimes we “proxy and pray.”

Test Idea #4Product On-boarding / FRE

What Engineers Will Ship If You Let Them

Key Goals of Onboarding Process

• Get user to a concrete success as soon as possible.• For more complex sales, lay groundwork for future

success by e.g. inviting in team or getting permission to follow-up.• For B2C, especially in social, you live and die on

virality during the onboarding experience.

In-app Tour Introducing Core Use

Cheat Code for B2B SaaS Sales

What We Learned From This Test

• Users who start onboarding tour 25% to 100%+ more likely to purchase software• Users sometimes have visceral reaction to elements

of the onboarding experience.• Engineers prefer in-app onboarding but it is

expensive as heck to build and does not always justify itself versus e.g. a Youtube video.

Key Points With Testing Offline Processes

• Your software/workflow has to close the loop back to the testing software. This can be tricky.• Virtually no CRM or A/B testing software accurately

models enterprise decisionmaking process. You might be forced to optimize for proxy conversions.• Volumes required for statistical significance are

really punishing, so sometimes we “proxy and pray.”

Test Idea #5Core Product Interactions

Too Complicated

80% of Users Use 20% of Features

Try Radically Simplifying

Improved Presentation and UX Of Core Workflow

What We Learned From This Test

• Just hiding the advanced features increased task success by 10%, which directly improved sales• Further jump of 15% when we did the redesign– Absolutely no increase in sales…– … but we saw in the funnel that this was solely due to

less people hitting the trial limits …– … which we were able to profitably exploit with the

quota limit test described earlier

Key Points With Testing Core Interactions

• Test the entire funnel. Sometimes task success doesn’t (unfortunately) help bottom line metrics.• A good understanding of where people spend their

time/energy is vital – test the 90% interaction, not the 2%.

ROI on ~$500 A Month Testing Budget

Febru

ary, 2

010

March, 2

010

April, 2010

May, 2

010

June, 2010

July, 2010

August,

2010

Septem

ber, 2010

October,

2010

November,

2010

December,

2010

January

, 2011

Febru

ary, 2

011

March, 2

011

April, 2011

May, 2

011

June, 2011

July, 2011

August,

2011

Septem

ber, 2011

October,

2011

November,

2011

December,

2011

January

, 2012

Febru

ary, 2

012

March, 2

012

April, 2012

May, 2

012

June, 2012

July, 2012

August,

2012

Septem

ber, 2012

October,

2012

November,

2012

December,

2012

January

, 2013

Febru

ary, 2

013

March, 2

013

April, 2013

May, 2

013

June, 2013

July, 2013

August,

2013

Septem

ber, 2013

October,

2013

November,

2013

December,

2013

January

, 2014

Febru

ary, 2

0140.00%

0.50%

1.00%

1.50%

2.00%

2.50%

3.00%

3.50%

Let’s Talk Testing Challenges

A/B Testing Across System Boundaries

Not A Hard Problem

Our Marketing Site

A/B

Gets A Wee Bit Trickier With 2 Sites

Our Marketing Site

A/B

The Web Application

Total Freaking Madness

Our Marketing Site

A/B

The Web Application

iPhone app

Offline processesAndroid app

Partner site

Content minisite

Offsite checkoutJapanese subsidiary

Sanity Restored

Consumer ApplicationsIdentity & A/B Manager

Lessons Learned With This

• Think about strategy for transitioning people between systems – easier to build into design earlier than later• Cookies with unique identities are a stopgap, not a

solution. You need to understand the sales process.• This is a good opportunity to standardize on an A/B

testing tool and workflow… or you’ll hate your life• This is a “high class problem” which can be deferred

until $50M+ in revenue, but when you solve it, it is going to cost serious time/money.

Increasing A/B Testing Adoption In Engineering and Marketing Teams

Single Most Important Lesson: One Line of Code

@user.card_limit = ab_test("user_card_limit_May_2012", [8, 15], :conversion => "purchase")

Other Lessons Learned

• Organization should learn rather than guess better.– Best investment ever: organization-wide wiki.– Next best: weekly “What we tried this week” email.

• Engineers love A/B testing when you describe it like a puzzle or a video game, with bragging rights.• Marketers love A/B testing when it’s positioned as a

way to increase their status in the organization rather than a replacement for their talents.

Things To Never Be Without

• CS should be able to force-override a user’s participation in an A/B test (and, relatedly, “see what users is seeing” is great if you can get it).• A/B tests deeper than surface level need unit/etc

tests. Make sure your framework supports it.• Test switcher UI in development/staging is a huge

win.• URL query parameter to flip tests a huge win.

Thanks for Listening!

• I love talking about this and you cannot possibly waste my time: @patio11 or patrick@kalzumeus.com• Free email mini-course on CRO in software

companies: http://bit.ly/patrick-optimizely• Liked this presentation? I deliver something like

this twice a month, free, to my email list. http://training.kalzumeus.com to sign up