Dispatches From The A/B Testing Trenches
Hiya guys! Patrick (patio11) here. You signed up for periodic emails from me about making and selling software.
For several years I ran a high-end consulting company. Most of my clients looked like Fog Creek -- B2B SaaS companies with $10 to $50 million a year in revenue. The biggest win for several of my consulting clients was convincing them to A/B test regularly.
Why aren't you A/B testing already? (You are? Really? Name the test that's running today. 95% of you -- and some of the savviest companies I've worked with -- don't do it regularly. A/B testing is like flossing, intending to do it won't keep your teeth in your mouth.)
Tip #1: A/B Testing Is Not Just For Design Elements
Some of the most dramatic wins I've ever gotten from A/B testing were on pricing/packaging, in-application behavior, and process improvements in checkout. All three of these are often considered "high-risk", high-cost optimizations. A/B testing helps to lower the (actual and perceived) risk among various stakeholders for making the change.
Three quick stories.
1) We doubled the revenue of a SaaS company by re-pricing their product, using the advice I already emailed you about pricing and selling to Enterprises. Yeah, doubled. Server Density was gracious enough to contribute a full case study, which I strongly encourage you to read.
2) A deceptively high-leverage place to test is the onboarding experience for your software. (You might remember my free video about that.) I've seen increases of 20% to 100% by doing fairly pedestrian work there. It is often very specific to the application, but a quick win in B2B SaaS is making sure you get all team members invited into the software as fast as possible. Honeybadger took my advice on this and reported that it 3x-ed their conversion rate from a free trial to paid, and the implementation was really simple and elegant.
3) I tested a full-app redesign of Bingo Card Creator, my first SaaS application. This was a hit with customers but failed to increase revenue (like about 75% of your A/B tests will), but I learned enough by failing to take another whack at the problem, and scored a 50% increase in sales with a two-character test. (And, by the way, if that doesn't prove that results are uncorrelated from effort I don't know what will.)
Tip #2: The Biggest Problem With A/B Testing Isn't Technology, It Is Organization
Stop me if you've heard this one before: "We know we should be A/B testing and plan on doing it sometime, but Tim in Marketing can't do it himself and Engineering is busy getting the next release ready, so..."
I've heard variants of that from a half-dozen clients, many of them exceptionally savvy. I have two technological solutions for you: implement your choice of Visual Website Optimizer or Optimizely (I have happy clients on both, personal preference is VWO). This will let Tim from Marketing do A/B tests without taking more than 10 minutes of engineering time, once. (You embed J/S snippets in two places.)
The bigger problem is getting buy-in from people that A/B testing is important and worth doing on a routine basis. My usual strategy for achieving this is pointing to an incoming invoice with a five figure amount on it and saying "Hey, since you're paying me this much, you should probably listen to my advice." Since you're saving five figures by hearing this in email instead of face to face, listen to this advice: commit to doing one A/B test every Monday with a recap email to the team every Monday, each of the next four weeks. (I'd like you to spend eight but I think you'll be more likely to actually do it if I say "four", so I'm saying "four.")
Your goal at this Phase 1 of your A/B testing strategy is not to make money, though you probably will. It is to collect the experience of winning for your team, because most people who start A/B testing will find one That Can't Possibly Be Worth That Much Money moment and after that becomes part of your team's verbal lore you're set for life.
Tip #3: That "Stereotypical Startup Landing Page" Exists For A Reason
You've probably noticed that software company product pages tend to look very similar these days, particularly early-stage companies coming out of YCombinator and what have you. You can practically play design-by-checklist:
- big, bold headline
- prominent graphical element, often a video or screenshot
- single call to action to either the free trial signup or pricing page
- secondary call to action to a product tour or the pricing page
- understated navigation away from the landing page
- most textual content below the fold
I have listened to designers sneer about this being uncreative (accurate!) and derivative (certainly!) and probably-purchased-from-ThemeForest-for-$15 (possible!). If your designer tells you this, punch them in the face have a discussion about how landing page design pays their salary. (Or show them 37signals A/B testing results, which are some of the prettiest explorations of high-performant landing pages I've seen.) The reason why many companies under independent control have tended to converge on many similar design elements is partially a matter of fashion but mostly parallel evolution towards something which dominates most competing design.
Personal anecdote time: my own pages have always resembled the Stereotypical Startup Landing Page, but I recently had a very successful client (who sells a product to technical folks, mostly at Big Freaking Enterprises) replace a all-the-best-design-from-1996 text-heavy homepage with something more in the style of the Stereotypical Startup Home Page. We A/B tested this, beginning with a very basic rough-draft which just demoted the existing textual copy, added in prominent call to action buttons, and put in a motivational headline. One day of designer time increased trials by more than a quarter and will increase yearly sales by more than launching a new product. Subsequently, we spent a few weeks refining the design and copy, and smashed that improvement.
Your takeaway: The design of your most important page (typically the first page landed on by folks coming from your prime acquisition channel -- often the home page or an ad landing page) has huge power to move the needle for your business, even if your software is mature and the business is quite established.
If you're not A/B testing yet, though, I recommend against testing full-page redesigns to start. They're a lot of work to accomplish and, frequently, won't justify that. Three much easier tests to run: 1) Benefits-heavy, punchy titles for landing pages or the home page. 2) Call to action buttons -- don't go crazy here, but you'll often get a nice lift replacing "Submit" (ugh!) or "Sign Up" with something which actually promises customer value. 3) Reduce the prominence of distractions aside from the one or two conversions which you want to push on the home page, or increase their prominence. (I'm assuming they're the most "clickable" things above the fold already... right? Check that.)
As Long As I've Got You Here...
I made a 5 hour video course that teaches SaaS companies how to use lifecycle emails (targeted emails sent to individual customers based on their specific actions) to increase user happiness, get more sign-ups, convert more trials, and sell more software.
If you're interested in the email that your application can send automatically to get you $2,000 of revenue, or how to write the sort of marketing campaign that caused a "step function" increase in WPEngine's sales, that will be very worth your time. You'll hear more from me about lifecycle emails next time.
I'd love to hear what you currently do for email marketing and what your biggest challenges are. Do you not do any emailing at all at present? What is stopping you? Hit Reply and get in touch. I read everything.
Until next time.
Regards,
Patrick McKenzie