Four Key Factors for A/B Testing Success

Selecting the right A/B testing software is only one small part of achieving success. Other factors, such as determining what to test, avoiding false positives, and garnering organization-wide support for testing, are critical. Below are some excerpts from interviews with testing, analytics and conversion experts on launching a successful program.

1. What you test matters. Qualitative feedback can help.

A/B testing might help you choose the best-performing color for your call-to-action button, but it's not limited to that, and button color might not be the test that will get you the biggest win. A significant number of A/B tests lead to no results at all—meaning, version A and version B perform essentially the same. Testing isn't done in a vacuum, and there are many resources available to help you create tests that will actually produce results. Often the best place to start is to understand what's going on in the minds of your customers and website visitors through qualitative feedback tools.

Photo of Dr. Karl Blanks - Chairmain, Conversion Rate Experts“You don't need to be a mind reader—just ASK visitors why they didn't buy. We spend a lot of time gathering qualitative feedback, and getting inside the minds of visitors. We use traditional surveys, on-page surveys, an open-ended feedback button, heatmapping and more. Some tools are much more fruitful than others at generating ideas that lead to actual improvements in conversion. We select tools based on their IPM (insights-per-minute).”
Photo of Chris Goward - Founder and CEO of WiderFunnel“Collecting insights is a messy process. There are a number of tools for gathering qualitative feedback—post-purchase surveys, user testing, clickstream analysis, heatmaps, voice of customer—all for the purpose of understanding barriers in conversion funnel. You grab components from a variety of areas to enhance the testing: qualitative data from prospects, your past database of test results, competitive analysis, etc. Then, you use the quantitative testing to validate those concepts, as well as generate new ideas and trying random things sometimes to see what sticks.

Marketers often misuse qualitative data by being swayed by the numbers. They won't be statistically valid—only fodder for insights and inspiration. One interesting user comment is enough to inspire an idea, but not enough to implement without testing.”

Testing need not be limited to visual or copy changes on the website.

Photo of Tim Ash - CEO of SiteTuners“A lot of the time it's not the onsite experience that moves the needle. It's the backend—where you put tripwires to upsell or convert from free to paid, or the communication sequence in marketing automation—that has the 5x improvement that you never get through website changes. Testing companies have you believe that it's the percent of tests that finish and the testing velocity that matter. That's missing the point. If you only focus on testing or analytics, it's like fighting with one hand tied behind your back. You have to look at the technology set, the culture, the user experience and the backend touch points. You need the holistic view.”

Sometimes qualitative feedback can be more valuable than testing.

Photo of Stephane Hamel - Director of Innovation, Cardinal Path“People ask me if you could only use one tool or one type of data, what would you use? And to me, even though I'm heavily involved in quantitative data (i.e. typical web analytics), it always goes back to customer satisfaction, which is a qualitative appreciation of user experience collected through other means such as surveys and polls. Listening to your customers is certainly the best opportunity for identifying improvement opportunities.

There are a lot of low hanging fruits for which you don't need to conduct a formal test. An obvious issue—sometimes surfaced through other sources of data, such as your analytics, user feedback, or even experience and best practices—is sufficient to justify a change. The digital analytics data you collect can get you very far in your optimization process. I think keeping at least one test running is a good opportunity to learn and give you the confidence and the experience so that when you're going to need a real and rigorous test, you'll know how to do it efficiently.”

2. Testing needs dedicated resources.

As with any role, testing is less successful when employees with other full-time responsibilities are in charge of it. It becomes a one-off task rather than a continuous process of improvement. Many experts say that when the web analyst is in charge of A/B testing, it doesn't get the attention it deserves.

Photo of Stephane Hamel - Director of Innovation, Cardinal Path“Imagine a pyramid. At the top, there are business skills: defining the business strategy and objectives. A foundational pillar is the enabling capabilities: the technology and people who know about the site content architecture, its features, the supporting technology and how to use the testing tool and collect the right data. The other pillar represents analytical skills: the ability to understand data and turn it into insight. Each of the three sets of skills are critical to success, and I have never seen anyone strong in all three—thus the necessary teamwork and complementarity of the team. Testing can only bring value if it is aligned with business goals and priorities, if you know how to leverage the proper tools and concepts, and turn that into actionable insight.”
Diagram of Testing Resource Pyramid
Photo of Tim Ash - CEO of SiteTuners“Analysts are not the same as people doing conversion rate optimization. Analysts are typically in the back room producing reports and doing analysis, but most of it never sees the light of day. It's like driving down the freeway at 300 mph looking in the rearview mirror. The CRO crowd is action-oriented, they're innovators, they're trying different stuff. Getting them to coexist is difficult.”

In addition to having a team dedicated to testing and optimizing the website, it's important to have the entire company involved, including allowing test ideas to come from anyone at the organization, not just the testing team. This often requires a culture shift in the direction of testing and optimization.

Photo of Tim Ash - CEO of SiteTuners“If you think about the conversion maturity model, which involves dimensions like customer experience, tools and technology, company structure, cultural aspects, politics—where most companies hit the wall is the level of cultural change at the organization. The right people aren't talking to each other. It has to be driven from the top. There needs to be a champion at the C-level with a CRO team reporting to them.”

3. Understand what statistical significance really means.

Most A/B testing tools will track statistical significance of a test as it runs, telling you, for example, that the new treatment currently has an 85% chance of consistently out-performing the original. Standard best practice is that a test should run until it reaches a 95% confidence level. This means that there is a 5% chance that the winner is really a false positive.

However, there are other factors to consider beyond the confidence level calculated by the tool. For example, most experts say a test should run for at least a week, if not longer, and be exposed to a certain amount of traffic, even if statistical significance is achieved earlier.

Photo of Justin Rondeau - Chief Testing Evangelist and Editor, WhichTestWon“A 95% confidence level might not be accurate based on other factors. People are fickle. People act differently on different days. If you have enough traffic to get a test called on Monday, you should still let it run throughout the week. User behavior is never the same later on in the week.

Sometimes, if there's a new feature, there's a major spike in use, but you have to look at it over time to see if it sticks. Or, if you change something to where people have to relearn how to do a process, it might cause a drop initially, but you have to let things normalize. Confidence level doesn't take into account these other variables.”

4. Testing trumps best practices and fancy designs.

While tests run by other companies can provide inspiration for your own testing endeavors, their results shouldn't be taken at face value.

Photo of Chris Goward - Founder and CEO of WiderFunnel“People will start to be more skeptical of best practices. Everyone's copying the leaders of the industry, with rotating homepage carousels, or overemphasized security symbols, but you have to figure out what works for you…

Also, businesses will abandon doing website redesigns and will start iteratively testing all components of their website, including layout and information architecture. If you just flip a switch to a whole new look, that may hurt your business. You can run dramatic tests in less time than it takes to do a full website redesign by simply testing the look and feel of the side-wide templates.”
Photo of Tim Ash - CEO of SiteTuners“Many of the problems we encounter with conversions have to do with graphic designers dressing up the page—throwing in stock photography of people and people's faces, or using motion in an undisciplined way, which are distractions. Before you deploy the page and invest all that time, make sure the important thing is getting the attention. Often this means getting rid of the extra broken embellishments.”