A/B Testing Software Market Overview

What Is A/B Testing Software?

A/B Testing IconIn Internet marketing, A/B or split testing is the process of dividing web traffic among multiple versions of a webpage (or email, etc.) and evaluating which variation performs best at achieving a desired outcome, such as free trial signups or purchases. A/B testing can be performed on websites, emails, paid search, banner ads, mobile apps, and other marketing channels, including offline. The focus of this guide is A/B testing on websites.

The testing process helps optimize a website's ability to convert visitors into leads or customers (or some other goal) through iterative changes. Through split testing, companies can ensure that a website change is in fact an improvement before releasing the new version to all website traffic. It is an important part of conversion rate optimization and can help companies increase revenue without spending more money to acquire web traffic.

A/B testing is considered superior to sequential or before-and-after testing, where website changes are launched across the board and then results are compared, because it is less susceptible to fluctuations in traffic patterns over time.

With A/B testing, it is usually best to alter one element of a page at a time, such as a headline, call-to-action, or button placement, so that any resulting change in conversion rate can be accurately attributed. Tests should run long enough and be exposed to enough traffic to achieve statistical significance in the results. Standard industry practice is that a test should reach 95% confidence that the winner achieves the desired outcome (purchases, click-throughs, etc.) at a higher rate. This means there is a 95% chance that the results are statistically significant, or a 5% chance that they present a false positive. Usually, the winning version is released to all website traffic, and another test is commenced.

On websites, users might test elements like headlines, number of form fields, layout and design, pricing and promotional offers, amount of text, use of images, navigational or checkout flow, and the language, placement and design of the call to action.

How A/B Testing Software Works

A/B testing software automatically and randomly splits web traffic among the different variations of a webpage being tested, calculates conversion rates based on a defined goal, and measures statistical likelihood that one version will consistently perform better than another.

The most basic of A/B testing software and tools require the user to design and code each variation of the webpage being tested. More advanced tools offer a WYSIWYG (What You See Is What You Get) interface that allows non-technical users to create variations without having to write the underlying HTML or CSS code of a page.

The most sophisticated A/B testing software allows users to run targeted campaigns among specific visitor segments, based on data such as demographics, device type or referral source. It can also perform multi-page campaigns, which test one element, such as a banner, across multiple pages of a visitor's session. Some tools can approach one-to-one personalization, which attempts to offer a unique and relevant experience for each site visitor.

There are a few different ways of setting up testing technology. Most A/B testing tools use a snippet of code, usually JavaScript, on a website to divide traffic and track conversions. Once embedded, the code allows marketers to set up tests directly within the tool's interface without requiring additional IT or development time.

Client-side vs. Server-side

Diagram of client-side laptop and server-side server communicating with one anotherDifferent types of visitors behave differently, and A/B testing tools can help users identify when certain variations perform better with certain visitor types. For example, variation A might perform better overall, but variation B has a higher conversion rate for customers on a mobile device. Learning this fact allows companies to optimize content for specific visitor types. Users might segment website visitors based on a number of attributes, including geolocation, device type, new vs. returning visitor, demographics, referral source, etc.

A/B testing technology can be client-side or server-side. Client-side means that the tool is executed by the website visitor's browser. When the page is loaded, the JavaScript in the website's html code calls back to the tool's server, which tells the browser which page content to display. Many of the tools in this report are client-side—for example, Optimizely, Monetate, Maxymiser and Google Analytics Content Experiments.

With server-side technology, the page is compiled by the server and presented to the browser in its finished form. Adobe Target, Unbounce and SiteSpect are server-side technologies. SiteSpect uses a unique, alternate method, called the proxy server or intercept method. The tool acts as a proxy server between the visitor's browser and the website's server, and controls which website content is presented to the visitor.

Photo of Tim Ash - CEO of SiteTuners“With server-side tools, the content management systems are building in testing, maintaining multiple versions of content and serving them up based on segmentation. It offers control over deployment regardless of browser problems. Client-side tools are usually easier for the marketer. They have WYSIWYG editors and make it easy to make surface level changes without needing IT, which is a huge pain point. It's hard to use them to restructure the registration path or the checkout, though, because you'll touch the CMS.”

Synchronous vs. asynchronous

Diagram of Synchronous and asynchronous loadingWith client-side A/B testing tools, the JavaScript embedded in the website's code can either function synchronously or asynchronously. Synchronous means that the testing tool's script loads completely (i.e., calls back to the server and waits for an answer) before the actual webpage begins to load. This can cause the page to load more slowly.

Asynchronous means that the webpage loads content in parallel, while the testing script runs. This can cause a flicker effect if the original content loads for a fraction of a second before the variation appears. One way to reduce the flicker effect is to introduce a tiny delay in the loading of content to allow the script to get a head start.

Some A/B testing tools offer both options.

Photo of Chris Goward - Founder and CEO of WiderFunnel“There is still no best tool or tool approach. The JavaScript approach has a lot of benefits in getting testing easily implemented. They've solved the biggest problem, which is the technical barriers to testing. The downside is load time and site speed, and the lack of flexibility of what you can manipulate. There are cases where it will have a flicker or it loads more than it needs to. Other approaches might allow for more flexibility and speed. I don't think we've hit the best solution yet as an industry.”

Differentiating Factors among A/B testing Tools

Most website A/B testing tools offer the basic functions: dividing web traffic among variations, calculating which version is the winner, and identifying statistical confidence in the results. There are several differentiating capabilities to consider, however.

Multivariate Testing IconMultivariate testing

Multivariate testing uses several different values for multiple elements on one page to create countless combinations or versions of that page, each of which are exposed to a random segment of live traffic. For example, if a user creates three headlines, three images and three calls-to-action, there would be 27 possible combinations for a multivariate test. An analyst can measure the impact each of those variables (in this example, headline, image, CTA) has on the webpage's conversion rate. Because they present many different versions of a page, multivariate tests require more time and traffic to achieve statistical significance. However, they allow marketers to test many elements of a webpage at once.

Split URL Testing IconSplit URL testing

With split URL testing, rather than sending traffic to slightly different variations of the same webpage using the same URL, the testing software is dividing traffic between two different URLs, such as www.example.com/offerA and www.example.com/offerB. The URLs are developed by the user and hosted by the website's server, rather than the testing tool. Split URL testing requires the user to create and code each webpage. However, it allows users to more easily test versions that are dramatically different from each other.

Multi-page Testing Data IconMulti-page testing

Multi-page testing allows users to test an element that spans multiple pages of the website, while providing a consistent user experience for the site visitor. For example, if a company wants to test a design element of a multi-step checkout process, visitors who got the original variation on step 1 will continue to see the same variation through the rest of the checkout flow.

Segmentation and Targeting IconSegmentation and targeting

Different types of visitors behave differently, and A/B testing tools can help users identify when certain variations perform better with certain visitor types. For example, variation A might perform better overall, but variation B has a higher conversion rate for customers on a mobile device. Learning this fact allows companies to optimize content for specific visitor types. Users might segment website visitors based on a number of attributes, including geolocation, device type, new vs. returning visitor, demographics, referral source, etc.

Users can define one specific visitor segment and run a test only on that type of visitor (excluding all others). Alternatively, users can run a test on several different visitor types and analyze results for each segment separately. The most advanced of tools allow for ad-hoc, after-the-fact segmentation, in which a user does not have to pre-define visitor segments prior to running the test, but can surface interesting segments in the post-test analysis.

Once a user understands which content performs best for each visitor type, he or she can begin targeting. Targeting is showing optimized content for different visitors, based on what was learned in segmentation.

Personalization IconPersonalization

Personalization is similar to targeting, but instead of multiple visitor segments it optimizes content for website visitors on a near one-to-one basis. This can include product recommendations based on past purchases, geolocation, demographics, and other factors. The concept behind personalization is to show the most effective, relevant content to each individual based on all data available, in order to provide the best customer experience and to entice the individual to take the desired action, such as make a purchase.

Photo of Ryan Garner - Co-founder of Clearhead“Increasingly you are seeing vendors re-position testing as one piece of the puzzle, with the broader capability being personalization. The idea is that instead of there being a one size fits all version, there's a right banner for the right person. If I could just get the right experience to the right person at the right time, they are more likely to convert. It's a great idea in theory, and it makes for good marketing. We will see how it plays out.”

Advanced Testing Capabilities IconAdvanced testing capabilities

Nearly all testing tools allow users to A/B test visual components of a webpage such as language, layout and design. Not all enable more structural or complex changes, however. More advanced tools offer the ability to test things like new product features, search algorithms, price, checkout flow, and testing on secure https webpages.

Mobile-specific A/B Testing IconMobile-specific A/B testing

Most testing tools allow users to segment visitors based on device (e.g., desktop vs. mobile). Many enable mobile-specific testing on a user's website, and some enable native app testing. A few, for example, offer software development kits (SDKs) that allow native app testing for iOS or Android apps. There are also point solutions focused on mobile app testing such as Artisan, Apptimize and Leanplum, which are not covered in this guide.

A/B testing a native app is inherently differently than testing a website, because the app is downloaded and run locally on a mobile device rather than accessed via a live Internet connection. Mobile app testing initially required app owners to push an updated version of the app to users (and obtain approval from the app store) in order to run a new test. Several software vendors have developed various methods of working around this requirement.

Predictive IconPredictive

Predictive capabilities allow testing and targeting software to predict visitor behavior, based on previous actions and the behavior of other, similar website visitors, and tailor content accordingly. Predictive targeting requires some self- or machine-learning capabilities on the part of the tool, where a computer model ingests data from various sources and makes a best guess regarding the most effective content to present to each visitor.

Photo of Bryan Eisenberg -  Keynote Speaker, Advisor, Bestselling Author“Predictive tools help you understand what will happen by starting to create models, for example, of all your past customers who purchased from you. The next step is, how can you make it happen? This is where machine-learning tools that automate tasks come in—to make actions happen on the data without a heavy human dimension. The goal is revenue optimization; you're looking for anything that helps you produce revenue based on the data.”

Adaptive Algorithms IconAdaptive algorithms

In some A/B testing software tools, the user defines the percent of web traffic that should be allocated to each variation being tested. For example, a user might decide to send 75% of traffic to the original version, which has a known conversion rate, and 25% to the new treatment. Once the test has reached statistical significance and seen enough web traffic, the winning version is released to 100% of web traffic.

However, some tools can use an adaptive algorithm to adjust the division of web traffic as test results come in. This allows a company to take advantage of the winning variation by sending more traffic to it, while still exploring the possibility that the lower performing variation might still win. The use of adaptive algorithms helps discourage users from ending tests too early and getting false positives, while still allowing them to take advantage of what could be the winning variation. This method is also called the “multi-armed bandit” algorithm.

Photo of Ryan Garner - Co-founder of Clearhead“Many testing tools are really geared toward ‘winner takes all’ testing. With a bandit or adaptive approach, you can continue to explore the performance of other variations. Say, for example, you have three different versions of a banner and the metric you care about is clicks on that banner. After three weeks, version A got the most clicks. In theory, you'd run version A forever, but the adaptive approach questions the winner. Are the other two worse always? Maybe there is a seasonal variation or a different banner performs better over the holidays. You dial up the winner, but you're constantly testing the other two throughout the year, with just a few impressions, to surface those insights.”

Tool Integration IconIntegration with other tools

A/B testing software often integrates with other marketing technologies. Many integrate with web analytics tools, so that users can perform more robust segmentation and reporting, as well as view test results directly within the analytics tool. Some also integrate with other data sources, such as customer relationship management tools and third-party data providers like Oracle's BlueKai, to provide more variables for the purposes of segmentation and targeting.