Best A/B Testing Tools45Qubit1https://dudodiprj2sv7.cloudfront.net/product-logos/7U/U0/7LQJX3DQ4V7D.JPEGOptimizely2https://dudodiprj2sv7.cloudfront.net/product-logos/nE/W8/NPJ6TVND0MJQ.PNGMonetate3https://dudodiprj2sv7.cloudfront.net/product-logos/EN/2t/7AALPRLAT012.JPEGEvergage4https://dudodiprj2sv7.cloudfront.net/product-logos/My/MB/W9MT6MQL1T3R.PNGAB Tasty5https://dudodiprj2sv7.cloudfront.net/product-logos/ol/OJ/TU1LGKGBBVMS.PNGMaxymiser6https://dudodiprj2sv7.cloudfront.net/product-logos/YP/B0/HML1KVEZIT8S.JPEGAdobe Target7https://dudodiprj2sv7.cloudfront.net/product-logos/V1/nr/BNKFU2DZMX77.PNGKameleoon8https://dudodiprj2sv7.cloudfront.net/product-logos/Ug/L9/M3370WPSHPIB.JPEGVisual Website Optimizer9https://dudodiprj2sv7.cloudfront.net/product-logos/vk/UI/2X31AXTEHCNO.JPEGDynamic Yield10https://dudodiprj2sv7.cloudfront.net/product-logos/xC/7E/Z52W1SBN5P4Z.PNGLeadpages11https://dudodiprj2sv7.cloudfront.net/product-logos/bg/QG/83001J4G8LRN.JPEGInstapage12https://dudodiprj2sv7.cloudfront.net/product-logos/t6/AK/S9A6T94SQQWU.JPEGExponea Experience Cloud13https://dudodiprj2sv7.cloudfront.net/product-logos/Dw/dh/MCCNTF6LWBOT.JPEGWebtrends Optimize14https://dudodiprj2sv7.cloudfront.net/vendor-logos/ic/Pm/J3JLC3T6ZKKS-180x180.PNGGoogle Content Experiments15https://dudodiprj2sv7.cloudfront.net/vendor-logos/a6/c6/JGG6SW6IUMTJ-180x180.PNGClickTale16https://dudodiprj2sv7.cloudfront.net/product-logos/hM/RI/3Q2CCMTR2ARX.JPEGSiteSpect17https://dudodiprj2sv7.cloudfront.net/product-logos/a1/2a/W65DT1ATM7WJ.JPEGConvert Experiences18https://dudodiprj2sv7.cloudfront.net/product-logos/Ln/Dn/PALREBEL2RS0.PNGBranch19https://dudodiprj2sv7.cloudfront.net/product-logos/Zr/Za/770MSUG65RZI.JPEGBound20https://dudodiprj2sv7.cloudfront.net/product-logos/yN/Hv/2MKHF7X6W3QG.JPEGZarget21https://dudodiprj2sv7.cloudfront.net/product-logos/QS/gE/EJPRR1ZCATTM.PNGCampaigner22https://dudodiprj2sv7.cloudfront.net/product-logos/JI/Bg/ZNJ4HAU319FO.PNGVero23https://dudodiprj2sv7.cloudfront.net/product-logos/7V/K6/TE0VM7E67HK3.JPEGAvenseo24https://dudodiprj2sv7.cloudfront.net/vendor-logos/o2/hh/F8TXA5RVR5KP.pngBounceX25https://dudodiprj2sv7.cloudfront.net/product-logos/X9/la/MBUWE7RFTT1R.PNG

Best A/B Testing Tools

TrustMaps are two-dimensional charts that compare products based on satisfaction ratings and research frequency by prospective buyers. Products must have 20 or more ratings to appear on this TrustMap, and those above the median line are considered Top Rated.

A/B Testing Tools Overview

What is A/B Testing?

A/B or split testing divides web traffic among multiple versions of a page and provides information on which version performs better at achieving a desired outcome such as signups or purchases.


The testing process aims to optimize a website’s ability to convert visitors into leads or customers. The goal is generally to increase revenue without spending more money to acquire traffic.


A/B Testing Features & Capabilities

Most A/B testing tools use a snippet of code, usually JavaScript. The code is placed on a website to control content and track traffic and conversions. This allows marketers to set up tests directly from the tool’s interface, without requiring additional IT or development time.


All A/B testing tools include the following features:


  • Splitting traffic among the different variations being tested

  • Calculating conversion rates

  • Measuring statistical likelihood that one variation will consistently outperform the other


Basic A/B testing tools may require custom design and code. Some advanced tools offer a WYSIWYG interface for creating variations. This is without needing to use HTML or CSS, which works well for non-technical users.


More sophisticated tools run targeted campaigns. These tools display different versions based on demographics, device type or referral source. They can also perform multi-page campaigns. This means testing one element (like a banner) on multiple pages during a session. These advanced tools help create a personalized user experience, with the goal of maximizing conversions.


A/B Test Best Practices

It is generally best to change one element of a page at a time so that any change in outcome can be attributed. With this approach, you’ll know which element results in better conversion.


Elements to test include:

  • Call-to-action

  • Headlines

  • Button placement


Some split testing tools allow multivariate testing. This allows you to change multiple elements on one page, creating countless combinations. An analyst can then measure the impact of each variable on conversion.


Split testing should run long enough to reach statistical significance - at least 95% confidence. The amount of time needed varies depending on the amount of traffic to the page. Because they involve many versions, multivariate tests require more time and traffic to reach statistical significance.


A/B Testing and Conversion Rate Optimization (CRO)

A/B testing is often part of conversion rate optimization. CRO refers to the process of continually improving a website’s effectiveness at reaching its goal. Usually, the goal is to turn visitors into customers or leads.


It is a usually a cyclical workflow, involving multiple steps:


  • Monitoring website visitor behavior

  • Gathering insight and visitor feedback

  • Testing a hypothesis for improving conversions

  • Evaluating results


Key technologies in the CRO workflow include:



Pricing Information

Getting pricing information for a specific A/B testing tool typically requires asking for a quote. You can find basic plans for as little as $50 per month. The more sophisticated tools can run up to $18,000 per year.

Excerpts from TrustRadius Buyer's Guide to A/B Testing

A/B Testing Software Market Overview

What Is A/B Testing Software?

A/B Testing IconIn Internet marketing, A/B or split testing is the process of dividing web traffic among multiple versions of a webpage (or email, etc.) and evaluating which variation performs best at achieving a desired outcome, such as free trial signups or purchases. A/B testing can be performed on websites, emails, paid search, banner ads, mobile apps, and other marketing channels, including offline. The focus of this guide is A/B testing on websites.

The testing process helps optimize a website's ability to convert visitors into leads or customers (or some other goal) through iterative changes. Through split testing, companies can ensure that a website change is in fact an improvement before releasing the new version to all website traffic. It is an important part of conversion rate optimization and can help companies increase revenue without spending more money to acquire web traffic.

A/B testing is considered superior to sequential or before-and-after testing, where website changes are launched across the board and then results are compared, because it is less susceptible to fluctuations in traffic patterns over time.

With A/B testing, it is usually best to alter one element of a page at a time, such as a headline, call-to-action, or button placement, so that any resulting change in conversion rate can be accurately attributed. Tests should run long enough and be exposed to enough traffic to achieve statistical significance in the results. Standard industry practice is that a test should reach 95% confidence that the winner achieves the desired outcome (purchases, click-throughs, etc.) at a higher rate. This means there is a 95% chance that the results are statistically significant, or a 5% chance that they present a false positive. Usually, the winning version is released to all website traffic, and another test is commenced.

On websites, users might test elements like headlines, number of form fields, layout and design, pricing and promotional offers, amount of text, use of images, navigational or checkout flow, and the language, placement and design of the call to action.

How A/B Testing Software Works

A/B testing software automatically and randomly splits web traffic among the different variations of a webpage being tested, calculates conversion rates based on a defined goal, and measures statistical likelihood that one version will consistently perform better than another.

The most basic of A/B testing software and tools require the user to design and code each variation of the webpage being tested. More advanced tools offer a WYSIWYG (What You See Is What You Get) interface that allows non-technical users to create variations without having to write the underlying HTML or CSS code of a page.

The most sophisticated A/B testing software allows users to run targeted campaigns among specific visitor segments, based on data such as demographics, device type or referral source. It can also perform multi-page campaigns, which test one element, such as a banner, across multiple pages of a visitor's session. Some tools can approach one-to-one personalization, which attempts to offer a unique and relevant experience for each site visitor.

There are a few different ways of setting up testing technology. Most A/B testing tools use a snippet of code, usually JavaScript, on a website to divide traffic and track conversions. Once embedded, the code allows marketers to set up tests directly within the tool's interface without requiring additional IT or development time.

Client-side vs. Server-side

Diagram of client-side laptop and server-side server communicating with one anotherDifferent types of visitors behave differently, and A/B testing tools can help users identify when certain variations perform better with certain visitor types. For example, variation A might perform better overall, but variation B has a higher conversion rate for customers on a mobile device. Learning this fact allows companies to optimize content for specific visitor types. Users might segment website visitors based on a number of attributes, including geolocation, device type, new vs. returning visitor, demographics, referral source, etc.

A/B testing technology can be client-side or server-side. Client-side means that the tool is executed by the website visitor's browser. When the page is loaded, the JavaScript in the website's html code calls back to the tool's server, which tells the browser which page content to display. Many of the tools in this report are client-side—for example, Optimizely, Monetate, Maxymiser and Google Analytics Content Experiments.

With server-side technology, the page is compiled by the server and presented to the browser in its finished form. Adobe Target, Unbounce and SiteSpect are server-side technologies. SiteSpect uses a unique, alternate method, called the proxy server or intercept method. The tool acts as a proxy server between the visitor's browser and the website's server, and controls which website content is presented to the visitor.

Photo of Tim Ash - CEO of SiteTuners“With server-side tools, the content management systems are building in testing, maintaining multiple versions of content and serving them up based on segmentation. It offers control over deployment regardless of browser problems. Client-side tools are usually easier for the marketer. They have WYSIWYG editors and make it easy to make surface level changes without needing IT, which is a huge pain point. It's hard to use them to restructure the registration path or the checkout, though, because you'll touch the CMS.”

Synchronous vs. asynchronous

Diagram of Synchronous and asynchronous loadingWith client-side A/B testing tools, the JavaScript embedded in the website's code can either function synchronously or asynchronously. Synchronous means that the testing tool's script loads completely (i.e., calls back to the server and waits for an answer) before the actual webpage begins to load. This can cause the page to load more slowly.

Asynchronous means that the webpage loads content in parallel, while the testing script runs. This can cause a flicker effect if the original content loads for a fraction of a second before the variation appears. One way to reduce the flicker effect is to introduce a tiny delay in the loading of content to allow the script to get a head start.

Some A/B testing tools offer both options.

Photo of Chris Goward - Founder and CEO of WiderFunnel“There is still no best tool or tool approach. The JavaScript approach has a lot of benefits in getting testing easily implemented. They've solved the biggest problem, which is the technical barriers to testing. The downside is load time and site speed, and the lack of flexibility of what you can manipulate. There are cases where it will have a flicker or it loads more than it needs to. Other approaches might allow for more flexibility and speed. I don't think we've hit the best solution yet as an industry.”

Differentiating Factors among A/B testing Tools

Most website A/B testing tools offer the basic functions: dividing web traffic among variations, calculating which version is the winner, and identifying statistical confidence in the results. There are several differentiating capabilities to consider, however.

Multivariate Testing IconMultivariate testing

Multivariate testing uses several different values for multiple elements on one page to create countless combinations or versions of that page, each of which are exposed to a random segment of live traffic. For example, if a user creates three headlines, three images and three calls-to-action, there would be 27 possible combinations for a multivariate test. An analyst can measure the impact each of those variables (in this example, headline, image, CTA) has on the webpage's conversion rate. Because they present many different versions of a page, multivariate tests require more time and traffic to achieve statistical significance. However, they allow marketers to test many elements of a webpage at once.

Split URL Testing IconSplit URL testing

With split URL testing, rather than sending traffic to slightly different variations of the same webpage using the same URL, the testing software is dividing traffic between two different URLs, such as www.example.com/offerA and www.example.com/offerB. The URLs are developed by the user and hosted by the website's server, rather than the testing tool. Split URL testing requires the user to create and code each webpage. However, it allows users to more easily test versions that are dramatically different from each other.

Multi-page Testing Data IconMulti-page testing

Multi-page testing allows users to test an element that spans multiple pages of the website, while providing a consistent user experience for the site visitor. For example, if a company wants to test a design element of a multi-step checkout process, visitors who got the original variation on step 1 will continue to see the same variation through the rest of the checkout flow.

Segmentation and Targeting IconSegmentation and targeting

Different types of visitors behave differently, and A/B testing tools can help users identify when certain variations perform better with certain visitor types. For example, variation A might perform better overall, but variation B has a higher conversion rate for customers on a mobile device. Learning this fact allows companies to optimize content for specific visitor types. Users might segment website visitors based on a number of attributes, including geolocation, device type, new vs. returning visitor, demographics, referral source, etc.

Users can define one specific visitor segment and run a test only on that type of visitor (excluding all others). Alternatively, users can run a test on several different visitor types and analyze results for each segment separately. The most advanced of tools allow for ad-hoc, after-the-fact segmentation, in which a user does not have to pre-define visitor segments prior to running the test, but can surface interesting segments in the post-test analysis.

Once a user understands which content performs best for each visitor type, he or she can begin targeting. Targeting is showing optimized content for different visitors, based on what was learned in segmentation.

Personalization IconPersonalization

Personalization is similar to targeting, but instead of multiple visitor segments it optimizes content for website visitors on a near one-to-one basis. This can include product recommendations based on past purchases, geolocation, demographics, and other factors. The concept behind personalization is to show the most effective, relevant content to each individual based on all data available, in order to provide the best customer experience and to entice the individual to take the desired action, such as make a purchase.

Photo of Ryan Garner - Co-founder of Clearhead“Increasingly you are seeing vendors re-position testing as one piece of the puzzle, with the broader capability being personalization. The idea is that instead of there being a one size fits all version, there's a right banner for the right person. If I could just get the right experience to the right person at the right time, they are more likely to convert. It's a great idea in theory, and it makes for good marketing. We will see how it plays out.”

Advanced Testing Capabilities IconAdvanced testing capabilities

Nearly all testing tools allow users to A/B test visual components of a webpage such as language, layout and design. Not all enable more structural or complex changes, however. More advanced tools offer the ability to test things like new product features, search algorithms, price, checkout flow, and testing on secure https webpages.

Mobile-specific A/B Testing IconMobile-specific A/B testing

Most testing tools allow users to segment visitors based on device (e.g., desktop vs. mobile). Many enable mobile-specific testing on a user's website, and some enable native app testing. A few, for example, offer software development kits (SDKs) that allow native app testing for iOS or Android apps. There are also point solutions focused on mobile app testing such as Artisan, Apptimize and Leanplum, which are not covered in this guide.

A/B testing a native app is inherently differently than testing a website, because the app is downloaded and run locally on a mobile device rather than accessed via a live Internet connection. Mobile app testing initially required app owners to push an updated version of the app to users (and obtain approval from the app store) in order to run a new test. Several software vendors have developed various methods of working around this requirement.

Predictive IconPredictive

Predictive capabilities allow testing and targeting software to predict visitor behavior, based on previous actions and the behavior of other, similar website visitors, and tailor content accordingly. Predictive targeting requires some self- or machine-learning capabilities on the part of the tool, where a computer model ingests data from various sources and makes a best guess regarding the most effective content to present to each visitor.

Photo of Bryan Eisenberg -  Keynote Speaker, Advisor, Bestselling Author“Predictive tools help you understand what will happen by starting to create models, for example, of all your past customers who purchased from you. The next step is, how can you make it happen? This is where machine-learning tools that automate tasks come in—to make actions happen on the data without a heavy human dimension. The goal is revenue optimization; you're looking for anything that helps you produce revenue based on the data.”

Adaptive Algorithms IconAdaptive algorithms

In some A/B testing software tools, the user defines the percent of web traffic that should be allocated to each variation being tested. For example, a user might decide to send 75% of traffic to the original version, which has a known conversion rate, and 25% to the new treatment. Once the test has reached statistical significance and seen enough web traffic, the winning version is released to 100% of web traffic.

However, some tools can use an adaptive algorithm to adjust the division of web traffic as test results come in. This allows a company to take advantage of the winning variation by sending more traffic to it, while still exploring the possibility that the lower performing variation might still win. The use of adaptive algorithms helps discourage users from ending tests too early and getting false positives, while still allowing them to take advantage of what could be the winning variation. This method is also called the “multi-armed bandit” algorithm.

Photo of Ryan Garner - Co-founder of Clearhead“Many testing tools are really geared toward ‘winner takes all’ testing. With a bandit or adaptive approach, you can continue to explore the performance of other variations. Say, for example, you have three different versions of a banner and the metric you care about is clicks on that banner. After three weeks, version A got the most clicks. In theory, you'd run version A forever, but the adaptive approach questions the winner. Are the other two worse always? Maybe there is a seasonal variation or a different banner performs better over the holidays. You dial up the winner, but you're constantly testing the other two throughout the year, with just a few impressions, to surface those insights.”

Tool Integration IconIntegration with other tools

A/B testing software often integrates with other marketing technologies. Many integrate with web analytics tools, so that users can perform more robust segmentation and reporting, as well as view test results directly within the analytics tool. Some also integrate with other data sources, such as customer relationship management tools and third-party data providers like Oracle's BlueKai, to provide more variables for the purposes of segmentation and targeting.

Page 2 is available in the complete TrustRadius Buyer's Guide to A/B Testing and covers

  • Four Key Factors for A/B Testing Success

The Best A/B Testing Software by Market Segment

Methodology

Different companies have varying online testing and personalization needs. We have thus compared A/B testing tools across three different segments to help software buyers identify the solutions that most closely align with their needs. There are a few different ways one could segment the market. One method might be volume of website traffic or revenue generated by the website. Another might be the type of website, such as e-commerce, lead generating, brochure or advertising-based. For this guide, we have segmented by the size of the company using the product1 based on number of employees, as that was the data readily available to us from reviews on TrustRadius. Our market segments are:

  • Small Businesses: 1-50 employees
  • Mid-Size Companies: 51-500 employees
  • Enterprises: >500 employees

Number of employees is a useful though imperfect way of approximating complexity of website optimization needs. There are certainly exceptions. Some smaller companies might run high traffic websites that generate a lot of revenue and run more sophisticated testing and targeting operations. Some larger companies may be far less reliant on their websites. See the User Ratings and Feedback section of this guide for more comprehensive coverage of each product, including the strengths and areas for improvement.

Our goal was to create an objective visual depiction of which A/B testing vendors offer the leading solutions in each market segment. We have created each TrustMap on two dimensions:

  1. Average User Ratings: The average “Likelihood to Recommend” rating, which is a representation of overall satisfaction, by customers within that company-size segment who have written reviews on TrustRadius. All ratings and reviews come from authenticated end-users of the software.
  2. Segment Adoption: The estimated relative number of websites using the product within that company size segment. It is intended to facilitate a rough comparison of market penetration by order of magnitude among the products, as customer data is unavailable.

Because vendors do not typically share customer counts, we used public data from Builtwith.com, which estimates the number of websites using a particular technology based on tags in the website's code. Note that one customer might use a tool on multiple websites. Additionally, one website might use multiple A/B testing tools.

For the enterprise segment, we used the number of top 10k websites ranked by traffic using the software, per Builtwith.com. For mid-size company market, we used the number of the top 100k websites using the software and subtracted those also in the top 10k, which are likely enterprises. For the small business segment, we used data from the entire Internet and subtracted those also in the top 100k, which are likely mid-size companies or enterprises. Segment adoption is plotted on the X or horizontal axis and uses a logarithmic scale in order to create a more even spread.

We draw lines for the median Average User Rating and Segment Adoption to create a graph with four quadrants. Products in the upper right area of the quadrant, which are at or above the median for both Average User Rating and Segment Adoption, are “Leaders” for that segment. Tools in the upper left quadrant, which have an Average User Rating at or above the median but below-median Segment Adoption, are “Strong Performers”.

We have omitted products for which there are fewer than two reviews in a particular segment or high standard deviation among a small number of reviews (i.e., one very negative review bringing down an average). Otherwise, we do not make our own judgments as to whether a product is an “enterprise” solution, for example. We include products solely based upon whether they are used by that type of company, per the existence of reviews on TrustRadius. The TrustMaps will evolve to become more statistically accurate as we gather more data for each product.


1 Many consultants have written reviews of testing products on TrustRadius. Often, the consultant's company is a small business, but the product is being used for an enterprise client. In that case, we re-classified those reviewers according to the client's probable company size, for the purposes of this report. When company size was unknown, we eliminated the data point.

Pages 5-18 are available in the complete TrustRadius Buyer's Guide to A/B Testing and cover

  • The Best A/B Testing Software for Enterprises
  • The Best A/B Testing Software for Mid-Size Companies
  • The Best A/B Testing Software for Small Businesses
  • AB Tasty Profile
  • Adobe Target Profile
  • Google Analytics Content Experiments Profile
  • Maxymiser Profile
  • Monetate Profile
  • Optimizely Profile
  • Qubit Profile
  • SiteSpect Profile
  • Unbounce Profile
  • Visual Website Optimizer Profile
  • Webtrends Optimize Profile

Trends in A/B Testing Software

Democratization of Access

As we discussed in our Buyer's Guide to Digital Analytics Software, the free version of Google Analytics helped democratize access to web analytics and enabled a massive culture shift in digital marketing toward data-driven decision making. Similarly, the introduction of easy-to-use, low-cost A/B testing tools has facilitated a rise in popularity of A/B testing.

According to a recent survey of TrustRadius members and website visitors interested in conversion rate optimization, 43% of respondents use more than one A/B testing tool, and 48% of respondents plan to spend more on A/B testing software in the coming year. Online testing is not as ubiquitous as web analysis, however. The survey revealed that all respondents are using at least one web analytics tool, but 9% of respondents aren't using an A/B testing tool at all.

TrustRadius Survey Results for Number of A/B Testing Tools Used for CRO

TrustRadius Survey Results for Next Years Spending on A/B Testing Software

Most of the experts we spoke to agree that split testing is becoming more widely adopted as a practice in digital marketing.

Photo of Justin Rondeau - Chief Testing Evangelist and Editor, WhichTestWon“Testing is becoming more mainstream. Years ago it was difficult to get a program implemented, but now the emerging technology has reduced resource constraints and made the barrier of entry to testing really low.”
Photo of Hiten Shah - Co-founder, KISSmetrics and Crazy Egg“Historically, we saw a lot of designers focused on usability testing using Crazy Egg. Now, marketers and product people are using the tool to inform their A/B testing. This is partly because of Optimizely and GWO [Google Website Optimizer] prior to that; everybody is thinking about A/B testing.”

This also means testing is becoming more widely adopted within organizations—being practiced across various business units rather than restricted to the marketing team. This is perhaps how companies begin using two or more A/B testing tools, each targeted at a different technical skill level.

Photo of Justin Bougher - Director of Product Management, SiteSpect“The optimization team is shifting from an isolated marketing group to a centralized function that works with all parts of the organization.”

Personalization: The Next Big Thing?

Some software vendors appear to be ahead of the rest of the market on personalization. Personalization is a form of targeting that attempts to serve up the most relevant content to a customer or visitor on a near 1:1 basis. Many of the vendors we spoke to said personalization was a core feature of their product. However, some consultants said their clients don't have the resources yet to practice personalization on their websites.

Photo of Ryan Garner - Co-founder, Clearhead“Our clients barely have enough time to create three banners, much less individualized experiences for each customer. A few of our larger clients are really targeting four or five segments. One-to-one personalization has a ton of potential, though in practice we're not seeing a lot of it yet.”

Some see it as an incredibly valuable capability in conversion rate optimization.

Photo of Jason Burby - President, Americas, POSSIBLE“Personalization will absolutely lead to greater engagement and conversion. Do you respond better to things that are relevant and personal to you? Of course. It's not a topic to be disputed.”

According to a 2014 survey by Ascend2, 77% of marketers do not use personalization on their website at all, and only 5% use it extensively. Of those that do use personalization, the majority are manually creating multiple versions of webpages for different customer segments rather than using marketing technology to automate the generation of personalized content. The concept is becoming more mainstream, however.

TrustRadius Survey Results for Company Usage of Personalization

Photo of Pete Koomen - Co-founder and President, Optimizely“A/B testing was not a muscle most companies had a few years ago. We've seen that change dramatically over the past few years. We see a similar trend with personalization. It is a powerful value proposition and most companies are in the early stages of building that muscle. We see it as a direction the industry is going.”
Photo of Justin Rondeau - Chief Testing Evangelist and Editor, WhichTestWon“We've seen that one of the most underused features of testing technology has been segmentation and personalization, but that's changed. It's been increasing.”
Photo of Ryan Garner - Co-founder, Clearhead“We are seeing that the personalization tools offered by many of the optimization platforms today don't scale. They provide interesting targeting capabilities, but then require a person to manually set up and manage all the rules and logic for what to offer their customers. We believe it's the algorithms that are ultimately going to win this space.”

Some in the industry think personalization has the potential to make visitors concerned for their privacy. Others disagree.

Photo of Bruce Ernst - VP of Product Management, Monetate“One of the flip sides of personalization is I may start seeing content oriented toward something I thought was private. We're seeing a lot of concern with that, and a trend toward not crossing that ‘creepy' line, so we're building in guardrails to our product.”
Photo of Jason Burby - President, Americas, POSSIBLE“With ultimate personalization, there's this idea that people can get scared. You hear about privacy, or you'll hear concerns about freaking people out, or that it's too complex. Personalization is a journey. You crawl, then walk. A little more relevant is better than not relevant at all. Then you keep moving down that spectrum.”

Will Testing Be Standalone or Embedded in Other Tools?

The software covered in this report is focused on A/B testing a website. However, split testing is a concept that can be used in any marketing channel, including mobile apps, email marketing, search engine marketing, display advertising, and even offline. A/B testing therefore could potentially become a common feature embedded in different types of software, such as marketing automation, content management systems and e-commerce platforms. Unbounce, for example, is actually a landing page builder that also allows users to A/B test their landing pages.

Photo of Paras Chopra - Founder and CEO, Wingify“Some email or marketing automation tools offer testing, but I don't think they do it properly. Just having a testing feature does not do it justice. The main value is that these tools let you create experiences in a quick manner. You might have to plug in a lot of coding, or what you can test is limited. A lot of these tools do not even include statistical significance in their reporting.”

Alternatively, some standalone A/B testing tools are enabling split testing of different channels, such as email and mobile apps.

Photo of Ryan Garner - Co-founder, Clearhead“Does it make sense to bake testing into an email product or a CMS or e-commerce platform, versus coming from a third-party tool? Some vendors believe yes. Just like we saw in the web analytics space, we believe there's value in having a specialized product with broad reach, which allows you to optimize across platforms.”

One key capability of standalone testing tools is integration with web analytics software. This allows the testing tool to use visitor segments already set up in the analytics platform, and the analytics tool to access and report on testing data. Some experts discussed whether analytics and testing software might one day converge. Content Experiments, for example, is an A/B testing feature embedded in the web analytics tool Google Analytics. Mixpanel, a web analytics tool focused on event-based tracking, recently added A/B testing of iOS apps to its analytics platform.

Photo of Justin Rondeau - Chief Testing Evangelist and Editor, WhichTestWon“For now, integrating standalone testing and analytics tools is sufficient. Having them converge isn't necessary. As we get more sophisticated, however, that might change.”
“I think the landscape is changing considerably in that they're getting closer and closer to being tied together. It's good for organizations because there are different counting methodologies with different testing solutions versus analytics. It creates frustration and data validation issues. As the testing solutions get more deeply integrated with analytics, organizations are going to be empowered to make use of historical data.”
Analytics and Testing Consultant

Mobile App Testing and Optimization

As the use and monetization of mobile apps evolve, optimizing native apps has become more important.

Photo of Richard White, CEO, UserVoice“If you go back three years on mobile, 80% of revenue is paid apps. You buy the app for $3 and that's it. It's not a strong environment to care about retention or loyalty or optimization. Now, 80% of revenue is in-app purchases. Getting customer feedback in the app is important.”

This means app owners are more interested in conversion rate optimization software for apps, such as A/B testing and customer feedback tools. Point solutions focused on split testing iOS and Android apps have emerged, such as Artisan, Apptimize and Leanplum. Additionally, many of the tools covered in this report offer software developer kits for A/B testing iOS or Android apps.

Photo of Ryan Garner - Co-founder, Clearhead“We tend to have a unique testing roadmap for each device with our clients—mobile, tablet and desktop. Mobile is a growing area where our clients are increasingly testing. We're also seeing a lot of vendors aggressively pursuing native app testing.”

It's not just about adapting web-focused CRO technology to mobile apps, however. Mobile changes how companies can approach their conversion rate optimization strategies.

Photo of Mark Treschl, President and CTO, OpinionLab“Mobile is huge, because it provides us with the opportunity to interface with customers that are in the moment when it comes to physical experiences. We know that folks are out in the world, using smartphones to help them when they're buying. We can do things like enable in-store feedback using a mobile device, and capture geolocation.”
Photo of Jason Burby - President, Americas, POSSIBLE“The location side of mobile optimization is really powerful. If you're in my app and I know you're in a store or near a competitor's store, what am I going to do to maximize that?”

Page 20 is available in the complete TrustRadius Buyer's Guide to A/B Testing and covers

  • Further Resources for Your Evaluation of A/B Testing Software

A/B Testing Products

Listings (1-25 of 61)

Qubit

194 Ratings

Qubit uses visitor history data to understand different user segments and serve personalized messages to segments using JavaScript. It is available as either a managed or self-service model. Data is collected using Qubit's own Universal Variable data model, or by integrating the user's existing...

Optimizely

233 Ratings

Optimizely is a website optimization platform, providing A/B and multivariate testing. It enables users with and without technical expertise to make dynamic changes to their websites, test the variations to live traffic and gather immediate results.

Monetate

93 Ratings

Monetate is a website optimization and personalization product. The company offers a strong product tailored to the needs of e-commerce and retailers. Customers are mostly enterprise brands.

Evergage

54 Ratings

Evergage’s cloud-based platform is designed to empower marketers to increase engagement conversions of website visitors and users through real-time personalization based on deep behavioral analytics...without the need for developers.

AB Tasty

44 Ratings

AB Tasty is a SAAS application created for e-marketers that enables them to optimize their website and conversion rate without technical knowledge. They can test several versions of their pages to identify which one has the biggest impact on their business objectives, e.g. click-through rate on a...

Maxymiser

82 Ratings

Maxymiser is a NYC-based company offering website optimization tools. The company offers strong professional services in addition to a technology platform. Maxymiser competes with Adobe Test and Target, SiteSpect, Optimizely, and Webtrends Optimize.

Adobe Target

109 Ratings

Adobe Test and Target is an A/B, multi-variate testing platform which Adobe acquired as part of the Omniture platform in 2009. It is now part of the Adobe Marketing Cloud. It offers tight integration with Adobe analytics and content management products.

Kameleoon

29 Ratings

Kameleoon is a website optimization solution. Its very first A/B testing solution was launched in 2012. The initial A/B testing solution was enhanced in 2014 with a module for iOS and Android applications and a platform dedicated to web personalization. A/B Testing & Website Optimization...

Visual Website Optimizer

58 Ratings

Visual Website Optimizer is an A/B testing tool that allows marketing professionals to create different versions of their websites and landing pages using a point-and-click editor, with no HTML knowledge required. Users can then see which version produces maximum conversion rate or sales....

Dynamic Yield

29 Ratings

Dynamic Yield is an open-ended omni-channel personalization and optimization platform designed to serve highly optimized customer experiences at scale. The vendor says that its SaaS-based solution empowers marketing professionals, eCommerce brands and media companies (publishers), with real-time...

Leadpages

40 Ratings

Leadpages is landing page software that aims to help businesses grow. The vendor says their customers use Leadpages to collect more leads which can drive more sales. The platform offers a variety of ways to engage potential customers--the vendor says that Leadpages differentiates within the...

Instapage

26 Ratings

Instapage is designed for post-click optimization and maximizes conversions for advertisers and marketers by enabling them to create, personalize, and optimize post-click experiences at scale. With Instapage, marketers can build personalized post-click experiences using Instablocks(TM) and update...

Exponea Experience Cloud

21 Ratings

With its marketing cloud, Exponea aims to provide real-time 1:1 omni-channel personalization, a single 360-degree customer view, and dynamic A/B testing with AI. It equips marketers to raise conversion rates, improve acquisition ROI, and maximize customer lifetime value. The vendor says Exponea...

Webtrends Optimize

25 Ratings

Webtrends Optimize is a web optimization platform and is offered in conjunction with a comprehensive set of professional services. The product competes most often with Adobe Test and Target, Maxymiser, SiteSpect, and Optimizely.

Google Content Experiments

71 Ratings

Google Content Experiments is a tool that can be used to create A/B test from within Google Analytics.

ClickTale

39 Ratings

ClickTale provides a full video playback option for each individual visitor's session (including mouse movement, clicks and keystrokes). In addition, it uses meta statistics to create visual heatmaps and behavioural reports, as well as conversion analytics and traditional web statistics.

SiteSpect

19 Ratings

Key features include: A/B, split, and multivariate testing campaign managementTargeting and personalizationFront-end usability testing Back-end testing using Origin ExperimentsSite acceleration with SiteSpect AMPS(R)Mobile site and native app support

Convert Experiences

8 Ratings

Convert Experiences is an enterprise A/B testing and personalization solution for conversion optimization and data-driven decisions in high-traffic websites. This solution offers A/B, split, and multivariate testing. It enables users to make dynamic changes to their websites, test the variations on...

Branch

5 Ratings

Branch provides solutions that are designed to unify user measurement across different devices, platforms, and channels. The vendor aims to help companies get a holistic view of various user touch points and ensure that links take the user to the right place on the website or native app.Customers...

Bound

5 Ratings

Bound (formerly Get Smart Content) is a website segmentation and targeting platform that enables marketers to read a website visitor's digital body language and serve relevant content based on the person's stage in the buying journey. Using behavioral, demographic, and firmographic data, Bound...

Zarget

5 Ratings

Zarget is an all-in-one website optimization tool offering A/B testing, heat maps, funnel analysis, and split URL testing. With its Chrome plugin, Zarget allows users to create experiments from the comfort of their own browser. Zarget lets users bypass login screens, and get real time reports as...

Campaigner

12 Ratings

Campaigner is an email marketing solution built by marketers that is designed to help small, medium and large businesses strengthen customer relationships and drive sales. The vendor's value proposition is that Campaigner provides advanced features such as industry-leading A/B split testing,...

Vero

1 Ratings

Vero is an email marketing solution built to make customer engagement simpler with features such as in-app behavior tracking, audience segmentation, and workflow automation.

BounceX

7 Ratings

BounceX calls itself a people-based marketing cloud. It includes behavior-based personalization capabilities for website content, email, and other channels.

A/B Testing Articles

Buyer's Guide to A/B Testing

  • Discover the 4 key factors for A/B Testing success based on insights from conversion experts
  • Quickly find out which A/B Testing software platform best supports your needs
  • Compare pros & cons of top 11 software based on 200+ in-depth reviews and ratings
 
Read more...

Top 10 Fastest Growing A/B Testing Software Products Q4 2014

Page views on TrustRadius for A/B Testing software products grew 15 percent from Q3 to Q4. The most evaluated AB Testing software product on TrustRadius in both Q4 by a wide margin was Optimizely with more than 4,000 unique page views in Q4 2014, followed by Monetate and Qubit.

 
Read more...

Survey Reveals Businesses Expect to Spend More on CRO Software Next Year

Earlier this summer, we announced the results of our 2014 Conversion Rate Optimization (CRO) survey. The survey asked respondents about their CRO practice maturity, software usage and spend. The results underscore the rapid adoption of Digital Analytics and A/B testing software among business of all sizes.  
Read more...

New Chart Compares Feature Availability Across 9 A/B Testing Software Products

Last month we published our first ever Buyer's Guide to A/B Testing Software, which ranked and explored the strengths and weaknesses of several A/B testing software products, based on end-user reviews on TrustRadius. We're following up with another useful tool for companies looking to choose the right A/B testing software: A feature comparison chart, which...  
Read more...

Top 10 Fastest Growing A/B Testing Software Products

A/B Testing is an emerging but rapidly growing software category.Page views for A/B Testing software products – a measure for how many people are running evaluations – grew 30 percent from Q2 to Q3. While page views are not an absolute representation of category growth for a variety of reasons (including our own improvement in search rankings), they are a good proxy. In this article, learn which are the fastest growing (most evaluated) products.  
Read more...