The Best A/B Testing Software by Market Segment

Methodology

Different companies have varying online testing and personalization needs. We have thus compared A/B testing tools across three different segments to help software buyers identify the solutions that most closely align with their needs. There are a few different ways one could segment the market. One method might be volume of website traffic or revenue generated by the website. Another might be the type of website, such as e-commerce, lead generating, brochure or advertising-based. For this guide, we have segmented by the size of the company using the product1 based on number of employees, as that was the data readily available to us from reviews on TrustRadius. Our market segments are:

  • Small Businesses: 1-50 employees
  • Mid-Size Companies: 51-500 employees
  • Enterprises: >500 employees

Number of employees is a useful though imperfect way of approximating complexity of website optimization needs. There are certainly exceptions. Some smaller companies might run high traffic websites that generate a lot of revenue and run more sophisticated testing and targeting operations. Some larger companies may be far less reliant on their websites. See the User Ratings and Feedback section of this guide for more comprehensive coverage of each product, including the strengths and areas for improvement.

Our goal was to create an objective visual depiction of which A/B testing vendors offer the leading solutions in each market segment. We have created each TrustMap on two dimensions:

  1. Average User Ratings: The average “Likelihood to Recommend” rating, which is a representation of overall satisfaction, by customers within that company-size segment who have written reviews on TrustRadius. All ratings and reviews come from authenticated end-users of the software.
  2. Segment Adoption: The estimated relative number of websites using the product within that company size segment. It is intended to facilitate a rough comparison of market penetration by order of magnitude among the products, as customer data is unavailable.

Because vendors do not typically share customer counts, we used public data from Builtwith.com, which estimates the number of websites using a particular technology based on tags in the website's code. Note that one customer might use a tool on multiple websites. Additionally, one website might use multiple A/B testing tools.

For the enterprise segment, we used the number of top 10k websites ranked by traffic using the software, per Builtwith.com. For mid-size company market, we used the number of the top 100k websites using the software and subtracted those also in the top 10k, which are likely enterprises. For the small business segment, we used data from the entire Internet and subtracted those also in the top 100k, which are likely mid-size companies or enterprises. Segment adoption is plotted on the X or horizontal axis and uses a logarithmic scale in order to create a more even spread.

We draw lines for the median Average User Rating and Segment Adoption to create a graph with four quadrants. Products in the upper right area of the quadrant, which are at or above the median for both Average User Rating and Segment Adoption, are “Leaders” for that segment. Tools in the upper left quadrant, which have an Average User Rating at or above the median but below-median Segment Adoption, are “Strong Performers”.

We have omitted products for which there are fewer than two reviews in a particular segment or high standard deviation among a small number of reviews (i.e., one very negative review bringing down an average). Otherwise, we do not make our own judgments as to whether a product is an “enterprise” solution, for example. We include products solely based upon whether they are used by that type of company, per the existence of reviews on TrustRadius. The TrustMaps will evolve to become more statistically accurate as we gather more data for each product.


1 Many consultants have written reviews of testing products on TrustRadius. Often, the consultant's company is a small business, but the product is being used for an enterprise client. In that case, we re-classified those reviewers according to the client's probable company size, for the purposes of this report. When company size was unknown, we eliminated the data point.