A/B Testing Tools Overview
What is A/B Testing?
A/B or split testing divides web traffic among multiple versions of a webpage and provides information on which version performs better at achieving a desired outcome, such as signups, downloads, demo requests, or purchases.
The testing process aims to optimize a website’s ability to convert visitors into leads or customers. There are two primary goals associated with A/B testing:
To increase revenue without spending more money to acquire traffic.
To prove that a new design concept or feature is worth creation before budgeting engineering and design resources to create it.
A/B Testing Software Features
All A/B testing tools include the following features:
Splitting traffic among the different variations being tested
Calculating conversion rates
Measuring statistical likelihood that one variation will consistently outperform the other
Basic A/B testing tools may require custom design and code. Some advanced tools offer a WYSIWYG interface for creating variations without needing to use HTML or CSS. This works well for non-technical users.
More sophisticated tools can run targeted campaigns. These tools display different versions based on demographics, device type, web visitor behavior, or referral source. They can also perform multi-page campaigns. This means testing one element (like a banner) on multiple pages during a session. These advanced tools help create a personalized user experience, with the goal of maximizing conversions.
A/B Test Best Practices
It is generally best to change one element of a page at a time so that any change in outcome can be attributed to that specific change. With this approach, you’ll know which element results in better conversion.
Elements to test include:
Certain split testing tools allow multivariate testing. This allows you to change multiple elements on one page, creating countless combinations. An analyst can then measure the impact of each variable on conversion.
Split testing should run long enough to reach statistical significance - at least 95% confidence. The amount of time needed varies depending on the amount of traffic to the page. Because they involve many versions, multivariate tests require more time and traffic to reach statistical significance. So if you plan on running multivariate tests, budget more time for the experiment than you would for a simpler A/B test.
A/B Testing and Conversion Rate Optimization (CRO)
A/B testing is often part of conversion rate optimization. CRO refers to the process of continually improving a website’s effectiveness at reaching its goal and increasing conversions. Usually, the goal is to turn visitors into customers or leads. However, other CRO goals can be things like downloading a whitepaper or increasing the number of event registrations.
It is usually a cyclical workflow, involving multiple steps:
Monitoring website visitor behavior
Gathering insight and visitor feedback
Testing a hypothesis for improving conversions
Key technologies in the CRO workflow include:
In-page web analytics (heat mapping, visitor recordings, etc.)
A/B testing tools
A/B Testing Software Comparison
Before investing in an A/B Testing solution, consider these three key factors:
Use case: who will be using the tool the most within your organization and what goals are they trying to achieve? For example, will the marketing team be the primary users of the A/B testing tool? Or will web developers use the tool to help test out new feature ideas too? This matters because not all tools are equally as user-friendly for both types of users. WYSIWYG editors are a must-have feature for marketing teams, while developers and engineers users likely need access to advanced A/B testing capabilities.
Support options: what customer support options are available for each A/B testing tool you’re considering? Vendors may provide various levels of support, such as phone, email, documentation, live chat, or automated support. If you’re new to the world of A/B testing, training, courses, and certifications can be valuable resources.
In-house vs. agency route: does your business need an in-house A/B testing solution, or would it be better served by working with a conversion rate optimization (CRO) agency? If you decide to go the in-house route, you’ll need to purchase an A/B testing platform and make sure you have the resources set aside to manage the tool. In contrast, all the operating costs and resources will be taken care of if you decide to work with a CRO agency. However, CRO services can be substantially more expensive than basic A/B testing tools.
Another thing to consider is that agencies will have a wealth of knowledge and experience behind them, while your in-house A/B testing program may not. But your own A/B testing program will be solely focused on your own business goals in a way that an agency will not be.
Getting pricing information for a specific A/B testing tool typically requires asking for a quote. You can find basic plans for as little as $50 per month. The more sophisticated tools can run up to $18,000 per year.
Frequently Asked Questions
A/B testing is important because it helps companies make evidence-based decisions about how to design and structure their website. Making decisions based only on intuitions or the most popular idea can lead to bad design choices, and ultimately cost your website in terms of organic traffic or other key performance indicators (KPIs).
A/B testing can also help engineering and development teams test out new feature ideas before devoting resources to producing the new feature. This way, teams can get a preview of how the new feature will be received and work out any kinks before creating the new feature.
There are many different A/B testing tips and tricks you can find on the internet. But some of the most important best practices to keep in mind are:
- If you’re just starting your A/B testing program, get buy-in from executive stakeholders
- When you can help it, only test one element at a time
- Make sure to let your experiments run long enough to reach statistical significance
- Think about how the test variations will render on different device types (e.g. desktop, mobile, tablet)
For more information on these tools and what features they include, see our A/B testing software category page.
According to end-user feedback collected on TrustRadius.com, the top-rated A/B testing tools include:
For more information about which products are better suited for specific company sizes, view our A/B Testing Software TrustMap. TrustRadius also has hundreds of reviews of over 50 A/B testing solutions that include detailed information about product pros and cons.
The price of A/B testing tools can vary depending on the number of features included. More basic tools can start at $50 per month while more comprehensive platforms can cost upwards of multiple thousands of dollars per month.
Many A/B testing vendors don’t directly disclose pricing information on their website but will provide a quote upon request.