Trends in A/B Testing Software

Democratization of Access

As we discussed in our Buyer's Guide to Digital Analytics Software, the free version of Google Analytics helped democratize access to web analytics and enabled a massive culture shift in digital marketing toward data-driven decision making. Similarly, the introduction of easy-to-use, low-cost A/B testing tools has facilitated a rise in popularity of A/B testing.

According to a recent survey of TrustRadius members and website visitors interested in conversion rate optimization, 43% of respondents use more than one A/B testing tool, and 48% of respondents plan to spend more on A/B testing software in the coming year. Online testing is not as ubiquitous as web analysis, however. The survey revealed that all respondents are using at least one web analytics tool, but 9% of respondents aren't using an A/B testing tool at all.

TrustRadius Survey Results for Number of A/B Testing Tools Used for CRO

TrustRadius Survey Results for Next Years Spending on A/B Testing Software

Most of the experts we spoke to agree that split testing is becoming more widely adopted as a practice in digital marketing.

Photo of Justin Rondeau - Chief Testing Evangelist and Editor, WhichTestWon“Testing is becoming more mainstream. Years ago it was difficult to get a program implemented, but now the emerging technology has reduced resource constraints and made the barrier of entry to testing really low.”
Photo of Hiten Shah - Co-founder, KISSmetrics and Crazy Egg“Historically, we saw a lot of designers focused on usability testing using Crazy Egg. Now, marketers and product people are using the tool to inform their A/B testing. This is partly because of Optimizely and GWO [Google Website Optimizer] prior to that; everybody is thinking about A/B testing.”

This also means testing is becoming more widely adopted within organizations—being practiced across various business units rather than restricted to the marketing team. This is perhaps how companies begin using two or more A/B testing tools, each targeted at a different technical skill level.

Photo of Justin Bougher - Director of Product Management, SiteSpect“The optimization team is shifting from an isolated marketing group to a centralized function that works with all parts of the organization.”

Personalization: The Next Big Thing?

Some software vendors appear to be ahead of the rest of the market on personalization. Personalization is a form of targeting that attempts to serve up the most relevant content to a customer or visitor on a near 1:1 basis. Many of the vendors we spoke to said personalization was a core feature of their product. However, some consultants said their clients don't have the resources yet to practice personalization on their websites.

Photo of Ryan Garner - Co-founder, Clearhead“Our clients barely have enough time to create three banners, much less individualized experiences for each customer. A few of our larger clients are really targeting four or five segments. One-to-one personalization has a ton of potential, though in practice we're not seeing a lot of it yet.”

Some see it as an incredibly valuable capability in conversion rate optimization.

Photo of Jason Burby - President, Americas, POSSIBLE“Personalization will absolutely lead to greater engagement and conversion. Do you respond better to things that are relevant and personal to you? Of course. It's not a topic to be disputed.”

According to a 2014 survey by Ascend2, 77% of marketers do not use personalization on their website at all, and only 5% use it extensively. Of those that do use personalization, the majority are manually creating multiple versions of webpages for different customer segments rather than using marketing technology to automate the generation of personalized content. The concept is becoming more mainstream, however.

TrustRadius Survey Results for Company Usage of Personalization

Photo of Pete Koomen - Co-founder and President, Optimizely“A/B testing was not a muscle most companies had a few years ago. We've seen that change dramatically over the past few years. We see a similar trend with personalization. It is a powerful value proposition and most companies are in the early stages of building that muscle. We see it as a direction the industry is going.”
Photo of Justin Rondeau - Chief Testing Evangelist and Editor, WhichTestWon“We've seen that one of the most underused features of testing technology has been segmentation and personalization, but that's changed. It's been increasing.”
Photo of Ryan Garner - Co-founder, Clearhead“We are seeing that the personalization tools offered by many of the optimization platforms today don't scale. They provide interesting targeting capabilities, but then require a person to manually set up and manage all the rules and logic for what to offer their customers. We believe it's the algorithms that are ultimately going to win this space.”

Some in the industry think personalization has the potential to make visitors concerned for their privacy. Others disagree.

Photo of Bruce Ernst - VP of Product Management, Monetate“One of the flip sides of personalization is I may start seeing content oriented toward something I thought was private. We're seeing a lot of concern with that, and a trend toward not crossing that ‘creepy' line, so we're building in guardrails to our product.”
Photo of Jason Burby - President, Americas, POSSIBLE“With ultimate personalization, there's this idea that people can get scared. You hear about privacy, or you'll hear concerns about freaking people out, or that it's too complex. Personalization is a journey. You crawl, then walk. A little more relevant is better than not relevant at all. Then you keep moving down that spectrum.”

Will Testing Be Standalone or Embedded in Other Tools?

The software covered in this report is focused on A/B testing a website. However, split testing is a concept that can be used in any marketing channel, including mobile apps, email marketing, search engine marketing, display advertising, and even offline. A/B testing therefore could potentially become a common feature embedded in different types of software, such as marketing automation, content management systems and e-commerce platforms. Unbounce, for example, is actually a landing page builder that also allows users to A/B test their landing pages.

Photo of Paras Chopra - Founder and CEO, Wingify“Some email or marketing automation tools offer testing, but I don't think they do it properly. Just having a testing feature does not do it justice. The main value is that these tools let you create experiences in a quick manner. You might have to plug in a lot of coding, or what you can test is limited. A lot of these tools do not even include statistical significance in their reporting.”

Alternatively, some standalone A/B testing tools are enabling split testing of different channels, such as email and mobile apps.

Photo of Ryan Garner - Co-founder, Clearhead“Does it make sense to bake testing into an email product or a CMS or e-commerce platform, versus coming from a third-party tool? Some vendors believe yes. Just like we saw in the web analytics space, we believe there's value in having a specialized product with broad reach, which allows you to optimize across platforms.”

One key capability of standalone testing tools is integration with web analytics software. This allows the testing tool to use visitor segments already set up in the analytics platform, and the analytics tool to access and report on testing data. Some experts discussed whether analytics and testing software might one day converge. Content Experiments, for example, is an A/B testing feature embedded in the web analytics tool Google Analytics. Mixpanel, a web analytics tool focused on event-based tracking, recently added A/B testing of iOS apps to its analytics platform.

Photo of Justin Rondeau - Chief Testing Evangelist and Editor, WhichTestWon“For now, integrating standalone testing and analytics tools is sufficient. Having them converge isn't necessary. As we get more sophisticated, however, that might change.”
“I think the landscape is changing considerably in that they're getting closer and closer to being tied together. It's good for organizations because there are different counting methodologies with different testing solutions versus analytics. It creates frustration and data validation issues. As the testing solutions get more deeply integrated with analytics, organizations are going to be empowered to make use of historical data.”
Analytics and Testing Consultant

Mobile App Testing and Optimization

As the use and monetization of mobile apps evolve, optimizing native apps has become more important.

Photo of Richard White, CEO, UserVoice“If you go back three years on mobile, 80% of revenue is paid apps. You buy the app for $3 and that's it. It's not a strong environment to care about retention or loyalty or optimization. Now, 80% of revenue is in-app purchases. Getting customer feedback in the app is important.”

This means app owners are more interested in conversion rate optimization software for apps, such as A/B testing and customer feedback tools. Point solutions focused on split testing iOS and Android apps have emerged, such as Artisan, Apptimize and Leanplum. Additionally, many of the tools covered in this report offer software developer kits for A/B testing iOS or Android apps.

Photo of Ryan Garner - Co-founder, Clearhead“We tend to have a unique testing roadmap for each device with our clients—mobile, tablet and desktop. Mobile is a growing area where our clients are increasingly testing. We're also seeing a lot of vendors aggressively pursuing native app testing.”

It's not just about adapting web-focused CRO technology to mobile apps, however. Mobile changes how companies can approach their conversion rate optimization strategies.

Photo of Mark Treschl, President and CTO, OpinionLab“Mobile is huge, because it provides us with the opportunity to interface with customers that are in the moment when it comes to physical experiences. We know that folks are out in the world, using smartphones to help them when they're buying. We can do things like enable in-store feedback using a mobile device, and capture geolocation.”
Photo of Jason Burby - President, Americas, POSSIBLE“The location side of mobile optimization is really powerful. If you're in my app and I know you're in a store or near a competitor's store, what am I going to do to maximize that?”