Google Content Experiments (discontinued)

Real time data

Personalization

Back end development needed for detailed tests.

Some training required.

Has a great analytics engine in its backend which uses multi-arm bandit methodology- and thus can perform multiple variations at once.

Multi-arm bandit means that it's also really effective in finding the winning solution.

It can be based on Analytics Goals via Optimize so you can drive things that are important to the business.

Their documentation is not the best and it's quite a steep learning curve.

They also don't tell you particularly well what sorts of things you should be testing.

Compared to other suppliers of A/B testing tools- it needs a simpler interface. Optimize is starting to answer that - but is still quite Beta-like.

Easy to use and implement.

Easy to understand the reports.

Seamless integration with Google Analytics.

No support for multivariate testing, a feature that was pulled from the product when it was rebranded as Google Content Experiments from Google Website Optimizer.

Free

Easy to use if you already use Google Analytics - literally a few minutes to set up a test

Fully integrated with Google Analytics - so you can use your existing conversion goals, and no confusion over metrics.

Multivariate testing

Ability to drop losing variants during a test

Ability to manually choose split of traffic between variants

It is free

Tests are easy to set up. Experiment creation is a 4-step process that can be completed with limited knowledge of Google Analytics

Experiments is integrated with your Google Analytics account, utilizing existing goals or allows you to create new goals for testing directly within the set up area

Allows for advanced segmentation, filtering, and traffic allocation

No multivariate tests: Google Content Experiments tests on an A|B|n platform. As such, you must create full iterations of landing pages to test against each other. If you are testing a full layout change, the A|B|n model will suit your needs just fine. However, if you’d like to test multiple elements at one time or a small element, like button color, you would need to create a new landing page for each different color button you will test. This can become quite cumbersome.

Basic testing platform: This platform will also present challenges if you would like to test the influence of one element across multiple pages. You will need a much more sophisticated tool for this type of testing.

Content Experiments defaults to a Multi-Armed Bandit algorithm. This automatically adjusts the amount of traffic allocated to each variation based on sample size and performance metrics, resulting in less traffic to "under-performing" variations. In theory the Multi-Armed Bandit is much more efficient than your classic A|B testing method, concluding tests much quicker while reducing potential revenue loss. Unfortunately, we have found that for testing with small sample sizes, like those commonly conducted for B2Bs or SMBs, the multi-armed bandit has the potential to create an invalid test that will never declare a winner. This is because traffic to the variation is typically reduced so severely and so quickly that there is not a significant enough sample size to give it a competitive opportunity.

Testing on a specific segmentation is challenging: To test on a distinct segment of your visitors requires more advanced knowledge of Analytics to set up a new filter view and cannot be completed within Google Content Experiments.

Not ideal for marketers who want independence from IT: Content Experiments requires a snippet of code being placed on one experiment page for each test. It also requires a full version of the test page with a unique URL.

It's FREE

It ties into Google Analytics

Requires code setup for each experiment (less of an issue if also using Google Tag Manager)

Does not allow for multivariate testing

Does not allow for advancing testing and targeting

A / B testing: You define a control page (page A) and a variation (page B) of that original page to test against. The purpose of this test is to expose your audience to the different versions of a page to determine which version will result in more conversions for your site.

Does not support multivariate testing

Seamless integration with Google Analytics

Absolutely free of charge

Once Google's Universal Analytics supports Content Experiments and requires no additional javascript to implement, it will be a truly seamless experience.

It is free to get going - better than the competitors!

You use your Google Analytics conversions as your objectives, so as long as your goals are set up in GA then it's a breeze.

You can only optimise for one goal, so if you have several conversions, like phone call and email you need to do it manually.

It seems not to work that well for pages with lower amounts of traffic - not great for new or niche sites.

Integrates well with Google Analytics to perform segmentation analysis

Capability of using server redirects rather than java script redirects unlike most testing tools.

Easy to set up

It doesn't handle multivariate testing

Basic test configuration compared to other testing tools in the market

Still requires a developer to code new pages rather than CMS capabilities of some products

Quick and easy to create and set up experiments

Results are presented in a way that is familiar and easy for any Google Analytics user to understand. So is great for beginners to conversion testing

Already integrated into Google Analytics and can measure results against your existing conversion goals

Allows nine possible variants to be A/B tested

Features more than one testing methodology, Bayesian (Multi-Armed Bandit,) and Full Factorial

Allows the user to select one of three possible confidence thresholds to ensure that experiment results are robust

Great value, it is free!

Should include a multivariate testing (MVT) option. Whilst the ability to test nine possible variants using A/B/n testing is great for strong bold tests, the ability test multiple elements on the same page is essential for a successful testing program

Reliance on your IT department to code the variants you wish to test and also to insert the experiment code on the test page can be an issue which may slow the testing process for some businesses

It is very easy to make the wrong decisions and find a false winner. A high level of conversion rate optimisation knowledge is needed to ensure that the results from the experiment are valid and of statistical significance

Should not default to Bayesian (multi armed bandit,) methodology and a 95 percent confidence threshold as a those new to testing may not fully understand the impact of these setting and whilst these options can be changed in “Advanced Options” the combination of these two factors could easily find an incorrect winner

Google Experiments should not declare the winning variant as it does not understand your business or business cycles, so can easily find a false winner. Although the option to set a minimum time for the experiment to run before declaring a winner goes some way towards mitigating this issue it is limited to a maximum of two weeks which may not be long enough for some businesses

When you need to measure against event-based goals

If you need to see how the test variations performed against secondary goals

Given that the the platform requires you actually code a new page with a unique URL, this tool can be good for radical redesigns.

Great insights into other information about your testing groups, like whether or not they're mobile, screen size, browser, or really any dimension available in GA.

Not a great solution for people who don't have the ability to code a page from scratch, or need to implement a test without the help of IT. This tool requires implementation of a few different code snippets on different pages and the ability to code a new page. If you're looking for a WYSIWYG editor - try Optimizely.

Google CE really gives in-depth insight into why some pages perform better than others. You can see where the readers eyes travel most often.

Talk about putting yourself in the audience's shoes! Thats exactly what A/B testing does

There is also multivariate testing for advanced changes.

There was a somewhat high learning curve which I would like to see flattened.

Google needs a more robust help section integrated in CE to help users navigate better.

There should be more mock testing windows during evaluation stages.