A/B tests for non-coders done right
August 09, 2019

A/B tests for non-coders done right

César Costas Carrera | TrustRadius Reviewer
Score 6 out of 10
Vetted Review
Verified User

Overall Satisfaction with Optimizely

It is used by one team so far. It helps to address two problems: we do A/B testing, so we set up the tests and keep track of the results with Optimizely. We also wrap new developments into A/B tests so we can measure if we made a mistake towards the client's outcome, even if the expected result is to be flat (A being as good as B).

Pros

  • The traffic reports don't seem to be far from our internal traffic reports.
  • Non-coders appreciate the interface and graphs.

Cons

  • It's slow at loading and it's blocking, so it damages your site performance.
  • It uses a naive approach, where two tests running in the same page are still two A/B tests, not an A/B/C, damaging the results due to interference.
  • Rarely do we save from doing a big development after a test runs. The most common is we fail when testing a big development.
  • The increase in conversion is hard to measure. There are A/A tests (no changes) that report that one of the As is a winner and increases conversion.
  • It saves time as a quick "techie" CMS where you can customise more than the traditional copy and images.
The reports and the segmentation are better (or at least feel more accurate).
Optimizely is appropriate for promotional landing pages and small numbers of tests. It is not appropriate for big projects or for SEO landing pages.

Optimizely Web Experimentation Feature Ratings

5.1
a/b experiment testing
4
Split URL testing
6
Multivariate testing
Not Rated
Multi-page/funnel testing
7
Cross-browser testing
10
Mobile app testing
Not Rated
Test significance
4
Visual / WYSIWYG editor
1
Advanced code editor
3
Page surveys
Not Rated
Visitor recordings
8
Preview mode
6
Test duration calculator
4
Experiment scheduler
Not Rated
Experiment workflow and approval
Not Rated
Dynamic experiment activation
Not Rated
Client-side tests
1
Server-side tests
Not Rated
Mutually exclusive tests
7
5.0
Standard visitor segmentation
5
Behavioral visitor segmentation
Not Rated
Traffic allocation control
5
Website personalization
Not Rated
6.6
Heatmap tool
Not Rated
Click analytics
Not Rated
Scroll maps
Not Rated
Form fill analysis
Not Rated
Conversion tracking
7
Goal tracking
7
Test reporting
7
Results segmentation
5
CSV export
Not Rated
Experiments results dashboard
7
Platform Integration Features ()
Not Rated

Using Optimizely

Tracking the results is tedious. You need to browse the detail page to get any information and the loading times for the list of tests are massive.
The navigation & usability are so difficult that my team spends days browsing the tests and copying data into spreadsheets to make it quickly available and make sense visually.

Optimizely Reliability

As I mentioned before, tests have conflicting results between them, it slows down the loading times, and the more events/features/segmentation, the slower your site becomes.

Comments

More Reviews of Optimizely Web Experimentation

  1. Home
  2. A/B Testing Tools
  3. Optimizely Web Experimentation Reviews
  4. User Review of Optimizely Web Experimentation