Item: Optimizely Web Experimentation
Rating: 7
Author: Verified User

Overall Satisfaction with Optimizely

Use Cases and Deployment Scope

Used for simple A/B tests and to force visitor traffic to pages that have specific messaging or UI elements

Pros and Cons

Pros

Very easy to configure experiment targeting, traffic allocation, and other settings you would want to customize for your testing program
Allows easy selection of experiment types like Multivariate, Multipage, A/B
Permits simple UI changes using its code editor and other options which allows for decreased reliance on development resources. Product and Design teams can launch new experiments easily on their own
Support is first rate: knowledgeable, great communicators who take the time to explain things so that you can learn and apply to future experiments. Optimizely customer support is among the best I've encountered from a web software service

Cons

Improvements can be made with experiment results reporting specifically customization of data within the data export or whatever is viewed on the summary page
Different organizations could have different thresholds for statistical significance. Optimizely is set to 95%. It would be nice to have this value configurable.
It is difficult to ascertain whether a losing variation is a statistically significant loss (only shows low chance to beat baseline). Having this would assist with pausing variations that are not performing very well.
Customizable, distributable reports would be a great feature to add. Currently you must click through each individual experiment to see experiment details. Seeing a summary through email or selecting multiple experiments to view on one page would be a huge improvement for scanning health of tests.

Return on Investment

Optimizely has helped us optimize on micro conversion goals throughout our site
Our macro conversion goals have also been optimized using Optimizely

Alternatives Considered

Homegrown A/B platform and Adobe Test & Target

Not as strong.

Other Software Used

Geckoboard, Asana, Trello, JIRA Software

Review how front end or back end-heavy your tests plans will be. Optimizely allows for easy implementation of front end changes which won't bog down development resources. They should be able to focus on bigger fish than changing button colors, texts, or adding images to pages.

Using Optimizely

Users and Roles

5 - Growth, Digital Product, Engineering

Support Headcount Required

1 - A/B testing knowledge, Front End JS/CSS, QA

Business Processes Supported

Test new ideas/variations against current control version
Force visitors to see certain site changes
Reports to understand impact of site change

Innovative Uses

Segment traffic from the various inbound channels

Future Planned Uses

Analytics
Performance indicators

Likelihood to Renew

Overall a great experience from ease of use and flexibility for the tool. Reporting improvements could be made, but what exists currently is enough to give businesses a sense of the directional movement of a change to the site.