Item: Optimizely Web Experimentation
Rating: 9
Author: Verified User

Use Cases and Deployment Scope

We use Optimizely Web Experimentation for content testing (images and/or copy), recommendation algorithm tests for product carousels, and feature testing (net new features that we develop and want to validate & calculate revenue incrementality before launching). With the first two: we want to test along the customer journey to understand what pieces of content or product assortments provide the best benefit to her on site. We also use audience segmenting to be able to target the right users (coming from X marketing channel, mobile-only users, or logged-in users) to be able to refine our testing.

Pros and Cons

Extensions - allows the company to tailor the tool to unlock certain capabilities
Developer-friendly, with GA and other integrations
Intuitive results page that can be shared out

Hard to use extensions without some knowledge of HTML/CSS
Inconsistent user counts between GA & Optimizely when it comes to calculating MAUs, hard to be able to forecast budget and overages when our source of truth (GA) differs from our testing platform (Optimizely)
Impressions model doesn't support scaling personalization experiences - hard to run a serious data driven testing program when you have to cut tests short before 14 days (ideal length to get to any statistical read) in order to save on impressions

Return on Investment

Validated business case for going into new testing approach
Increased conversions from certain audiences by running and launching successful experiences tailored to those users

Scalability

It has been easy to deploy Optimizely on our site, however, we continuously monitor the load of the Optimizely PJS on our site, and we find that the load of having extensions, custom metrics, custom audiences, targeted tests, tests that have been paused but not archived, all have a negative impact on the site in that it is extra Javascript that might weigh down the load of the page and hurt our page performance. We haven't yet been easily able to manage this with the current Optimizely set up we use – it requires a lot of manual maintenance and attention to ensure it is kept as clean as possible.

Optimizely Platform Scope

For launch and analysis: it is so helpful for us to only look at one report to see key test results. For targeted experiences: if we're testing on Paid Search audiences, and if we want to launch that experience at 100%, it is much easier for us to scale that experience out.

Alternatives Considered

Adobe Target

We use AEM so Adobe Target would be a natural choice, it integrates naturally with the Experience Fragments and all the content we already hold there. However - with extensions - we've been able to unlock a similar workflow to be able to seamlessly test. Optimizely has the unique capability to be able to feature test, which is how we have been able to evolve our onsite experience and not only be able to quantify the revenue impact of our content or copy changes, but of entire site changes.

Other Software Used

Salesforce Commerce Cloud, Adobe Experience Manager

Likelihood to Recommend

Optimizely Web Experimentation is well suited for a team that tests across multiple facets of testing – full tech developed features, pieces of content or recommendations, landing page optimization. However - if a team only does limited testing, and only in the content testing area, other tools may integrate better IF they are not using Optimizely's CMS (and if they have a CMS that easily supports testing - like Adobe AEM x Adobe Target). However - as testing scales - Optimizely Web Experimentation will start to make more and more sense.

a program for beginner experimentation, all the way to a mature testing & personalization program

Overall Satisfaction with Optimizely Web Experimentation