If you just want to get A/B going, just get Optimizely already.
May 02, 2014

If you just want to get A/B going, just get Optimizely already.

Anonymous | TrustRadius Reviewer
Score 9 out of 10
Vetted Review
Verified User

Software Version

Optimizely Platinum Service

Modules Used

  • Primary Testing Technology
  • Customized integrations for fuller eCommerce testing

Overall Satisfaction with Optimizely

Optimizely is used heavily across all our product lines of business' websites. It is primarily managed by our eCommerce team, executing tests asked for by business owners and/or through our own internal website optimization test ideas. The core of adopting Optimizely was to enable us to better manage website testing without the need for extensive development, due to our having a small eCommerce team. Its flexibility (after initial installs and modifications) allow non-technical colleagues to create and manage their own tests, which is vital given I am not able to support all items due to lack of resources beyond just myself. This is critical given the number of tests needed on an ongoing basis for us to continue to refine and improve our overall customer experiences.
  • Installing it is quite simple for more standard A / B testing. Only taking a few minutes to actually place code on site is a dream and you can almost immediately get tests designed, let alone launched. (If you had the right folks in place at the right time, I cannot imagine you could not get a live test going within a single business day.)
  • Their support team is highly active and responsive to requests for information or troubleshooting. (NOTE: We have the Platinum account giving us more dedicated support staff). They are highly knowledgable and able to answer most questions quickly; else, they are highly effective on researching issues with other members of their organization (which speaks to the company's fluidity)
  • The UI of the tool is very simple to use and elegant in design. That is an essential piece of why the tool is so effective / usable for non-technical staff.
  • Due to the nature of the tool, it can be difficult to make the tool work correctly with the standard install for tracking tests for more complex web pages, example being checkout pages. Due to checkout pages having a great amount of dynamic content (due to user input of forms, etc), it can conflict slightly with other online marketing tracking. Custom scripting is needed for this kind of testing, so its ease-of-install-use is a harder path for these more complicated pages.
  • There are certain times the online tool will not work correctly and have to relogin via a new browser session. In a crunch time for test creation, it can be a bit troublesome. (But a minor issue.)
  • The tool tends to be a bit "too smart" for tracking the changes made in creating a variation. Example is if you are working to change text on a page, say larger and different color. Through the process of using the tool to make the modifications, you possibly would try different sizes/colors. Each time you make a selection, the change is "logged" for the Optimizely test variation... this creates a great amount of "extra change code" that is unnecessary and have to clear it out via the code editor.
  • Highly effective in providing internal efficiency in getting A/B tests live
  • Has shown us already opportunities to improve overall customer pathing throughout our websites, resulting in more conversions
  • Enables us to quickly remedy serious errors in content and/or add elements to site that may not be able to be added due to other development projects "locking" website code bases. (Recently, we had a large development features update being done, and we needed to launch a small edit to our footer. We used Optimizely instead and took about 5 minutes!)
  • Adobe Test and Target,Google Analytics,Monetate,Magento
I've had limited experience with some of these, extensive with others. I think overall they are solid tools, and mainly depends on the organization you're working for. There are a few main reasons we selected Optimizely, considering we are a very small eCommerce department servicing an aggressive direct-response organization:

1) Budget vs. Ease of Use. The pricing model of Optimizely fit within our budget, especially considering the ease of use of the tool for most tests. The others we considered were just too expensive (and better for enterprise-level / high budget organizations.)
2) Integration. For most, really just adding the script to your websites. Tools like T&T, though valuable, are a MUCH steeper learning curve to get onto your website, let alone it working effectively for reporting back on results. (NOTE: I at one point was a reseller of the T&T product).
3) Comfort with the team. Overall, the Optimizely support teams are highly knowledgeable, approachable and always ready to help. Customer service is vital.
There's no question we'll continue with the tool. The ROI is too good. We've already embedded it recently into a new customized eCommerce platform we launched. The design of it had Optimizely in mind and is one of the technologies we won't continue without.

I'm also confident future refinements of the tool / extended functionality will address some of the minor headaches for the complex testing we wish to do. I am certain of this given the number of iterations I've seen since starting with the tool.

It's definitely one of the top three technologies we're using on a daily basis and that says a lot, as we never could test before.
The tool is fantastic for simpler, easy A/B testing. Just getting that strip of code on your site can enable a great number of test options. Multi-variate testing is also not too difficult, only for more complex coded web pages.

When considering the tool, I would highly suggest (almost require) that a demo with the Optimizely team has participants from your entire organization who'd be involved -- business owners for budget/ongoing Goals, marketing team who likely will use the tool ongoing to support business owners, technical/web ops individual to understand nature of the technology, analysts who'd monitor and report findings.

The key questions to ask when considering:

- Do I have enough technical support to install / test / finalize the overall Optimizely code install? Can they help if I need more technical integrations?
- Do I have the right design resources in place to assist with building out specific assets needed for tests?
- Are the right people in place to monitor and track the performance?
- Do I have 6 months worth of tests in mind so I can start right away?
- What is the overall budget capacity to continue paying for the service to ensure a return on the investment?

Using Optimizely

9 - The entire eCommerce team uses the tool, but each individual is a different part of the process. Designers use to create test variations. Technical folks use for installing the tests/working more complex testing. Analyst maintains the tests for ongoing reporting. This combination of expertise on a small team needs a tool like Optimizely in order to handle testing smoothly. It also speaks to its ease of use given the various types of individuals who can all use it effectively.
3 - To use it effectively, the ideal support team know basic web code to some degree. They also need a bit of creativity to understand/utilize the various test variation creation functions offered within the tool. Though its easy to use for even the most un-technical person, the management of the tool itself on the site / interactions with their support DOES need folks who are more technical in nature / background.
  • I need to test whether content A or content B will engage folks more on my website. (A/B)
  • I need to make an emergency change to the website and our webmaster is out of town. (Legal issues / quick fixes)
  • Would adding advertising to my site (banners) as a revenue stream impact my ability to convert people on my own site?
  • Adding snippets to my site due to not having ability to embed in actual site code.
  • Quickly hide pages within my site that were not supposed to be made live / but doing so remotely b/c I was not at the machine allowed to access the web server.
  • Utilize it for CSS changes globally across the site using the multi-page test variation.
  • Developing a full mobile-rich web experience with responsive design techniques / using Optimizely to understand what's working (since mobile optimization is somewhat new for us this year as a focus.)
  • Further split-test variations in relation to online marketing channel advertising / personalization of content.

Optimizely Implementation Rating and Lessons

In retrospect:

- I think I should have stressed more demo's / workshopping with the Optimizely team at the start. I felt too confident during demo stages, and when came time to actually start, I was a bit lost. (The answer is likely I should have had them on-hand for our first install.. they offered but I thought I was OK.)

- Really getting an understanding / asking them prior to install of how to make it really work for checkout pages / one that uses dynamic content or user interaction to determine what the UI does. Could have saved some time by addressing this at the beginning, as some things we needed to create on our site for Optimizely to "use" as a trigger for the variation test.

- Having a number of planned/hoped-for tests already in-hand before working with Optimizely team. Sharing those thoughts with them would likely have started conversations on additional things we needed to do to make them work (rather than figuring that out during the actual builds). Since I had development time available, I could have added more things to the baseline installation since my developers were already "looking under the hood" of the site.
Yes - Ph1 - Adding standard script to the website

Ph2 - Refine the script to allow for better full-path tracking along side of other OLM tracking (like GA), determined after some initial live tests.

Ph3 - Branching the technology across all other owned websites (each having a small flavor of the OLM and script enhancements.)
Change management was a minor issue with the implementation - It wasn't too difficult to utilize the technology once folks got comfortable. The change happened with the approach of "we can do tests, how do we decide what to do?" We needed to work with business owners to create an ongoing testing roadmap, which hadn't existed before. Before Optimizely, we were lucky to get even a few tests off the ground in a year's time. With Optimizely, the backlog of 50+ tests became the focus, and we had to really figure out how we manage this moving forward, as multiple tests would dilute results and/or clutter our entire website. A project management-esque process had to be introduced.
  • GA tracking. It would break and we couldn't see conversions via our GA reports for transactions involving tests. Took a bit of time with Optimizely to figure out a solution.
  • Being ready with tests. We were so focused on getting it installed, we didn't have anything on deck to actually test the implementation and had to resort to small minor changes to validate. We could have save a good amount of time had we been able to use the first planned test as the basis for ensuring the tech was working properly.
  • Getting the tech to span across entire website. Due to how our custom platform is developed, it was a bit more difficult than anticipated to install the Optimizely code across every single page of our website.

Optimizely Support

They are just fantastic. We have our main account rep but then a few others have helped us out in a pinch (or if he's not available.) Always respond within same day of a request and the advantage for ongoing account reps is they know our business, is growing with our technical enhancements and then lead-time to resolve issues is much shorter b/c they're are effectively part of our own team.
Quick Resolution
Good followup
Knowledgeable team
Problems get solved
No escalation required
Immediate help available
Support cares about my success
Quick Initial Response
Yes - We knew testing would be a HUGE focus for us for the coming years and was critical we had a clear path to helping troubleshoot and resolve issues. Having the premium support team available to us allows that freedom. Additionally, we know as a business we are launching new product lines / sites frequently, so having the additional support available to us ensures we can keep on track (and not have to worry too much about the increased level of investment... )
We had a big problem -- there was a huge desire to test messaging / layout on our shopping cart page. We were about 2 days from our promised deadline and had a huge amount of traffic coming with planned-for online marketing campaigns. We realized during our build-out that there was an issue with "activating" the variation on the page. Our site code was designed in a way that Optimizely couldn't "see" the elements we were testing unless something was in the cart. Raising the issue with the Optimizely team, they needed about three hours to determine the root-cause of the issue and a proposed solution. They provided this to us same-day and I took it to my development team. We then were able to get this proposed fix in place on our side (effectively creating a small "beacon" for Optimizely to see) and we managed to get the test running just in time. (And the result was great and we ended up installing the variation as our new control!)

Using Optimizely

It's overall great and easy to use. I think I don't rate it a 10 because I'm actually TOO technical... because its meant for a broad audience, I think I'm looking at the UI differently than someone with less technical expertise. It's to0 layman in some cases, but yet can also be confusing when dealing with their proprietary Optimizely programming language.
Like to use
Relatively simple
Easy to use
Well integrated
Feel confident using
  • Starting a variation. Quickly asking for the URL, quickly loads and you're off and running to make your variation changes.
  • Targeting the traffic. Their simple sliders and/or input % to allocate the traffic (and how you target them) is very easy to understand and use.
  • Adding the script. It generates in seconds after creating a project and elegant in delivering the link to you to use for integration.
  • Preview functionality. It sometimes takes a while for the live preview to push to their servers for a full check.
  • Google Analytics integration. It allows for the insertion of your account information, but the tracking/tool sometimes would break more complex pages like for a checkout, and then would render our GA data to be incorrect. Took some time to refine the tool / adding additional scripts to get it right. (NOTE: Our GA reporting needs are slightly custom as far as the metrics we're tracking..)
  • Editor Settings VS. Live pages to target. It's a bit confusing to understand the difference between the two and sometimes would get confused that I wasn't developing off correct editor setting VS what page(s) I was actually activating the test on.