If you just want to get A/B going, just get Optimizely already.
May 02, 2014
If you just want to get A/B going, just get Optimizely already.
Score 9 out of 10
Vetted Review
Verified User
Software Version
Optimizely Platinum Service
Modules Used
- Primary Testing Technology
- Customized integrations for fuller eCommerce testing
Overall Satisfaction with Optimizely
Optimizely is used heavily across all our product lines of business' websites. It is primarily managed by our eCommerce team, executing tests asked for by business owners and/or through our own internal website optimization test ideas. The core of adopting Optimizely was to enable us to better manage website testing without the need for extensive development, due to our having a small eCommerce team. Its flexibility (after initial installs and modifications) allow non-technical colleagues to create and manage their own tests, which is vital given I am not able to support all items due to lack of resources beyond just myself. This is critical given the number of tests needed on an ongoing basis for us to continue to refine and improve our overall customer experiences.
- Installing it is quite simple for more standard A / B testing. Only taking a few minutes to actually place code on site is a dream and you can almost immediately get tests designed, let alone launched. (If you had the right folks in place at the right time, I cannot imagine you could not get a live test going within a single business day.)
- Their support team is highly active and responsive to requests for information or troubleshooting. (NOTE: We have the Platinum account giving us more dedicated support staff). They are highly knowledgable and able to answer most questions quickly; else, they are highly effective on researching issues with other members of their organization (which speaks to the company's fluidity)
- The UI of the tool is very simple to use and elegant in design. That is an essential piece of why the tool is so effective / usable for non-technical staff.
- Due to the nature of the tool, it can be difficult to make the tool work correctly with the standard install for tracking tests for more complex web pages, example being checkout pages. Due to checkout pages having a great amount of dynamic content (due to user input of forms, etc), it can conflict slightly with other online marketing tracking. Custom scripting is needed for this kind of testing, so its ease-of-install-use is a harder path for these more complicated pages.
- There are certain times the online tool will not work correctly and have to relogin via a new browser session. In a crunch time for test creation, it can be a bit troublesome. (But a minor issue.)
- The tool tends to be a bit "too smart" for tracking the changes made in creating a variation. Example is if you are working to change text on a page, say larger and different color. Through the process of using the tool to make the modifications, you possibly would try different sizes/colors. Each time you make a selection, the change is "logged" for the Optimizely test variation... this creates a great amount of "extra change code" that is unnecessary and have to clear it out via the code editor.
- Highly effective in providing internal efficiency in getting A/B tests live
- Has shown us already opportunities to improve overall customer pathing throughout our websites, resulting in more conversions
- Enables us to quickly remedy serious errors in content and/or add elements to site that may not be able to be added due to other development projects "locking" website code bases. (Recently, we had a large development features update being done, and we needed to launch a small edit to our footer. We used Optimizely instead and took about 5 minutes!)
- Adobe Test and Target,Google Analytics,Monetate,Magento
I've had limited experience with some of these, extensive with others. I think overall they are solid tools, and mainly depends on the organization you're working for. There are a few main reasons we selected Optimizely, considering we are a very small eCommerce department servicing an aggressive direct-response organization:
1) Budget vs. Ease of Use. The pricing model of Optimizely fit within our budget, especially considering the ease of use of the tool for most tests. The others we considered were just too expensive (and better for enterprise-level / high budget organizations.)
2) Integration. For most, really just adding the script to your websites. Tools like T&T, though valuable, are a MUCH steeper learning curve to get onto your website, let alone it working effectively for reporting back on results. (NOTE: I at one point was a reseller of the T&T product).
3) Comfort with the team. Overall, the Optimizely support teams are highly knowledgeable, approachable and always ready to help. Customer service is vital.
1) Budget vs. Ease of Use. The pricing model of Optimizely fit within our budget, especially considering the ease of use of the tool for most tests. The others we considered were just too expensive (and better for enterprise-level / high budget organizations.)
2) Integration. For most, really just adding the script to your websites. Tools like T&T, though valuable, are a MUCH steeper learning curve to get onto your website, let alone it working effectively for reporting back on results. (NOTE: I at one point was a reseller of the T&T product).
3) Comfort with the team. Overall, the Optimizely support teams are highly knowledgeable, approachable and always ready to help. Customer service is vital.
Using Optimizely
9 - The entire eCommerce team uses the tool, but each individual is a different part of the process. Designers use to create test variations. Technical folks use for installing the tests/working more complex testing. Analyst maintains the tests for ongoing reporting. This combination of expertise on a small team needs a tool like Optimizely in order to handle testing smoothly. It also speaks to its ease of use given the various types of individuals who can all use it effectively.
3 - To use it effectively, the ideal support team know basic web code to some degree. They also need a bit of creativity to understand/utilize the various test variation creation functions offered within the tool. Though its easy to use for even the most un-technical person, the management of the tool itself on the site / interactions with their support DOES need folks who are more technical in nature / background.
- I need to test whether content A or content B will engage folks more on my website. (A/B)
- I need to make an emergency change to the website and our webmaster is out of town. (Legal issues / quick fixes)
- Would adding advertising to my site (banners) as a revenue stream impact my ability to convert people on my own site?
- Adding snippets to my site due to not having ability to embed in actual site code.
- Quickly hide pages within my site that were not supposed to be made live / but doing so remotely b/c I was not at the machine allowed to access the web server.
- Utilize it for CSS changes globally across the site using the multi-page test variation.
- Developing a full mobile-rich web experience with responsive design techniques / using Optimizely to understand what's working (since mobile optimization is somewhat new for us this year as a focus.)
- Further split-test variations in relation to online marketing channel advertising / personalization of content.
Optimizely Implementation Rating and Lessons
- Implemented in-house
Yes - Ph1 - Adding standard script to the website
Ph2 - Refine the script to allow for better full-path tracking along side of other OLM tracking (like GA), determined after some initial live tests.
Ph3 - Branching the technology across all other owned websites (each having a small flavor of the OLM and script enhancements.)
Ph2 - Refine the script to allow for better full-path tracking along side of other OLM tracking (like GA), determined after some initial live tests.
Ph3 - Branching the technology across all other owned websites (each having a small flavor of the OLM and script enhancements.)
Change management was a minor issue with the implementation - It wasn't too difficult to utilize the technology once folks got comfortable. The change happened with the approach of "we can do tests, how do we decide what to do?" We needed to work with business owners to create an ongoing testing roadmap, which hadn't existed before. Before Optimizely, we were lucky to get even a few tests off the ground in a year's time. With Optimizely, the backlog of 50+ tests became the focus, and we had to really figure out how we manage this moving forward, as multiple tests would dilute results and/or clutter our entire website. A project management-esque process had to be introduced.
- GA tracking. It would break and we couldn't see conversions via our GA reports for transactions involving tests. Took a bit of time with Optimizely to figure out a solution.
- Being ready with tests. We were so focused on getting it installed, we didn't have anything on deck to actually test the implementation and had to resort to small minor changes to validate. We could have save a good amount of time had we been able to use the first planned test as the basis for ensuring the tech was working properly.
- Getting the tech to span across entire website. Due to how our custom platform is developed, it was a bit more difficult than anticipated to install the Optimizely code across every single page of our website.
Optimizely Support
| Pros | Cons |
|---|---|
Quick Resolution Good followup Knowledgeable team Problems get solved No escalation required Immediate help available Support cares about my success Quick Initial Response | None |
Yes - We knew testing would be a HUGE focus for us for the coming years and was critical we had a clear path to helping troubleshoot and resolve issues. Having the premium support team available to us allows that freedom. Additionally, we know as a business we are launching new product lines / sites frequently, so having the additional support available to us ensures we can keep on track (and not have to worry too much about the increased level of investment... )
We had a big problem -- there was a huge desire to test messaging / layout on our shopping cart page. We were about 2 days from our promised deadline and had a huge amount of traffic coming with planned-for online marketing campaigns. We realized during our build-out that there was an issue with "activating" the variation on the page. Our site code was designed in a way that Optimizely couldn't "see" the elements we were testing unless something was in the cart. Raising the issue with the Optimizely team, they needed about three hours to determine the root-cause of the issue and a proposed solution. They provided this to us same-day and I took it to my development team. We then were able to get this proposed fix in place on our side (effectively creating a small "beacon" for Optimizely to see) and we managed to get the test running just in time. (And the result was great and we ended up installing the variation as our new control!)
Using Optimizely
| Pros | Cons |
|---|---|
Like to use Relatively simple Easy to use Well integrated Consistent Convenient Feel confident using | None |
- Starting a variation. Quickly asking for the URL, quickly loads and you're off and running to make your variation changes.
- Targeting the traffic. Their simple sliders and/or input % to allocate the traffic (and how you target them) is very easy to understand and use.
- Adding the script. It generates in seconds after creating a project and elegant in delivering the link to you to use for integration.
- Preview functionality. It sometimes takes a while for the live preview to push to their servers for a full check.
- Google Analytics integration. It allows for the insertion of your account information, but the tracking/tool sometimes would break more complex pages like for a checkout, and then would render our GA data to be incorrect. Took some time to refine the tool / adding additional scripts to get it right. (NOTE: Our GA reporting needs are slightly custom as far as the metrics we're tracking..)
- Editor Settings VS. Live pages to target. It's a bit confusing to understand the difference between the two and sometimes would get confused that I wasn't developing off correct editor setting VS what page(s) I was actually activating the test on.
Yes, but I don't use it