Optimizely Web Experimentation vs. VWO

Optimizely is a great A/B testing tool for any size business. The Optimizely platform democratizes A/B testing through a simple interface, excellent training tools, easy implementation and low dependence on IT resources. Utilizing Optimizely I've been able to run robust experimentation programs with a variety of resources (acting as a 1-person team with limited access to IT resources as well as managing a team with high access to resources). Additionally, Optimizely is well-priced and positioned to be affordable for a variety of business sizes.

Our marketing team does a lot of creative testing around messaging and imagery. VWO is well suited for this type of testing and can yield great results if you define your conversion goals correctly. However, when we have tried to do more substantial/advanced changes through VWO (such as re-positioning content or modifying elements in a form), we have had some challenges and not been able to get the desired tests working correctly.

• Optimizely makes it extremely easy to setup almost any type of split or multi-variate testing experiment. It has a variety of test triggers types & many ways to segment incoming visitors.
• I believe Optimizely's results dashboard is one of the best among split testing software. It allowed my team to get insights into the running of tests, like time to significance and a view of metric changes over time.
• Optimizely is very easy to use for non-engineers. It has a powerful visual interface that allows marketers and advertisers to setup complex variations without 1 line of code.
• In terms of documentation and implementation, I believe Optimizely is ahead of its competitors in terms of its focus on reliability and speed.

• VWO has a strong support team willing to help provide answers to questions during the setup process. During setup, we had some questions regarding implementation across a significant number of sites and they were able to ease dev team concerns and provide detailed best practices to streamline integration.
• Some of the initial results of a few of the initial tests raised some questions internally. We setup a call and were able to quickly address the questions and find some opportunities to leverage moving forward.
• I discovered a bug within the UI that lead to some questions internally. I was able to report it and the problem was fixed rather quickly and their team followed up with a thank you for reporting it and to report the issue was now resolved.

• Actual creation of an A/B test is confusing and difficult (like the differences between Pages vs URLs etc.).
• Setting up a URL Redirect Experiment is so convoluted it seems like it's not even a feature Optimizely wants you to use. Instead, it's like I'm tricking Optimizely into alternating between 2 URLs. That really needs to be cleaned up as it would be very helpful.

• The user interface within VWO does take a bit of time to get used to, especially as it pertains to switching back and forth between tests. When running multiple experiments on a site at a time, a clear and succinct dashboard for everything in one place would be helpful (as opposed to needing to switch between A/B, multivariate, etc).

Because it's an incredible and essential tool for my line of work as a conversion optimization specialist. Really couldn't do my job nearly as effectively without it. It's paid for itself many times over and I feel like I'm only beginning to unlock the tools potential.

It's great value and we think we've ironed out all the major teething troubles. However, if we experience any more bugs or problems that significantly slow us down then we're seriously considering switching to Optimizely, which I haven't personally tested but have heard great things about from my CRO peers

Without a doubt, after 8 years using various different testing tools, Optimizely is by far the easiest to use. Feedback from developers and more technical teams has always been positive, mostly due to the code editor allowing for easy implementation along with the creation of advanced audience segments using advanced javascript. For business users and non-technical people, the wysiwig editor allows for smaller changes to be made quickly whilst somewhat educating the user on the types of code changes that are being made.

I gave Visual Website Optimizer a rating of 8 because it is overall a great product to use. Setting up and keeping track of various tests is easy and straight forward. The only reason why this product is not rated higher is because the support documents online leave a lot of room for improvement.

Web interface is nearly always available. Javascript snippet (which actually runs the test) is distributed on a CDN for near-constant availability.

I have not experienced an application error or unplanned outage with VWO.

We have had some issues with screen flicker when loading experiment content. We had to dial back some of the creative changes to load smaller files.

VWO doesn't appear to slow down our website at all, though some customers with adblockers like UBlock Origin have been known to not see entire pages if VWO is making changes to the page at a macro level (background, font, etc). This is rare though.

They always are quick to respond, and are so friendly and helpful. They always answer the phone right away. And [they are] always willing to not only help you with your problem, but if you need ideas they have suggestions as well.

While their online document support is lacking a simple email to their support team will almost always get responded to the next day. It has however taken more than one email to explain the problem to the support team till they understood the problem. The solution I was given also only half fixed the problem the rest I figured out on my own.

The tool itself is not very difficult to use so training was not very useful in my opinion. It did not also account for success events more complex than a click (which my company being ecommerce is looking to examine more than a mere click).

Training was good, just limited to the onboarding process. They walked through all of the steps it takes to get started in VWO and each of the modules, along with giving us ideas for starting our first test. I feel like it could be better if there was a guided process within the VWO program to continue to educate you along the way, and a way to turn that off for experienced users.

In retrospect: - I think I should have stressed more demo's / workshopping with the Optimizely team at the start. I felt too confident during demo stages, and when came time to actually start, I was a bit lost. (The answer is likely I should have had them on-hand for our first install.. they offered but I thought I was OK.) - Really getting an understanding / asking them prior to install of how to make it really work for checkout pages / one that uses dynamic content or user interaction to determine what the UI does. Could have saved some time by addressing this at the beginning, as some things we needed to create on our site for Optimizely to "use" as a trigger for the variation test. - Having a number of planned/hoped-for tests already in-hand before working with Optimizely team. Sharing those thoughts with them would likely have started conversations on additional things we needed to do to make them work (rather than figuring that out during the actual builds). Since I had development time available, I could have added more things to the baseline installation since my developers were already "looking under the hood" of the site.

Overall, the implementation of VWO is straightforward. If you've got a straightforward way of deploying code to all of your test pages, either a good CMS or a TMS, then implementation should be a breeze. There is no tweaking to be done to the code itself, and once deployed it has the flexibility to cope with different VWO modules (tracking, conversion analysis, session analysis) without modification.

I think that Optimizely is more user-friendly than Adobe and Monetate, for example - Adobe is really suited for people with deep pockets and a distinct technical acumen that may not be suitable for a small business. Montetate is not user-friendly - even for seasoned marketers. Apptemize is a pretty great product but they killed their free product. Like Optimzely, they seem to be shooting for enterprise-level clients.

I used Google Optimize when it had just launched. It was therefore not yet a competitor to VWO. I haven't used it in roughly half a year time, so a lot has probably changed. I still use Hotjar for certain features that VWO offers, but which I think function better in Hotjar. I for instance prefer the Hotjar heatmaps, because I can elect to ignore certain elements on the webpage, such as a cookie consent pop up. A simple difference which makes me like Hotjar recordings more is that I can view a recording and click next, rather than going back to the overview and selecting the next recording.

It's all you need to design, install, analyze and measure simple tests with Optimizely. When test complexity increases, it becomes necessary to integrate third party analytics tools to perform more detailed analysis such as additional details like page views, average time spent on each page, and bounce rates by variation. Optimizely has very straightforward integrations, so integration with analytics is straightforward.

The product seems infinitely scalable for our needs (small business) and we've never had any issue with loading VWO-edited elements. I will say, though, that online customers with ad blockers have been known to not see certain VWO elements as their third-party scripts are disabled.

• Optimizely has had a tremendous positive impact on the ROI of our clients. Whether you're just starting your testing program and just want to get your feet wet with a few simple WYSIWYG tests, or you're ready to dive in to complex MVT test designs and Personalization Campaigns, Optimizely has everything you need to create, and maintain, a highly successful testing center.

• The ability to offer a CRO service is greatly enhanced by VWO, as it allows us to run tests for clients
• The editor means to some extent we don't need much developer resource to run the tests, lowering costs of doing so