Item: VWO
Rating: 8
Author: Tom Kerwin

Use Cases and Deployment Scope

We're using VWO primarily for Conversion Rate Optimisation work at the moment. We are also planning to split test in-product user experience features, but we're not sure it's going to do the job for that. I'm the main person who uses VWO, although test ideas are generated across the company and testing elements are designed and developed by whole the front end team. Before I arrived at the company, they were attempting to split test using a rudimentary in-house platform. VWO helps us to test ideas significantly faster and gives more control and visibility of the data.

Pros and Cons

It's great for getting tests set up quickly. As soon as you've added the one piece of JS across your site, you can start testing simple ideas in minutes.
It displays actionable results attractively and in a way that's accessible to non-statisticians.
It gives you good control over segmentation. If you need to, you can use custom JS, cookies and other technical trickery to get very fine-grained control over segmentation. At the moment we're running several tests on a landing page, each targeted at a different segment.
You can track multiple goals, including page views, clicks on page elements, form submissions and even custom JS triggers.

It's not always easy to set up a test. We had significant teething trouble due to certain idiosyncrasies of our website. One split test affected pages on the site that it wasn't supposed to, another caused huge lags in display times; more recently we had a test fail to capture visitors correctly. Over time, we've worked out how to address each problem, but it's certainly not as simple as it appears to be on first glance. I used to use VWO across a wide range of clients and almost all suffered from teething trouble of one kind or another.
Goal tracking is not always reliable. Some types of goals seem to be captured more consistently than others. This isn't a deal-breaker: you just have to make sure that your main testing goals are being tracked correctly.
We have several different views with the same URL, depending on viewer state. I'd like to be able to turn on tracking across all the views without all the views serving a test variation. We've managed to work around this, but I wish it had been simpler.
We've also had to develop our own workaround for capturing the versions of pages our visitors have seen in our own database. It's crucial for us to be able to track our visitors across different variations of offline media as well as the online media and store the details for later cohort analysis. I'd wish for VWO to provide an API of some sort for this purpose.

Return on Investment

The big one is that within 1 month of starting testing, we'd improved conversion by over 30%. And within 2 months we'd got another 30% win. We're on track to double our (already quite high) conversion rate within 3-4 months of me starting.
We have a much greater understanding of our customers and each test has given us insights that have driven improvements elsewhere in our product.
Our testing cycle length has shortened dramatically, so the timeframe from idea to test is now about 25% what it was. (not taking into account false starts from teething troubles.)

Alternatives Considered

google website optimiser,omniture test & target

It's much easier to get right than GWO. It's much easier to understand than Omniture.

That means it's much quicker to get tests running than either of the others. Both of which drove me spare with the level of complexity needed to try relatively simple things.

Likelihood to Renew

It's great value and we think we've ironed out all the major teething troubles. However, if we experience any more bugs or problems that significantly slow us down then we're seriously considering switching to Optimizely, which I haven't personally tested but have heard great things about from my CRO peers.

Likelihood to Recommend

I'd say it's ideal for a company with a relatively simple site who's just getting into split testing. There's a lot to successful split testing that VWO can't help you with, but as a way to get started quickly, it's hard to beat.

You're ready for VWO if:

you understand the fundamental principles of hypothesis-driven testing and you're looking to understand, empathise with and learn more about your visitors and customers
you have a clear idea of the business goals you're going to address through split testing and the metrics you're going to optimise for
you've developed a testing plan across a series of tests and you're prepared for the reality that some tests won't get to significance – or might surprise you in other disappointing ways...
you're new to split testing in practice – or you've tried one of the free testing platforms and found it too confusing, buggy or difficult
you're keen to split test and you want to be able to set up relatively simple tests quickly and with a minimum of technical involvement
but you're prepared for some headaches and tinkering to get it working just right
your budget isn't high enough to afford a more fully-featured or 'done-for-you' split testing solution.

Products Replaced

Yes - An internal testing platform. We wanted to replace it because:

it required back-end developer time to set up any test – that was a major bottleneck
it required database operations and fiddly Excel work to extract the results
it limited the ways we could set up tests

Key Differentiators

Price
Product Features
Product Usability
Prior Experience with the Product
Existing Relationship with the Vendor

The biggest factor for me was familiarity. I've been using VWO for several years across several clients. Although I'd suffered a few teething problems, they were all handled by VWO's rapid-response support team. Joining a new organisation, I wanted to make an impact fast, and so I wanted to use a tool I could hit the ground running with. It was also important that it was relatively affordable so I wasn't demanding a huge budget from day one.

Evaluation Lessons Learned

The only thing I might have done is ask my CRO contacts what they were using now, in case something better had come along recently. It sounds like Optimizely might be that something better, but without having asked early on, we've now invested time and energy into VWO instead.

VWO: Very Worthy Option if you want to move fast, tweak as you go and do it yourself.

Software Version

Overall Satisfaction with Visual Website Optimizer