Solid Testing Platform
September 17, 2018

Solid Testing Platform

Hovhannes Mkhitaryan | TrustRadius Reviewer
Score 8 out of 10
Vetted Review
Verified User
Review Source

Overall Satisfaction with VWO

The basic observation tools in VWO are being used by our User Experience Team to
1) Monitor success of split and A/B tests, and
2) Draft new hypothesis to fuel next rounds of tests.

A/B & Split testing tools are being used by the Development Team to set up and execute tests on our Shopify stores.
  • Test set up is a breeze and settings are presented in a thought out sequence.
  • Recordings, Heatmaps, and other observation tools are easily segmented by audiences we care about, making observation simple for our UX team.
  • Our customer success manager is always available for meetings and proactive in his advocation for the platform.
  • Observations currently (9/14/18) don't support outside domains.
  • Observation collection sample rate is a little low (for the price) - takes a bit longer to collect meaningful amount of data.
  • We've been able to roll out new design patterns with confidence that short term revenue is not negatively effected.
  • We've had more wins than losses, as the company is just ramping up a testing strategy. We know percentage of wins will go down as we optimize, but VWO has been vital on making calculated steps along the way.
Heatmaps are probably the weakest observation tool from the rest of features in VWO - we've been able to identify extremes of where clicks happen, where they don't, and where clicks happen where no link exists. That is the extent of it.

Scrollmaps have been more important, as they show "heat" of percentage of visitors that reach a certain length down a page. Our site has lots of content that we prioritize on the page but ultimately want to have impressions for all sections. VWO has allowed us to test for "scrollability" of designs, ensuring that visitors consume the content we put on our sites.
With the interactive player/recorder from VWO, we've been able to observe shopping behavior as it happens - with lots of bouncing around the site and content consumption. We've always assumed it's a straight funnel - recordings have (in small samples) proven otherwise. One challenge with recordings is that it is anecdotal and part of a small sample, so we need to pair the tool with other methods of analysis to get a full picture.
For sites with backend languages (i.e. PHP) it is simple to do A/B and split testing as we can rely on parameters set, which can be analyzed by PHP before page loads.

For sites like Shopify that don't truly have a server side language, we have to rely on split testing with different URL or by using Javascript to analyze parameters and change content after.