- Multivariant testing async. We have a limited development staff, so having our marketers be able to control experiments was a big plus.
- Multiple dimensions for testing, allowing us to isolate tests based on device was a benefit.
- Screen recordings, being able to view customer recordings was valuable to see what people are doing on our pages.
- Heatmaps loaded very slow. We have very long form sales copy and VWO could not load the pages with a heatmap view a majority of the time.
- Google and Facebook slap. We were shutdown from running ads to pages that had VWO embedded in them as it was perceived as cloaking. This was the biggest issue for us as we rely on optimizing for display spend.
1) Monitor success of split and A/B tests, and
2) Draft new hypothesis to fuel next rounds of tests.
A/B & Split testing tools are being used by the Development Team to set up and execute tests on our Shopify stores.
- Test set up is a breeze and settings are presented in a thought out sequence.
- Recordings, Heatmaps, and other observation tools are easily segmented by audiences we care about, making observation simple for our UX team.
- Our customer success manager is always available for meetings and proactive in his advocation for the platform.
- Observations currently (9/14/18) don't support outside domains.
- Observation collection sample rate is a little low (for the price) - takes a bit longer to collect meaningful amount of data.
VWO Review: "A cost effective tool with strong potential for increasing incremental revenue and learning about your customers."
- The dashboard and ease of setting up tests with VWO is particularly easy and intuitive.
- VWO are constantly adding new features so we can analyse different parts of the customer journey and gain more insightful data on our customers.
- The VWO conversion all-in-one platform is excellent for gathering insights on how customers use the website and backing up theories. It has helped us prioritise tests and identify issues we hadn't thought of.
- We're on an Enterprise plan and there hasn't been much communication with our account manager. Suggestions for improvement and test ideas from the VWO team would be appreciated and would help to merit its cost and benefits over other competitor tools.
- Adding more personalisation features to the tool would be very beneficial.
- Observations and Hypothesis layouts and structure could be improved so this area of the platform is easier to manage. Checklists, outcomes and graphs would be advantageous.
VWO Review: "Our all-in-one insight and testing platform of choice - could not live, or work, without it."
With VWO's change to being CRO focused, we find the integration between observations and testing to put it ahead of the many competitors.
- Integrated features and benefits between insight based tools and variate testing
- Great UI for what is a pretty complex system
- Really good support team, quick to answer and always find a solution
- Auto report on activity, unless I'm not aware of this feature my reports are created manually
- Grouping tests and analysis into folders, I have a lot of tests and analysis running so scanning and finding the right item can be tricky
- Usage reports, a weekly or monthly report emailed from the platform on our usage stats
• Pages that need to increase their CR
• Newly launched pages and products where we can monitor activity and engagement
• Proving or disproving hypothesis with variate testing
• General problem pages/content on which observations and hypothesis can be based
• Validating design decisions
• Solving arguments or disputes on a particular pattern or feature.
• I'd be concerned if we ever had a situation where we could not use VWO to learn something and better meet the needs of our users and business.
- Goal Measurement. I enjoy the different objects you can target, and how easy they are to set up. Specifically, targeting elements and measuring their usage.
- Survey Data. The survey data is very interesting. I do wish you could make the parameters for showing a survey more complex, but it works very well as is.
- A/B Testing. I much preferred the UI layout of these screens over Google Optimize. It's easy to set markers utilizing goals and visualize the performance of variations.
- A/B Testing Goals. Sometimes when I conduct an A/B test, I want to measure a goal I'm already tracking. However; I can't see these goals from the Test set up screen, so I end up having to recreate them. You might consider a method for adding them similar to the way funnels are built.
- The documentation and integration with GA is under-stressed. GA is a staple for all web companies, so I might suggest working to integrate this better and bring it to the forefront. There are some measurements that are difficult to accomplish with VWO, such as conversion tracking or bounce rate.
- I ran into constant problems distinguishing between https:// and http:// . I actually ended up just using "contains" with the url, which was less than ideal. I might suggest a better way to clarify this -OR- a setting that will accept either.
- Images: Which pictures interact best with the end user over others
- Copy: A/B testing copy as an overlay on top of photos or below. Placement of copy is crucial to mobile first strategies and above the fold landscapes.
- Videos: A//B testing videos and video length. Trying to cap the engagement rate against time being watched. Also A/B testing where videos are best placed. Are they best used in the home page or product page or both.
- Visually, for people who are not familiar with the platform- can be overwhelming. Copy outruns the visual capacity and can cause fear of use.
- Training: a university type excercose is also recommended. Though VWO can be used for many functions- an in-depth course should be offered and/or required. Especially for how A/B tests run.
- All in one platform: The fact that Visual Website Optimizer offers all the features available in one platform facilitates the research work of marketers a lot.
- A/B Test feature: Within the A/B test feature, users have the possibility to analyze the results not just on a summary level, but on a detailed one, too. The best part always comes from the detailed report, which offers a clearer view of the performance of the test.
- Support team: Visual Website Optimizer has a great support team for their users. Fast, knowledgeable, friendly people that you can lay your trust in them that by the end of the day (if not quicker,) they will be there with a solution for you. The support team can assist you in any difficulty you might face at that specific moment with clear directions, both by email or a call.
- Surveys are a great way to get insights from your website visitors. The current version offers you the possibility to do so, but in a very basic way: question -> answer -> the end. My recommendation would be to upgrade this feature by adding branching logic to each answer.
- Customers of Visual Website Optimizer can allocate visitors to their sub-accounts by designating 'testing visitors' or 'research visitors'. Both of them are equally distributed among the different features. But my recommendation would be to have the option to allocate visitors from the 'research visitors', for example, by each individual feature. To know from the start that 100% of research visitors will be distributed depending your needs. e.g. 25% recordings + 25 heatmaps + 30% surveys + 5% forms + 15% Track features.
- It would be great if the Heatmap feature would have the option to save the created reports in a PDF form. This will allow any additional user of the VWO account to overview them on a later point.
- Automatically creates heatmaps of test variations, so no extra "syncing" work is needed to be done with Hotjar or other tools.
- Easy to use interface that let's you quickly create, develop, implement, and start A/B tests.
- Extremely cost-effective compared to Optimizely
- A/B test analysis is not great. Instead of giving you a specific number on lift, it gives you a range and calculates "likeliness to beat control". This is due to the fact they use Bayesian statistics and not Frequentist. Though I don't think they need to move away from B Statistics, having a specific numbered lift, like X% better/worse, would be extremely helpful in communicating the effect a test has to clients.
- No easy way to see how long a test has been running. Must look at the date range on a different view in analysis and then count the days. Very annoying since it could easily be solved by just listing the duration in days somewhere.
- Cannot compare conversion rates over time. Optimizely and most other AB testing platforms create a nice chart that shows how the conversion rates of different variations have changed in relation to each other overtime. VWO does not. Instead it gives you a line chart of the amount of conversions per variation over time. So naturally the line chart just keeps growing because conversions keep coming in. But comparing amount of conversions is very misleading since its conversion rate we care about.
- Very powerful heatmaps and scrollmaps.
- Great screen recording with segmentation capabilities.
- Point and click interfaces for heatmaps. No need to use manual selectors or code.
- Usability. There's so much to take in and what feels like a steep learning curve.
- Performance. The app feels really clunky and slow.
- An inexpensive product allows small teams with limited budgets support the business with A/B testing, which is considered by some as a nice to have.
- Very easy to deploy and use.
- Visitor analysis tools give us more options for qualitative analysis.
- I think some of the automated reporting could be more intuitive.
VWO Review: "Lots of potential, but the confusing UI, data anomalies, and confusing support interactions made us switch to another tool"
- Fairly inexpensive for the suite of conversion optimization tools
- UI is confusing and there is not enough help tips (or they are poorly written) to denote what each setting or metric actually does/is.
- Support is provided by obviously non-native English speakers and we had a hard time getting each support rep to understand what we were asking, which was typically a question about what something within the platform (setting/metric) actually means.
- Statistics engine underlying result data is not explained well and often doesn't line up with other conversion tracking applications (by a large margin).
- Form analysis feature, which is something that we really wanted, doesn't work natively with Marketo forms without significant custom development.
- VWO uses sampling on the overall goal tracking, despite us explicitly negotiating a visitor/session quota price point so that we could track everything on our website. This was one of the most frustrating parts of using VWO--we couldn't trust the accuracy of the data for our overall goal tracking due to this sampling issue because it was wildly different from our actual conversions as tracked in our application.
- Heatmaps automatically on all pages of the website
- A/B testing quick/small changes on a page and see the results in term of form filled or button clicks
- Split URLs for bigger A/B tests where we test some brand new designs
- Targeting specifics users (new visitors, mobile users) for some A/B tests
- The hypothesis part where we need to enter the hypotheses before doing A/B test seems to be useless in our case, as we do that more outside of VWO and don't really see the added value of that to be inside the software
- The sampling of recording/heatmap seems to sometimes be pretty low for automatic recording, and the fact that we need to explicitly ask for one page to be tracked with more sample is sometimes a bit irritating
- AB testing
- Removing Outliers from the data and provide actionable insights
- Provide lots of options as a filter to target a specific audience set
- Heatmap and Visual recording of all the test
- Provide a Mobile app or an improved version of Mobile web.
- Send alerts of any actionable results happening on my AB test
- Algorithm to sample and remove outlier needs a lot of improvement
VWO isn't perfect, but it is a great tool if your organization is looking to get its feet wet with Conversion Rate Optimization. They are developing great tools and features, as well as providing a library of testing ideas and case studies in case you're having trouble thinking of what to test.
Their results reporting isn't iron-clad, but for many efforts it's good enough. I personally am not a big fan of how "confident" they are in their declaring winners - I've seen tests where one day it says a variation is a winner at 100% confidence, then the next day that's down to 81% (think about that for a second).
However, for most new testers, it's a great tool and definitely a lot easier to use than a tool like Google Optimize.
The price point is very reasonable as well, making it accessible to companies of all sizes.
- Their UI for creating tests is brilliant. Does not require coding knowledge to set up basic to intermediate tests.
- VWO's support team is one of the primary reasons to stick with them. They are very fast and always provide an in-depth solution.
- The targeting options are good too, they give you the liberty to test a different set of audiences as well.
- There is a problem with creating responsive tests. The process involves creating two sets of test for mobile and web audiences. Which is time-consuming.
- Their reporting could be better. At times there is just too much data to see on the reporting page and the viewer gets confused.
- The tests are slow to load on slow networks. Which gives away the purpose at the start itself.
We use VWO as the primary A/B testing platform in our optimization efforts. This lets our product, optimization, and engineering teams test out anything from simple changes to large impact concepts with minimal resources.
- Easy to use for most tests. The learning curve is generally low so we have been able to ramp up use quickly.
- Multi-goal reporting. This is highly valuable and easy to setup.
- Limited real time targeting. VWO lacks in behavioral segmentation such setting up segmentation by 'people who add something to their cart'.
- Reporting at scale is a very manual process. The API allows access to data, but that requires more dev work to setup useful reporting at scale.
Enterprise level and highly mature companies that are at a point where they need real-time behavioral segmentation (user has an item in their cart) are out of luck with VWO.
Easy to use, great support staff. The platform has a slight learning curve but is a welcome and necessary tool for anyone who wants to expand their marketing operations to include A/B testing and site analytic functionality.
- Calculates test duration - VWO instantly calculates the duration your test will need to produce a significant result based on the level of certainty you require.
- Editor - The page editor is very useful for making simple edits to use for A/B tests. It makes the job of editing HTML very easy with no technical expertise needed.
- Reporting - real-time reports for test are kept very simple and are very easy to understand.
- Mandatory hypotheses - VWO recently made it mandatory to add a hypothesis to each test. I dislike the structure I am given to write each hypothesis so I simple fill it empty text.
- Amazing and fast support team
- Easy to use and very flexible solution for split testing
- Well structured reports
- Heatmaps looks really appealing even though we didn't use them
- An option to load and send split testing data via APIs on front end would be appreciated
Primarily, VWO is used by our product and marketing teams to test headlines, funnel changes, and changes in page design.
From a business perspective, we use it to test for conversion optimization. Essentially, we want to track how users convert, across multiple goals within our platform.
If you want to make bigger changes you have to rely on split URL testing, which also works good.
- You have the ability to have heatmaps recorded on any A/B or Split URL test, allowing you to investigate where users are concentrating most of their time within a control or variation.
- VWO's ability to allow custom conversion tags to be placed and triggered on the website allows for more robust conversion tracking.
- VWO's UI is very easy to use and navigate allowing users who are not familiar with coding the ability to test a website without the need of a developer in most cases.
- Creating more technical documentation in the library, not just high level information. Example would be documentation on how you can run a test on your internal network before having a test go live.
- Organization and management of multiple A/B tests paused or running at once is very well laid out and intuitive.
- Heatmaps have provided a huge amount of actionable data in focused testing areas.
- A/B test overview data is very easy to make inferences from because of its simple and visually outstanding presentation.
- The Chrome extension tends to have issues loading Heatmaps. It's not uncommon to go to a Heatmaps page and have to refresh the page at least once in order for the extension to load correctly.
- The visual editor for making variations for A/B tests feels clunky and often doesn't appear in the final version (previewing the variation as a page on its own) as it does in the editor. At times the undo button has caused erratic behavior such as duplicating or "redoing" several of the same action at once rather than stepping backward.
- Overall, if there were any way to decrease the amount of time that the Chrome extension takes to load, I could see workflow across several areas being greatly improved.
- Setting up tests is fairly easy, setting up URL's, saved goals, and experiment-wide JS
- Predictive text for CSS
- Good response time with general questions about the product, live chat is much appreciated
- There should be a preview button within the visual/code editor. Sometimes I need to see a page outside of the VWO window and it is a hassle to back out of the editor, click on the preview tab, open a preview, go back to the variation tab, and reopen the the editor.
- Sometimes hard to remember where certain settings are in the menu layout. Customize traffic allocation can be difficult to find, and the 'other' tab in settings is not the most intuitive name.
- Opening the test editor should be more prominent, since that is the main tool used for developers.
VWO Review: "Great Website Optimisation Tool to help inform product decisions and improve user onboarding"
- Relatively easy for people without a developer background to make tactical changes to website content that may influence customer behaviour while comparing effects on both desktop and mobile.
- VWO provides a good overview of results and data captured, allowing teams to collaborate and structure multiple tests simultaneously with endless segmentation options that are particularly useful when optimising certain user groups.
- The A/B testing feature is particularly useful with clear indications of when results are significant and easy interfaces to compare versions.
- I only use A/B testing because that's the only thing someone has taught me hands-on to do.
- I know there is more functionality available but it is not clear when you're logged in which features require an upgrade vs. which ones you can access.
- The Demo that was organised previously was not practical with poor connectivity and not tailored to the audience (ie. our CTO and Head of Customer support were attending but learned nothing useful other than the things I had already shared previously by going through the platform.)
A lot less suited for more advanced tests that require developer support/html/code editing. The Hypothesis tool is not very practical and could be a lot more user-friendly and should not be a premium tool...
VWO Scorecard Summary
VWO is an A/B testing and conversion optimization platform that enables growing businesses to conduct qualitative and quantitative visitor research, build an experimentation roadmap and run continuous experiments on their digital properties.
With its 5 capabilities Plan, Track, Test, Analyze, and Target, it brings the entire CRO (conversion rate optimization) process at one place. VWO helps online businesses follow the process- and data-driven conversion optimization to boost their conversion rates.
With features like heatmaps, surveys, visitor recordings, and others, VWO enables businesses to develop an in-depth understanding of on-site visitor behavior. VWO facilitates conducting research-driven A/B testing to optimize web experience across desktop, mobile, and other devices, to deliver a unique experience based on visitor preferences.
According to the vendor, VWO has helped more than 4,500 brands across the globe to run over 600,000 experiments to date. VWO’s global customer base includes brands like Vodafone, Samsung, Toyota, and eBay.
|Startup Plan (10,000 monthly visitors, unlimited tests)||$49||per month (billed annually)|
|Enterprise Plan (Custom)||$999||per month (billed annually)|
|Business Plan (30,000-300,000 monthly visitors, unlimited tests)||$129||per month (billed annually)|
For any additional information about pricing and plans, visit https://vwo.com/pricing/
VWO Support Options
|Free Version||Paid Version|
|Video Tutorials / Webinar|
VWO Technical Details