Source to Resource: Balancing Internal and External Testing
April 24, 2017
Source to Resource: Balancing Internal and External Testing
Score 7 out of 10
Overall Satisfaction with Qubit
Qubit is addressing several significant needs for multiple departments (Customer Service, IT, Marketing, User Experience) within our company:
- User Experience Testing
- Customer Journey Reporting
- Dynamic Customer Segmentation
- A/B Testing
- Mobile Experience Testing
- Data Integration
- Tara and Trip have been excellent contacts and resources for questions and answers. Their Service and support and extremely helpful.
- The ease of test implementation is very handy. Requirements are discussed, the project is scoped and development starts. Updates are provided as steps are accomplished and tests are visible throughout the entire test period.
- The provided "opportunities" section helps define what next steps may be taken from a project management standpoint. This is a two edged sword. 1) It provides some light directionality communication on what next mid or low tier priorities exist. 2) This also proves the investment of the tool by adding a third voice to the conversation and consistently communicating ROI in potential dollars to be gleaned from the work that could be done.
- I have yet to be "impressed" by any of the design elements supplied by the design team. I know that they are working from a billable clock perspective, but many of the designs require two or three rounds of refinement before I have been happy with their finalized work. Even then, the design itself seems somewhat vapid. It may be that they have not been empowered to challenge formats or pages more specifically, but more often than not, I wonder what their background is, as many adjustments seem like design 101 from an eCommerce perspective.
- The timeliness of design turnaround is the least expedient of the group. Tara's ability to launch a project is excellent, her developers and QA team seem to have legitimate 48-72 business hour turnaround times, but the design feels like it takes much longer. This is then compounded by the aforementioned required set of approvals during which additional adjustments are made. Again, I do not know what requirements are provided, but it is the only part of the relationship that I would deem a "pain point."
- I would like to see a more Robust Customer Feedback tool or the addition of a automated report. There are plenty of false negatives or false positives in the reporting and so the breakdown within the tool doesn't help a great deal. Segmenting by type is more helpful, but the positive negative is often not correct.
The most significant change in the past year has been a slight redesign of our PDP's based on several A/B Tests we ran over 6 months leading up to the change. By making small adjustments to the convertible portions of our product pages, we were able to capture additional customers and provide incremental conversation rate increases across our assortments. This was the largest and most significant UX change we have made based on our experience focused projects.
- None that I have worked with
I was not directly involved in the decision-making process but understand that overall functionality, data integration, and UX testing weighed against the cost of the solution were the deciding factors.
- This has been our largest challenge based on our current IT infrastructure. We are working toward a more robust Data Integration with Google Big Query, but our own internal resource constraints have slowed the process.
- The segmentation data included in our A/B Testing has been key to better refining our personalization for our customer base both by acquisition and channel specifics.
Qubit has allowed us to provide multiple marketing messages throughout our customer journeys addressing conversion at every stage in the customer journey. This, in turn, has provided ample information to make marketing decisions to address problems before they occur, consequently allowing us to close conversion gaps in two areas rather than one.
Qubit works well in certain situations, but any internal backlog in IT, specifically Data Integration or Development, will run into consistent issues between published tests, the ability to hard code winning tests and the actual use of the data engine that Qubit provides. Having a fully Functional IT team is necessary in order to make the best use of the tool. This is somewhat at odds with some of the work we are requesting them to do for us, based on the fact that our IT backlog is significant. In short, we are using Qubit to provide additional IT bandwidth, but Data integration requires a decent amount of IT bandwidth.
4 - Qubit is used by four distinct departments:
- User Experience
2 - Our Qubit support is primarily an IT consideration. Although I maintain the project Status and am the primary relational contact point, the Support and Primary users reside with IT. Our developers are few but top notch. They work in conjunction with our Data Warehouse Architects to streamline the flow of information between our platform and the Qubit tool.
- Speed to Marketing Experience Changes with Conversion Uplift
- Marketing Channel Segmentation and Personalization
- Platform Data feeds into our CRM
- Qubit provides a second set of eyes for areas requiring attention in lower priority assortments or portions of our site. This provides some "low hanging fruit" projects that will drive incremental revenue for the company.
- As of right now, we are not looking into any additional uses as there is ample work in the opportunities presented to us currently.
- Incremental Conversion Rate lifts site wide
- UX Code Development and QA Projects
- Additional Customer Feedback Loops and Abandonment Surveys
Qubit is currently providing resources and support we do not have internally. Our relationship managers are exceptional and I feel well informed and well supported by their team. The tool is nice, but our contacts from the company are the real reason to maintain our relationship. They work hard for clarity and continue to help us push for additional opportunities.