Item: UserTesting
Rating: 7
Author: Daniëlle Duijst

Use Cases and Deployment Scope

We use UserTesting to quickly test prototypes and iterate on them. We love it because of the quick results we can get to continue our work. It helps us save time on recruitment, scheduling, analysis, and the creation of reports.

Pros and Cons

Scheduling of participants in a fast way
Supporting analysis of videos by auto-transcriptions and sentiment analysis
Templates for the different tests and inspiration for questions
Supporting your own recruited users in myPanel
That you can test unlimited for the money you pay, this helps us run a lot of tests.

Slow with implementing feedback from the users. [I] have emailed many times about different issues that are quick fixes [and] they always promise, but [they] don't implement the changes. I think they should invest more in user research themselves.
The MyPanel environment is extremely limited and you have no control over the messaging that's sent to your panel members, nor the compensation of tests.
In general, I find the way of how the clips creation works not so intuitive, but I'm happy the functionality is there. I just wish they did more research with real users to find out how to improve it. I'm sure you'll notice after five users what's wrong with it.

Return on Investment

Faster time to market
Cost savings
Possible increased Customer Satisfaction (but it's hard to measure that UT only led to the increase)

Support Rating

I give a 7 because the support, in general, is quite fast and helpful. However, the actual implementation of fixes is incredibly slow. I have contacted support for many different problems I had with the tool, yet they don't fix it. It's frustrating because you see that they do give priority to developing new features. You should focus on a good foundation as well. I even made screen recordings to make it easier for the dev guys, but it seems like I wasted my time.

Usability

For general usage it's fine, but once you start using it more and more I noticed a lot of small annoying things. It's more usable than other tools on the market, but still, I really wish they would do more research to improve the usability.

For example, making clips & reels is quite confusing in the beginning.

Customer Insights

The insights from UserTesting feed our UX Team and help us make decisions. Usually, after UserTesting we try to validate the changes with A/B testing to make sure the change also works for our own users.

It also helped us using one tool instead of many and having everything in one place. The automatically created graphs in the report also help us to share results quickly (although it would be nice if you could just export them instead of screenshotting).

In general, we have increased our time-to-market because we find usability issues much faster in the process, and developers can start working on implementation much faster than before.

Alternatives Considered

UserZoom

We used to use Validately when it wasn't part of UserZoom yet. A year ago, it was a decent tool but much more buggy and less intuitive than UserTesting. Also, UserTesting has a much bigger panel with more and qualitatively better testers on it. Analysis is also easier on UT.

Key Insights

Do you think UserTesting delivers good value for the price?

Yes

Are you happy with UserTesting's feature set?

Yes

Did UserTesting live up to sales and marketing promises?

I wasn't involved with the selection/purchase process

Did implementation of UserTesting go as expected?

Yes

Would you buy UserTesting again?

Yes

Other Software Used

Microsoft Excel, Miro, Calendly

Likelihood to Recommend

UserTesting is great for quick usability feedback where you don't have specific requirements for your users. Usually, we get results within the same day so that's great. The quality of users in the panel is very different. Usually, we get some good ones that really explain why they think something is good or bad.

UserTesting is not so great if you want to manage your own panel and do tests with your own users. Companies should be careful when doing research that they not only depend on a tool like UserTesting, because people get paid to do your tests, they don't have any personal involvement with your product. This could lead to biased answers and could lead companies in the wrong way. We tried to create our own users' panel on UserTesting, but the functionality is so limited that we switched back to our old ways. We do additional testing and interviews with our own real users so that we make sure we're not getting biased results.

Usability Pros and Cons

Pros	Cons
Like to use Technical support not required	Unnecessarily complex Inconsistent Slow to learn Lots to learn

Easy Tasks

The shortcuts during video analysis
Sending the test out and getting results back quickly
Using the predefined questions for a test

Difficult Tasks

When you launch a test and you want to pause it - it's impossible it will delete your whole test and you have to start over
Trying to use the MyPanel in general, no control over anything, the branding logo is tiny because of some standard pixel width and our users don't trust signing up with it cause it looks like a scam
Creation of clips, how to find them, how to make them in a nice compilation. Especially the first time it was extremely unclear and confusing.
When you login and you land on the last screen you were on, but without the logo on top left so you're stuck in that screen without any way to get back to the homepage of the dashboard without changing the URL...
When you create a test and launch it, you get sent back to test overview and you find that the oldest test is showing first instead of the test you just launched, very confusing.
There are some screening questions formulated very unclear, like the one about frequency of test (last time they tested or something) it's unclear what you're actually screening/filtering on.
Do user research on your own tool, ask users to create a clip reel to share with stakeholders, where would they start, what problems do they run into, is it really that intuitive as you think it is?

UserTesting has helped us speed up our process (but there is still a lot to improve)

Overall Satisfaction with UserTesting