Qualtrics: Bang for Your Buck
February 25, 2014
Qualtrics: Bang for Your Buck
Score 8 out of 10
- Research Suite
Qualtrics is used by a handful of people in our The New York Times Consumer Insight Group. We typically use it for:
- Web intercept studies related to ad sales, consumer marketing and strategic planning
- Qualitative research recruitment for our user experience team
- Customer support - I like being able to talk to someone on the phone who can help me with technical issues related to complex programming issues. Although the support line is sometimes busy, the majority of the time I am able to talk to someone live who can fix my issue in a timely manner.
- Intuitive programming - Coming from a basic background of survey programming, Qualtrics is the best I've used to date (compared to Survey Builder, Instant.ly, Key Survey and of course Survey Monkey). I like being able to seamlessly copy and paste text into Qualtrics very easily, use display logic appropriately and capture embedded data when needed
- Top line reporting - I've had success working directly in the Results feature as well as exporting to Excel. Running filters is quite easy before exporting to Excel so that you can analyze various subgroups. I've even pulled in a vendor to run traditional tables using Qualtrics' SPSS file.
- Cross Tabs - I like that Qualtrics offers the feature, but one of my biggest complaints is that it takes quite a long time to load responses when you need to merge columns or stubs together. Also, you have to merge two columns/stubs one at a time instead of doing all of your merging and running it at the very end. It is very time consumer and inefficient when doing simple things like age breaks, top 2 box, etc.
- Email response times - While phone support is strong, I recall email support taking more than one business day to get a response for some simple/standard questions. I would like to see Qualtrics invest in online chatting windows similar to retailers to talk through issues.
- Randomization - Currently you cannot nest response options together and randomize them.
- Cost per complete - Sometimes we have niche targets that are more difficult to reach. My team has had to purchase additional batches of survey completions. Given that the sample is solely ours and not from a gen pop panel, I think Qualtrics' costs could be a little more client-friendly when it comes to those who terminate vs. actual qualified respondents.
- Ability to recruit highly qualified individuals for qualitative research
- Conducting research for ad sales projects that require very short turnaround times
- Tracking attitudinal behavior over time
I didn't choose Qualtrics per se - the contract was already in place when I started at NYT. But the level of sophistication and customer support is far better than other competitive products.
The simple fact is that I have a lot of historical data and questionnaires already tied into Qualtrics. Having the same product in place makes it easy to look back at old studies for learning purposes as well as design purposes for consistency.
It is well suited when you are under budget restraints and cannot afford a full service vendor. Studies where you are simply looking at total and a handful of easy to filter sub groups such as age/gender/HHI. It is far less appropriate when you need more advanced statistical needs such as segmentation, max diff, etc.
- Being able to save money by programming ourselves but having a vendor do cross tabbing by exporting the data file to SPSS
Evaluation and Selection
One point docked since waiting times occur occasionally and another point docked for the long time it takes for an email response.
Problems get solved
No escalation required
Support understands my problem
Support cares about my success
Quick Initial Response
Yes - Some of our surveys are very complex. Using Survey Flow for pathing/display can be tricky, but I've learned a great deal by working with the support team.
Qualtrics is fantastic when it comes to including java script coding to make my surveys more presentable. I love it when they simply log in to my account and make a change on the spot. They are also really good for logic/piping/display complexities. Their support team is very patient.
I have trained one colleague who had no background in survey programming. She has picked it up quite nicely and can program things fairly easily without much assistance.
Like to use
Easy to use
Technical support not required
Quick to learn
Feel confident using
- Survey look and feel options/design
- Creating basic questions, but having the option to use grids, sliding scales, etc to change it up
- Top line reporting