It may seem odd for the CEO of a review site to take a stance against star ratings, but I believe they receive far too much emphasis. Don’t get me wrong. They have their place. In an era of content scanning and low attention spans, star ratings give a quick visual representation of sentiment. They do, however, need to be considered in context. 

For example, we find that simpler, less expensive products tend to average higher scores than more complex, expensive products on TrustRadius. This doesn’t necessarily mean they’re better. If your needs are more complex, you’re probably going to be better served by a more robust product, even if it receives a lower rating. I’m not 100% certain why we see this phenomenon but my guess is that users of more expensive products have higher expectations and are hence harsher critics.

More importantly, we find that there’s far more information value in the text of what people say to accompany their ratings than the ratings themselves. The average rating on TrustRadius is 3.96/5.00 and more than 75% of reviews score products a 4 or more out of 5. As a benchmark, per a March 2012 article in New York magazine, the average rating on Yelp is 3.8, and in 2009 the Wall Street Journal reported that Internet wide, the average consumer rating was 4.3.

Ratings are most useful when they represent extremes, helping you eliminate products or include them on your long-list. However you should never select one software product over based upon small rating differences. It’s critical to read the text that accompanies reviews. We’ve found that even raving fans are able to express where a product or service can be improved and what use cases it best serves and those insights can be very telling. For a terrific example check out this IBM Cognos review.

That’s why at TrustRadius, we’ve placed a huge emphasis eliciting in-depth reviews. Today, our average review is ~500 words and our longest over 2,500. According to Bazaarvoice, a software company that powers review forums for over 2,000 retailers/brands, the average consumer product review is under 100 words. Other business software review sites like the AppExchange tend to more closely mirror the consumer review approach of focusing on ratings with short comments.

If you’re making a critical product decision, the depth and quality of reviews is key.