Do five-star ratings systems have a ‘positivity’ problem?
Source: Amazon.com

A university study finds that the five-star online review system has become increasingly less useful because most products are now rated positively.

On Amazon.com, for example, the average star rating is 4.2 out of five, with well over half of the reviews being five-star ratings. Nearly half of all Yelp reviews are given a five-star rating, according to the study, “Mass-Scale Emotionality Reveals Human Behaviour and Marketplace Success,” published in the journal Nature Human Behaviour.

“This research demonstrates a ‘positivity problem’: Most reviews are positive. In this ‘sea of positivity,’ how do people distinguish products,” Derek D. Rucker, the Sandy & Morton Goldman professor of entrepreneurial studies in marketing at Kellogg and co-author of the study, said in a statement.

Instead, the “emotionality” of reviews was found to provide a better read into the value of a business. As part of the study, the researchers used computational linguistics to assess the language used in reviews of books, restaurants and movies. For instance, a positive and highly emotional review for a restaurant might read: “I love this restaurant, it is absolutely wonderful.” In contrast, a positive, but less emotional review might read: “This restaurant is excellent, dinner was flawless.”

Although the average star rating predicted more table reservations, emotional reads were better at predicting a restaurant’s future success. The researchers suggested an “emotional star rating” could be used as an alternative rating.

Overly-positive online reviews have been called out before as a flaw in the five-star rating system. Part of the problem is that only customers who have had particularly good or bad experiences tend to write reviews. Fake or manipulated reviews also distort the star system.

One solution, according to an article from the Harvard Business Review from 2019, is showing users the average score for all businesses in the relevant category (e.g., Amazon, toys). Reviewers who tend to not always give the same or similar score could also be given more weight in ratings.

BrightLocal’s “Local Consumer Review Survey 2020” found the star rating to be the most important factor consumers cared about when choosing a local business, cited by 84 percent. However, close behind were legitimacy, 81 percent; recency, 80 percent; sentiment, 79 percent; and quantity, 79 percent.

BrainTrust

“Five-star ratings systems have become meaningless for most consumers. What I consider a five-star result might only be worth three to you.”

Jeff Weidauer

President, SSR Retail LLC


Discussion Questions

DISCUSSION QUESTIONS: Can five-star rating systems be reimagined to become more granular and beneficial to consumers? Can other factors (i.e., legitimacy, recency, sentiment, quantity, length/detail) be incorporated into ratings? How do you use online reviews?

Poll

How would you rate the five-star rating system as a tool for retailers to deliver personalized offers for consumers?

View Results

Loading ... Loading …

Leave a Reply

14 responses to “Do five-star ratings systems have a ‘positivity’ problem?”

  1. Gene Detroyer Avatar
    Gene Detroyer

    This is so true. Do I stay away from anything that has a rating of 3.9 or less? I do.

    I believe the problem is one of definition. Which number represents “as expected.” In NYC the taxi app asks you to rate the driver on the five-point scale. The taxi is clean. The driver drives safely and appropriately. You get to you destination on time. Is that a five? That is what I give the guy. Similarly, you order something. It comes as expected, You open it. It works. To me on these scales that is a five. How could I give the driver anything less?

    Say you have been to a restaurant several times. You are very pleased. That is why you keep coming back. You have rated them five each time. Now you go and you are welcomed by name, the owner comes and greets you, the meal usually always good is great this night. Where is the room to give a higher grade?

    Thinking about ratings, I really only have two, the four/five range or the one, if it hasn’t been “as expected.” Maybe three should be the “as expected?”

  2. Mel Kleiman Avatar
    Mel Kleiman

    If you want a rating system of a product that gives real and valuable information, I would suggest a two-part rating system. The first review is written after the initial purchase but, more importantly, the second part is to ask the customer to rate the product or service again at 30 or 60 days in the future.

  3. Jennifer Bartashus Avatar
    Jennifer Bartashus

    There is no doubt rating systems skew to the positive and probably need to be improved given how heavily they influence behavior. Netflix used to use a star system, now it is just a thumbs up or thumbs down – for them, limiting choice worked. For retail, maybe adding in a metric like product return rate might be helpful and is less reliant on a customer taking the step to write an accompanying review. Alternatively, with natural language processing, maybe consumers write a review first and a suggested corresponding star rating populates afterwards that then has to be approved or adjusted.

  4. Jeff Weidauer Avatar
    Jeff Weidauer

    Five-star ratings systems have become meaningless for most consumers. What I consider a five-star result might only be worth three to you. The devil, as always, is in the details. When looking at reviews, I will read through some five-star, but then I read a few one- or two-star to get context and perspective. Providing a useful review – whether good or bad – is more than clicking a box.

  5. Steve Montgomery Avatar
    Steve Montgomery

    I have more faith in the overall ratings if they are negative than if they are positive. I look at the ratings system as similar to what people say when the restaurant manager comes to the table and asks, how is your meal? “Good” is likely the response unless it is really not good because few people are willing to say it is just OK. I agree with Gene that if it has not received at least a 3.9 (fair) rating then it is probably not something I would buy.

  6. Patricia Vekich Waldron Avatar
    Patricia Vekich Waldron

    Comments are the best, but who wants to wade through them for every choice one makes? How about a thumbs up, thumbs down, meh and a star (for outstanding). That’s more meaningful than the current numbering system.

  7. Bob Phibbs Avatar
    Bob Phibbs

    One of the problems with reviews is that most retailers ask for a review when the item is purchased. If there were to be two things I would like to see they would be what Amazon shows as a “Verified Purchase” and a followup of actual usage one year later. It sounds good but how would that happen exactly? Especially when many products are not like a mattress where you really want a long-term use, not a USB plug.

  8. Mary Pietsch Avatar
    Mary Pietsch

    Five-star reviews have become much less reliable than in the past. One issue is that five has become the expected rating for everything being up to expected levels. I’ve received countless solicitations from businesses requesting that I leave them a five-star rating if I am satisfied. A satisfied customer due to everything being at expected levels is a three, not a five. But if there is an average rating of three, most consumers would pass. The second issue is the number of planted reviews. Often on Yelp or Amazon, I see five stars and in the commentary notice a large number of reviews written just within days or even hours of each other, each giving five stars and the commentary basically says it is the best product, restaurant, etc. to ever exist. Obviously not real.

  9. Cynthia Holcomb Avatar
    Cynthia Holcomb

    Words are subjective to both the speaker and the receiver of the words. Layer on top of that the subjectivity of an individual’s interactive experience with a product in matching their unique expectation of the product, based upon the complete subjectivity of how the individual sees, lives, and experiences their own individual world. Each of us has unique worldviews based upon childhood, family, emotion, personal sensitivities, life experiences, and unmet expectations, just to name a few items in the hodgepodge of human decision-making leading to a product review.

    There is much scientific and technological exploration being done to break the code of human intent using sentiment analysis. The positivity or negativity or in between of a product review is a snapshot in time of an individual human’s emotional subjectivity at this one particular point in time on a given day. Solutions to tap into the mindset of individual product reviewers to calculate what is the “truth” behind the review is impossible, as there is no “truth” only the subjectivity of the words to voice an opinion on a particular day in time.

  10. Venky Ramesh Avatar
    Venky Ramesh

    Rating is a translation of emotional response to a linear scale – which in most cases is a lot of work for the customer. Instead, the rating itself should be presented as emoticons that represent typical customer emotional responses. Also, it is almost impossible to go through and make sense of thousands of review comments – it would make sense for the retailer to present a summary in the form of a word cloud.

  11. Jlauderbach Avatar
    Jlauderbach

    At best, the 5-star rating an index of purchaser sentiment. The index is skewed by the sample of purchasers it represents. As the article states most “raters” are happy purchasers or unhappy purchasers. The satisfied purchaser does not generally take the time to rate. What percentage of purchasers rated the product? Not mentioned was what impact did the marketing material play in setting the purchasers’ expectation?

    Is the rating based only of online purchasers or both online and physical store purchasers? Is there a difference in rating between the two?

    This type of rating system does not lend itself to granularity very well since the raters are unknown and requirements not defined nor weighted in overall score.

    Formal reviews of products tend to have more credence, but they take time to read. In this age of misinformation even reviews must be taken with some skepticism.

    As flawed as it is, I am sure the 5-star rating system plays a role in purchase decisions. If it were removed from products there probably would be an outcry from potential purchasers.

  12. Rick Moss Avatar
    Rick Moss

    Every system has its flaws, but it sure seems like some built-in corrections and possibly real human intervention is needed. On the movie review site, Rotten Tomatoes, which goes by critics reviews, Citizen Kane has been overtaken by Paddington 2 (yes, as in the bear) as a top-rated film. I rest my case.

  13. KirstenM Avatar
    KirstenM

    There is much fakery going on with ratings. They lack trust and transparency. Many are paid, or done by “friends” of the retailer. As well, consumers have been conditioned to judge everything, even if they are not qualified. Users are looking for the proverbial word-of-mouth recommendation. HomeStars.com is one platform for home services which requires users to register (like I just did for RetailWire). That vendors on HomeStars market their ratings success, says something about how they do it.

  14. James Tenser Avatar
    James Tenser

    Five point ratings need to be reimagined so that 1 means “Utter fail”; 2 means “disappointing”; 3 means “met all expectations”; 4 means “better than most”; and 5 means “above and beyond.”

    Present star scores are barely diagnostic at all. Even the brands who ask for these ratings learn virtually nothing actionable from the responses.

14 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Gene Detroyer
Gene Detroyer
Trusted Member
2 years ago

This is so true. Do I stay away from anything that has a rating of 3.9 or less? I do.

I believe the problem is one of definition. Which number represents “as expected.” In NYC the taxi app asks you to rate the driver on the five-point scale. The taxi is clean. The driver drives safely and appropriately. You get to you destination on time. Is that a five? That is what I give the guy. Similarly, you order something. It comes as expected, You open it. It works. To me on these scales that is a five. How could I give the driver anything less?

Say you have been to a restaurant several times. You are very pleased. That is why you keep coming back. You have rated them five each time. Now you go and you are welcomed by name, the owner comes and greets you, the meal usually always good is great this night. Where is the room to give a higher grade?

Thinking about ratings, I really only have two, the four/five range or the one, if it hasn’t been “as expected.” Maybe three should be the “as expected?”

1
Mel Kleiman
Mel Kleiman
Member
2 years ago

If you want a rating system of a product that gives real and valuable information, I would suggest a two-part rating system. The first review is written after the initial purchase but, more importantly, the second part is to ask the customer to rate the product or service again at 30 or 60 days in the future.

2
Jennifer Bartashus
Jennifer Bartashus
2 years ago

There is no doubt rating systems skew to the positive and probably need to be improved given how heavily they influence behavior. Netflix used to use a star system, now it is just a thumbs up or thumbs down – for them, limiting choice worked. For retail, maybe adding in a metric like product return rate might be helpful and is less reliant on a customer taking the step to write an accompanying review. Alternatively, with natural language processing, maybe consumers write a review first and a suggested corresponding star rating populates afterwards that then has to be approved or adjusted.

1
Jeff Weidauer
Jeff Weidauer
Member
2 years ago

Five-star ratings systems have become meaningless for most consumers. What I consider a five-star result might only be worth three to you. The devil, as always, is in the details. When looking at reviews, I will read through some five-star, but then I read a few one- or two-star to get context and perspective. Providing a useful review – whether good or bad – is more than clicking a box.

0
Steve Montgomery
Steve Montgomery
Member
2 years ago

I have more faith in the overall ratings if they are negative than if they are positive. I look at the ratings system as similar to what people say when the restaurant manager comes to the table and asks, how is your meal? “Good” is likely the response unless it is really not good because few people are willing to say it is just OK. I agree with Gene that if it has not received at least a 3.9 (fair) rating then it is probably not something I would buy.

2
Patricia Vekich Waldron
Patricia Vekich Waldron
Active Member
2 years ago

Comments are the best, but who wants to wade through them for every choice one makes? How about a thumbs up, thumbs down, meh and a star (for outstanding). That’s more meaningful than the current numbering system.

1
Bob Phibbs
Bob Phibbs
Active Member
2 years ago

One of the problems with reviews is that most retailers ask for a review when the item is purchased. If there were to be two things I would like to see they would be what Amazon shows as a “Verified Purchase” and a followup of actual usage one year later. It sounds good but how would that happen exactly? Especially when many products are not like a mattress where you really want a long-term use, not a USB plug.

1
Mary Pietsch
Mary Pietsch
2 years ago

Five-star reviews have become much less reliable than in the past. One issue is that five has become the expected rating for everything being up to expected levels. I’ve received countless solicitations from businesses requesting that I leave them a five-star rating if I am satisfied. A satisfied customer due to everything being at expected levels is a three, not a five. But if there is an average rating of three, most consumers would pass. The second issue is the number of planted reviews. Often on Yelp or Amazon, I see five stars and in the commentary notice a large number of reviews written just within days or even hours of each other, each giving five stars and the commentary basically says it is the best product, restaurant, etc. to ever exist. Obviously not real.

0
Cynthia Holcomb
Cynthia Holcomb
Member
2 years ago

Words are subjective to both the speaker and the receiver of the words. Layer on top of that the subjectivity of an individual’s interactive experience with a product in matching their unique expectation of the product, based upon the complete subjectivity of how the individual sees, lives, and experiences their own individual world. Each of us has unique worldviews based upon childhood, family, emotion, personal sensitivities, life experiences, and unmet expectations, just to name a few items in the hodgepodge of human decision-making leading to a product review.

There is much scientific and technological exploration being done to break the code of human intent using sentiment analysis. The positivity or negativity or in between of a product review is a snapshot in time of an individual human’s emotional subjectivity at this one particular point in time on a given day. Solutions to tap into the mindset of individual product reviewers to calculate what is the “truth” behind the review is impossible, as there is no “truth” only the subjectivity of the words to voice an opinion on a particular day in time.

0
Venky Ramesh
Venky Ramesh
2 years ago

Rating is a translation of emotional response to a linear scale – which in most cases is a lot of work for the customer. Instead, the rating itself should be presented as emoticons that represent typical customer emotional responses. Also, it is almost impossible to go through and make sense of thousands of review comments – it would make sense for the retailer to present a summary in the form of a word cloud.

0
Jlauderbach
Jlauderbach
2 years ago

At best, the 5-star rating an index of purchaser sentiment. The index is skewed by the sample of purchasers it represents. As the article states most “raters” are happy purchasers or unhappy purchasers. The satisfied purchaser does not generally take the time to rate. What percentage of purchasers rated the product? Not mentioned was what impact did the marketing material play in setting the purchasers’ expectation?

Is the rating based only of online purchasers or both online and physical store purchasers? Is there a difference in rating between the two?

This type of rating system does not lend itself to granularity very well since the raters are unknown and requirements not defined nor weighted in overall score.

Formal reviews of products tend to have more credence, but they take time to read. In this age of misinformation even reviews must be taken with some skepticism.

As flawed as it is, I am sure the 5-star rating system plays a role in purchase decisions. If it were removed from products there probably would be an outcry from potential purchasers.

0
Rick Moss
Rick Moss
2 years ago

Every system has its flaws, but it sure seems like some built-in corrections and possibly real human intervention is needed. On the movie review site, Rotten Tomatoes, which goes by critics reviews, Citizen Kane has been overtaken by Paddington 2 (yes, as in the bear) as a top-rated film. I rest my case.

1
KirstenM
KirstenM
2 years ago

There is much fakery going on with ratings. They lack trust and transparency. Many are paid, or done by “friends” of the retailer. As well, consumers have been conditioned to judge everything, even if they are not qualified. Users are looking for the proverbial word-of-mouth recommendation. HomeStars.com is one platform for home services which requires users to register (like I just did for RetailWire). That vendors on HomeStars market their ratings success, says something about how they do it.

0
James Tenser
James Tenser
Member
2 years ago

Five point ratings need to be reimagined so that 1 means “Utter fail”; 2 means “disappointing”; 3 means “met all expectations”; 4 means “better than most”; and 5 means “above and beyond.”

Present star scores are barely diagnostic at all. Even the brands who ask for these ratings learn virtually nothing actionable from the responses.

0