According to a new study, reviews on Amazon.com are at least slightly skewed, in large part because positive reviews help reviewers achieve higher rankings.
Trevor Pinch, a professor at Cornell University, surveyed 166 of Amazon's top 1,000 reviewers to probe whether Amazon's wealth of outside insights was as much of a "free lunch" as it seemed.
Asked why they contribute reviews, respondents overwhelmingly mentioned "self-expression" and "enjoyment." Altruistic reasons -- "Hope to help others decide whether to buy," "Wanting to share what I have liked with others" -- were also given.
Some wrote reviews to keep track of new books or movies. (Books remain the highest reviewed category.) Top reviewers also gain "small utilitarian benefits," largely free books although many reviewers said most of the free books aren't worth reading.
The problem appears to be the ranking system of the reviews. Higher rankings were rewarded for three reasons: "overall helpfulness" of the review (as rated by customer votes), the number of reviews the person has written, and the recentness of the review.
The need to deliver numerous and ever-current reviews are leading some prolific reviewers to "cut corners" to drive productivity. Prof. Pinch writes that this "seems to go against the grain of serious book reviewing."
The bigger issue is that driving up the "helpfulness" grade appears to encourage positive reviews. Indeed, some 88 percent of respondents reported that most or all of the reviews they wrote were positive.
Writing negative reviews is avoided because such reviews are more likely to be found "unhelpful" or might solicit "mean spirited" comments from authors or other reviewers in the Amazon Comments section. Reviewers who themselves produce the same sort of material under review are also particularly cautious about posting negative reviews.
The authors of the study said one solution might be paying reviewers while admitting that such remuneration could "undermine the literary value of their endeavors" and also raise doubts that upbeat reviews are just being incentivized with straight cash.
Writing for paidcontent.org, which studies digital content, Laura Hazard Owen said that although Amazon doesn't appear to be manipulating upbeat reviews, the study provides some insight into why the soundness of reviews on other sites, such as Yelp, are also sometimes questioned.
"Reading Pinch's interviews with reviewers, you get a sense of how hard it is maintain the integrity of a process that is dependent on a virtual army of unpaid but still presumably capitalist-minded laborers," she wrote. "If they're not paid, they are going to find other incentives and motivations, which may in some cases work at cross purposes with their primary mandate, to produce honest and independent-minded reviews."