Data analysis shows what you suspected all along: Amazon ratings are bullshit

After analyzing over 18 million reviews, independent analysis confirmed that Amazon ratings are heavily influenced by incentivized reviews. Incentivized reviews, in this case, means a discounted (or free) product received in exchange for an Amazon review.

Sellers incentives for reviews aren’t necessarily asking for positive reviews, but in general, they aren’t handing out products that they suspect you’ll hate. By law, these incentivized reviews must also be disclosed to the public in an effort to avoid dishonest sales practices.

Review Meta offers a free online tool that gives a detailed report of any product on Amazon and offers an honest review after stripping out incentivized versions. With the tool, and the relative ease of finding text associated with an incentivized review, Review Meta went to work analyzing the data and looking for patterns.

According to its data, only about 20 percent of reviews on Amazon feature an incentive. This small number, however, leads to significant shifts in overall review data.

The average Amazon product has a rating of 4.4 stars. When you factor in incentives, this rating jumps to 4.73 stars, while non-incentivized ratings hover at around 4.33 stars. This 0.40 ratings bump might seem insignificant, until you learn that even an average product (with its 4.4 star rating) jumps from the 54th percentile to the 94th percentile once you factor in the 0.40 star ratings bump.

Even when controlled for sellers that are only sending their best products out for review, the discrepancy was still 0.36 stars higher, only a 0.04 star variation from the non-control data.

Review Meta looked at at 807 products that had at least 10 incentivized and 10 non-incentivized reviews, a total of 1.7 million reviews. Of those, the vast majority (87 percent) received a higher rating from incentivized reviews. To put that into perspective, almost nine out of 10 reviewers rated a product higher than the average non-incentivized reviewer.

This leads to skewed ratings for any product that contains incentivized reviews, a problem Amazon recognizes and assures TNW that it’s “working on” it. Amazon’s representative also promises “some significant changes that will be rolled-out shortly.” A request to clarify when we might see the changes or what they entail went unanswered.

For now though, take those reviews with a grain of salt, if you weren’t already.