ReviewMeta works very hard to position itself as some kind of independent arbiter of Amazon reviews, but an examination of its methods proves two things: 1) ReviewMeta is not very accurate and 2) ReviewMeta does not like being reviewed.
An article from the Washington Post a while back claimed that Amazon is undergoing a fake review crisis. There are problems with Amazon reviews, no doubt, but this article is based on some pretty data – at least in how it pertains to the world of books, which is what I know, and what I’ll focus on here. I can’t speak to the world of diet supplements or fake tan or giant tubs of lube… alas.
Is ReviewMeta Reliable?
The article’s claims are largely based on a flaky site called ReviewMeta, which seems far better at getting publicity for itself than correctly analyzing the trustworthiness of Amazon reviews, which is a pity as it would be a wonderful tool if it were in any way accurate.
As soon as I heard about ReviewMeta, I immediately wanted to test it. Amazon tends to be plagued with all sorts of scams – which I have written extensively about over the last ten years – and I could instantly see the value in a tool which could identify fake reviews or suspicious products.
Naturally, I started with my own books, as I can be pretty sure they have no fake reviews. As well as the author, I handle all the publishing and marketing personally, and I’m fastidious about the rules both for ethical reasons and commercial ones; my name is literally my brand.
However, ReviewMeta seems to call into question a large number of my reviews and reviewers. And by extension me, I guess. And many of my fellow authors too, it seems, because a large portion of the random selection of books I checked had similar issues. What was going on here?
How ReviewMeta Works
The ReviewMeta site helpfully gives explanations for why its system made these determinations, and you can actually break down each component and get a further explanation. This transparency is hugely commendable.
Digging into this data, though, shows the extreme limitations of the site and the way it calculates the trustworthiness of reviews – at least how it pertains to the world of books.
Perhaps it is more accurate for jellybeans or computer peripherals, I really can’t say. But when it comes to books, it makes a number of pejorative assumptions about what is legitimate reader behavior, such as reviewing Book 2 of a series after reviewing Book 1 or mentioning the title of the book in the review, and these routine reader actions cause ReviewMeta to flag these reviews as questionable or suspicious.
This then casts aspersions on the integrity of the authors of these books – who are self-employed people working in an industry where reputation and integrity are critically important, not huge faceless brands… if that matters. Worse still, the site has been aware of these issues for two years, and not only have they not corrected them, they reacted in a hostile way when presented with this information.
Let’s take a look at some concrete examples. Once again, I’m happy to be the guinea pig here, and have all of you (and ReviewMeta) poke and prod my reviews and check the authenticity of same, because I have 100% confidence that they are all genuine.
ReviewMeta Case Study: David Gaughran
Here is ReviewMeta’s take on “David Gaughran” the brand and how trustworthy it is (that’s me, btw).
Okay, this doesn’t look good. And if you look down the page it shows each of the products they have assessed that led to this overall brand trustworthiness score. You can see many of my books have “failed” in the eyes of ReviewMeta and “Unnatural reviews detected” has been appended to several of my books.
Crikey.
You can click on each product and see how it came to that determination, and the supposed evidence for each component of that decision. Again, I stress, this transparency is truly commendable.
But this breakdown also reveals the faulty assumptions that led to these incorrect determinations about my reviews. And it’s not just my reviews, of course. These simplistic calculations affect most authors – feel free to test it out yourself.
Of course a reviewer of Book 1 and 2 is likely to review the third book in a trilogy. If you don’t take account for that wholly natural behavior when analyzing book reviews, then all your results will be skewed. Mentioning the title of the book is another pretty common thing that (genuine) book reviewers do, but ReviewMeta views with extreme suspicion.
I pointed all this out in a series of tweets to ReviewMeta back in 2016, and they responded with a pissy blog post, which ignored the constructive suggestions I made to improve ReviewMeta and properly take account of the way that reviewers review books – both to miss all the false positives they seem to be generating, and also take account of suspicious patterns in book reviews they were missing.
An Overarching Focus on PR
But none of these issues have been addressed by ReviewMeta in the last two years. Instead they seem exclusively focused on publicity. Here is founder Tommy Noonan talking to Techspot in November 2016, then CNET a few months later in February 2017, and there have been similar pieces over the last couple of years which I couldn’t be bothered linking to in Forbes, NYMag, Scientific American, Quartz, PBS NewsHour, BuzzFeed, ZDNet, Business Insider, and many, many more.
In all that time of furious self-promotion, I haven’t seen ReviewMeta improve the accuracy of its site.
The sad thing about all of this is that Amazon does have a fake review problem, one which is compounded by Amazon deploying a fake review detection algorithm that seems about as accurate as the one from ReviewMeta, perhaps for similar reasons too. Which means that authors innocent of any wrongdoing get genuine, organic reviews from bona fide reviewers removed every day and the scammers and cheaters with fake reviews keep getting away with it. Sites like ReviewMeta aren’t helping with this problem, they are making it worse.
But the worst part of all, perhaps, is the complete misunderstanding of how Amazon algorithms work. Reviews don’t cause sales, they are a symptom of sales.
Yes, a lot of overwhelmingly positive reviews will sway an on the fence-purchaser, but they don’t automatically lead to sales – not in the world of books, at least. Maybe if I’m looking for a phone charger and they are all more-or-less fungible, reviews might become the tie-breaker for a lot of people. Not with novels. I don’t care how many reviews the Da Vinci Code has, I’m never going to read it.
The continued media focus on “fake” reviews – driven in part by ReviewMeta’s relentless publicity drive – is taking attention away from much more serious issues that the media have not covered in any depth, such as clickfarming, bookstuffing, incentivized purchasing, and mass gifting.
That’s not to say that ReviewMeta couldn’t serve a useful purpose, or something like ReviewMeta. Unfortunately, ReviewMeta itself don’t seem interested in addressing the flawed assumptions underlying its product, and trying to make it more accurate.
Which is such a shame.
I tried to engage with them once more. I left a comment under that prickly blog post responding to my series of tweets on ReviewMeta.
In the most ironic twist since it rained on Alanis Morissette’s wedding day, that comment was deleted. It seems ReviewMeta doesn’t like being reviewed.
Guess Who Doesn’t Like Being Reviewed?
Okay, so I was wrong. Here’s something even more ironic: a user of ReviewMeta on the consumer side left a lengthy review of ReviewMeta on Trustpilot – a genuinely constructive review which sought to identify genuine shortcomings in ReviewMeta, based on their own experience.
And here’s the crazy part: ReviewMeta’s owner was so annoyed – once again – at being subjected to any kind of critical scrutiny that he contacted the reviewer and asked them to change or delete their review.
“This review is hurtful to our business… please consider deleting or changing your review.”
ReviewMeta
Even a fiction author couldn’t make it up.
