Metacritic works: Why the review-aggregation site is important for the average consumer

Everyone hates Metacritic.

Developers despise it. Bloggers think it’s damaging. And gamers? Well, when they aren’t using it to fuel fanboy rage, they want Metacritic to burn to the ground. That’s pretty much everyone, right? I can’t think of anybody that exists outside of those three groups … well, nobody except for the people that actually use Metacritic.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":728891,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"games,","session":"A"}']

For gamers, developers, and bloggers, Metacritic is just noise. We’re entrenched in gaming’s cultures and subcultures. We know, before a game comes out, if we want to buy it. The reviews only exist to validate our pre-formed opinions. When someone from within our echo-chamber speaks out on Metacritic, invariably they work backwards from another pre-formed opinion: “Metacritic is bad and it’s useless.”

You would think that any good criticism of Metacritic would start by attacking its ability (or inability) to help consumers. If you can prove that it doesn’t help people make good purchasing decisions, then everything else about the site falls apart.

For the unaware, Metacritic is a website that collects and collates review scores from dozens of outlets for music, movies, television, and video games. The site’s editorial team takes those scores and runs them through a proprietary algorithm that generates a final number out of 100. That number is the Metascore.

It’s an incredibly reductive process that torches down criticism until nothing remains but a magical number. Critics hate it for that, but the question is: does the Metascore work?

Does Metacritic’s review gathering and number crunching produce a tool that is useful to consumers? The website’s founder Marc Doyle told GamesBeat that he and his team designed Metacritic for entertainment consumers and that he has known from the beginning that it works.

“Starting out, our own team provided the important evidence [that Metacritic works],” Doyle told GamesBeat. “As we were developing the website back in 1999, we pulled together all the various critic reviews on Excel spreadsheets, calculated what would later become Metascores, and used them in deciding what we should see — and what to avoid. That’s where the idea for Metacritic really solidified.”

Even though Doyle believes in his website he also recognizes that it should only be one tool that consumers consult before making a purchase.

“Even for me, I’ll absolutely see movies, buy albums, and play games with low Metascores because they feature an actor I like or a genre I enjoy,” he said. “But for titles I’m less familiar with, I like to know what I’m getting into with entertainment considering how expensive it has become and how much time a game or TV series can consume.”

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":728891,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"games,","session":"A"}']

It isn’t surprising that Metacritic’s founder believes that it works, but I wanted to look into it myself.

I decided to try to figure out if Metacritic is useful for someone who only has a passing interest in gaming. My hunch is that busy people who don’t spend all their time reading message boards are probably more satisfied with their game-purchasing decisions if they use Metacritic.

That’s a really hard thing to prove. No research exists on the subject, so I’m going to put myself in the shoes of someone using Metacritic as it is intended. The GamesBeat team helped come up with questions that might come up for the average gamer, and I will attempt to figure out answers primarily using Metacritic.

Here is what I found:

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":728891,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"games,","session":"A"}']

I want to spend some money on a PlayStation 3 game. What is good right now?

The first thing Metacritic confronts consumers with is a chart for “new and notable” releases in movies, games, TV, and music. If a person got no further than this, they would see that a game called The Last of Us is listed high above the next closest game.

If they dig a little deeper to the High Scores section, which lists the best-scoring games for the past 90 days, they will see a slightly more confusing list that consists of smaller mobile games, downloadable content, and retail games. By selecting just one platform, in this case the PlayStation 3, the list is much easier to parse.

[aditude-amp id="medium3" targeting='{"env":"staging","page_type":"article","post_id":728891,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"games,","session":"A"}']

The Last of Us still dominates the list, but I can see other games like Hotline Miami and Dragon’s Crown near the top.

The entire top 10 for PS3, not including DLC, looks like this:

1. The Last of Us
2. Hotline Miami
3. Dungeons & Dragons: Chronicles of Mystara
4. Dragon’s Crown
5. GRID 2
6. Magic: Duels of the Planeswalkers 2014
7. Pool Nation
8. Metro: Last Light
9. Stealth Inc: A Clone in the Dark
10. Tales of Xillia

I would have no problems handing that list to anyone looking for a new PlayStation 3 game. Anyone who knows that they don’t like dungeons or dragons and wants a full retail game can quickly squeeze it down to just The Last of Us

Someone who is open to smaller games might even take a chance on something like Hotline Miami simply because it scored so well.

[aditude-amp id="medium4" targeting='{"env":"staging","page_type":"article","post_id":728891,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"games,","session":"A"}']

Consumers can quickly expand this list by searching for the best game released so far in 2013. That reveals games like BioShock Infinite, Tomb Raider, and MLB 13: The Show.

Metacritic isn’t perfect, though. These lists include downloadable content, and the site doesn’t distinguish between the add-on packs and the full games.

All I play is Call of Duty and Madden, but I want to try something new and daring. What is good like that?

I think many people believe that Metacritic’s system rewards conventional games over titles that try something new. The site’s tools do not reflect that. Instead, when I navigate to the High Scores for all systems in 2012, Metacritic gives me a list of quality titles that span most genres.

[aditude-amp id="medium5" targeting='{"env":"staging","page_type":"article","post_id":728891,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"games,","session":"A"}']

I don’t know what game you would expect to find on top of that list, but I doubt it is The World Ends with You: Solo Remix for iPad. That game has a 95, which makes it the best-reviewed game on Metacritic in 2012. The rest of the top 15 is just as unconventional:

1. The World Ends with you: Solo Remix for iPad
2. The Walking Dead: A Telltale Game Series
3. Bastion
4. Persona 4 Golden
5. Mass Effect 3
6. Punch Quest
7. Bejeweled HD
8. Cave Story
9. Beat Sneak Bandit
10. Tiny Wings HD
11. Journey
12. Xenoblade Chronicles
13. Mikey Shorts
14. Reckless Racing 2
15. Dishonored

Almost half of those titles are iOS games, but I think the most interesting item on the list is The Walking Dead. Telltale’s horror adventure title is the highest-rated console and PC game for all of 2012.

If Metacritic truly rewarded safe games that didn’t take any chances, then The Walking Dead wouldn’t be anywhere near the top of the list. The truth is that reviewers almost universally rewarded the title for attempting something new and for having the skills to execute the design.

This is exactly the game I would want under-informed consumers to learn about and try. If they use Metacritic, then it is very likely that will happen.

If The Walking Dead doesn’t sound good to them, then they could end up trying Bastion, Person 4 Golden, Cave Story, Journey, or Xenoblade Chronicles. These are all beloved games that in no way reflect our industry’s obsession with military shooters. In the battle to expand the audiences for different kinds of games, Metacritic fights for the good guys.

I want to get a baseball game. Which one is best?

The stereotype for the under-informed gamer is typically someone who only buys one or two games a year. They want their sports games and maybe Call of Duty. It’s easy to dismiss that kind of gamer as a casual because they don’t ingrain themselves in the culture, but while traditional media outlets ignore these people, Metacritic serves their needs.

For example, two officially licensed Major League Baseball games hit the market in 2013. One from Sony and another from 2K Games. A quick search on Metacritic for “MLB” instantly reveals that Sony’s MLB 13: The Show for PlayStation 3, which scored an 87, is probably a better bet than 2K Game’s MLB 2K13, which scored a 48.

It also reveals that the PlayStation Vita version of The Show (75) isn’t quite as good as the console version, but both iterations of The Show are better than 2K’s release.

This same strategy works for licensed games in entertainment.

For example, two different games based on The Walking Dead franchise hit consoles in the last years. Fans of the show or the books might have no idea which one is the good one, but Metacritic again reveals that the Telltale game (92) is amazing while the Activision shooter, The Walking Dead: Survival Instinct (32), is atrocious.

I have an Xbox 360, a PlayStation 3, and a PC, which one should I get BioShock Infinite for?

For this question, I again just used Metacritic’s search tool. The only three results for “BioShock Infinite” were for the game on Xbox 360, PlayStation 3, and PC. When I sorted by score, I found that Metacritic had the PC version and PlayStation 3 version of the game listed at a 94. Meanwhile, the Xbox 360 variant has a 93.

Someone might believe this means that the PlayStation 3 and PC versions are on par with one another. That isn’t the case.

DigitalFoundry, a website that runs comparison tests for games on different platforms, found that BioShock for PC has a better frame rate, richer lighting, and more-detailed textures than the game running on PS3. Everything else is identical. The website also found that the PS3 version runs slightly better in very rare occasions.

In a situation like this, Metacritic is clearly capable of misinforming consumers.

There are a ton of iPhone games — what releases are actually good?

Discovering quality games on consoles is tough for a lot of people outside of the hardcore. On iPhone, that difficulty applies to just about everyone. Apple’s mobile platform has thousands of games and great products struggle to standout in that crowd.

A lot of people use the iTunes charts but those follow sales trends rather than any measurement of quality. The Games section of the iTunes App Store, on the other hand, often offers up “New & Noteworthy” games and an “Editor’s Choice” section. These can quickly lead consumers to quality titles like Rymdkapsel. Apple’s market also has a robust user-reviews system that can give gamers an idea of how other people felt about a release.

In the case of a game like Rymdkapsel, gamers could check out Metacritic to see what reviewers think. A lot of mobile games don’t get enough reviews to qualify for a Metascore, but most of the big ones do. Rymdkapsel is one of those; it scored a 90 based on the reviews of nine critics. In this circumstance, Metacritic isn’t really providing any new information. At best, it is merely confirming the App Store user reviews.

Metacritic isn’t useless for mobile games. It is still an excellent way to find out about a bunch of high-quality titles in a short amount of time. Metacritic’s iOS landing page surfaces all the best-scoring games from the last 90 days and often features games that the iTunes App Store doesn’t.

For example, XCOM: Enemy Unknown is currently nowhere on the front page of iTunes’ gaming section, but it is right at the top of the best-reviewed iOS titles on Metacritic.

I saw a commercial for Defiance the game while watching the television program. I like the show, but is the game any good?

Earlier this year, publisher Trion Worlds inundated television with advertisements for its multiplayer online shooter Defiance. The sci-fi game is a companion to SyFy’s hour-long action-drama of the same name. When that show debuted, it was one of the highest-rated premieres in SyFy’s history. That means that several hundred thousand people got to see a commercial for Defiance the game a bunch of times. Some of them probably wondered about running out to the store to check out the game. Over 1 million did, according to the developer.

Maybe a lot more (or less) would have purchased the game without Metacritic.

Defiance for PC has an average rating of 65. That means reviews were generally mixed. According to Metacritic, Defiance has five positive reviews, 32 mixed reviews, and two negative reviews. The quotes Metacritic pulled from critic reviews reveal that the game’s shooting mechanics are solid and that the title is similar to Borderlands. The quotes also often ding the game for a lack of humor and for not living up to the quality of Borderlands. In general, fans of the show would probably enjoy it if they like shooting stuff with people online.

This is the kind of information that can give a person confidence in their purchasing decision, or it could help them decide to save their money. Either way, it’s good data to have.

A useful tool for normal people

Metacritic definitely isn’t perfect. Even its creator acknowledges that it is just one tool that consumers should use to make a decision, but it is a useful tool.

When I take a step back and put myself in the shoes of a less-informed gamer, I found that Metacritic is a shortcut to getting up to speed about the relative quality of recent releases. It’s easy to understand and mostly accurate.

If you’re a hardcore gamer, it’s probably still going to make more sense for you to cultivate a relationship with a couple of critics who mirror your tastes. For anyone who doesn’t have time for something like that, Metacritic is a smart way to make game-buying decisions.

But what about the evil publishers?

One of the big criticisms against Metacritic is that publishers use it as a barometer for success and base bonus contracts for developers on whether they reach a certain Metascore threshold. That is potentially unfair, but it’s not Metacritic’s fault that game publishers have skewed the website’s purpose.

It seems silly to scold Metacritic for something it can’t control especially when it offers up such a useful service. Although, some disagree that it is truly useful.

“When we put our faith in Metacritic as an impartial, scientific measure of quality, we should probably ask ourselves whether the crowd – the crowd of journalists as well as players – is really wise or just mediocre, incapable of recognizing and rewarding the new and different,” Epic Mickey creator Warren Spector wrote in a piece slamming Metacritic. “Before you come back at me with arguments, look at the highest rated games and then we’ll talk….”

But it’s hard to see what he means. Among the highest scoring games of all time on Metacritic are Half-Life 2, World of Goo, LittleBigPlanet, Portal, Grim Fandango, and Minecraft. Games like The Walking Dead and Dark Souls placed near the top of the website’s highest-rated releases for 2012 and 2011, respectively. These games are all — or at least were — very different, and the critical consensus recognized and rewarded that.

It seems to me that Metacritic works, and that’s probably why publishers keep using it.