Metacritic Scores vs. Sales: 2009 Holiday Season Analysis

Recently in GamePro, Julian Murdoch examined the reviews aggregator site Metacritic. His story opens with a rather startling observation from Activision Vice President of Marketing Robin Kaminsky at the 2008 Design, Innovate, Communicate, and Entertain (DICE) Summit. The opening slide to her presentation read: "For every additional five points over an 80 percent average review score, sales may as much double."

Incredible! If true, game sales must increase exponentially as they score above an aggregate of 80 on the website. Surely, an employee of Activision — let alone a VP of marketing — must have solid data to back up such a bold declaration.

But I'm the skeptical type, so I did a little of my own research. I’m here to tell you that Kaminsky's statement is a load of horse shit, and that publishers' insistence on a correlation between scores and sales is unfounded.

Murdoch’s article, which is well worth reading, focuses not on Metacritic’s influence (or lack thereof) on sales, but on the connection that publishers make between scores and sales. In other words, he investigates the shady practice of tying developer compensation to review scores.

My goal, however, is to see if any meaningful correlation between Metacritic scores and unit sales exists at all. When I previously studied this issue, I concluded that the former have virtually no influence on the latter. Critically acclaimed games, like the Wii’s Boom Blox: Bash Party (which received a Metacritic score of 86), sold poorly, while much lesser-scoring games, such as Night at the Museum or Terminator: Salvation, sold significantly more.

My sample size for that examination was small — I used May 2009 releases and 10 weeks of their sales data. To investigate this matter further, I decided to collect a much larger, more representative sample. My analysis for this article includes everything that came out between October 1 and December 31, 2009.

For each title, I collected 10 weeks of sales data as it was posted on VGChartz. Any releases that did not have the full 10 weeks and a Metacritic score were not included. Out of 364 titles (including all system ports between the Nintendo DS, Nintendo Wii, PlayStation 2, PlayStation 3, PlayStation Portable, Xbox 360), 218 made the cut. PC, iPhone, and digital releases were not included because of the lack of sales data.

Read on to page two for the breakdown.

First, I’ll analyze the data in the exact same fashion I did back in September. I combined cross-platform game sales and averaged Metacritic scores between platforms to arrive at one score for all versions of a title.

Click for a larger view

We have a few clear outliers — Modern Warfare 2, New Super Mario Bros. Wii, and Assassin’s Creed 2. These three were the only titles to sell more than three million copies (approximately 9.5 million, 5 million, and 3.3 million, respectively). They were also some of the highest scoring games (94, 87, 84, respectively). We can’t really tell much from the data by looking at this chart, so let’s remove the outliers from the analysis.

Click for a larger view

Here we get a slightly clearer picture, but we still have this giant mass of games in the top-left corner of the chart. Like my previous study, this data gives a similar impression — a few games sell well and generally score above 60, while most others sell poorly regardless of their score. Again, no games score extremely low and sell very high.

A few interesting examples, though, are in this chart. Tony Hawk: Ride (PS3, Xbox 360, and Wii) sold about 943,000 copies with an averaged review score of 46. On the other hand, the Xbox 360 exclusive Forza Motorsport 3 sold similarly (about 960,000 units) but scored twice as high — a 92. What’s going on here?

Another comparison that gives me pause — the PSP version of Grand Theft Auto: Chinatown Wars sold more than 120,000 copies with Metacritic score of 90. But Rogue Warrior, which appeared on the PS3 and Xbox 360, sold nearly the same with an aggregate score of 28. Yes, you read that right.

The cluster of games in the top-left corner deserves closer examination. It looks like most of these titles sell less than approximately 500,000 copies, so let’s zoom the chart in there.

Click for a larger view

The data shows no clear trendline. We have a large scatter of points between scores of 40 and 80, with few outliers. Again, I’ll make another comparison to GTA: Chinatown Wars — look at Star Wars: The Clone Wars: Republic Heroes, which sold almost four times as many copies with an averaged score of 45 — fully half that of Chinatown Wars.

But it may not be fair to combine cross-platform data like that and make valid comparisons — handheld and Wii versions can vary greatly from their PS3 and Xbox 360 counterparts. Certainly, that data is interesting, but let's review scores and sales for each system.

Click for a larger view

Again, the full chart doesn’t tell us much, other than that most games are going to sell less than a million copies, regardless of their Metacritic score. We see the same outliers, too. Let’s zoom right in to those that sold 500,000 copies or less.

Click for a larger view

Once again, no clear trends come out of the data. Each system’s data points scatter widely, with no clear correlation apparent between sales and scores. Interesting to note is that PSP games sell especially poorly (LittleBigPlanet is the handheld’s highest selling game with approximately 260,000 copies).

Also interesting is the sharp drop in quality for Wii games as they sell more copies. As we go beyond 300,000 units sold, Wii games drop from a Metacritic score of 72 to a score of 48 (LEGO Indiana Jones 2: The Adventure Continues, NERF: N-Strike Elite, Hasbro Family Game Night 2, Tony Hawk: Ride, Just Dance, respectively). I don’t think this is a trend of any sort, but it’s the only system to have such poorly scoring games also sell well.

---

Looking at the top five games in both sales and scores, we can see problematic examples. High scoring games such as Half-Minute Hero (PSP), Pro Evolution Soccer 2010 (Wii), and Might and Magic: Clash of Heroes (DS) all sold terribly low — 34,000, 57,000, and 40,000 copies, respectively.

And a few high sellers that far outpaced the aforementioned titles, Assassin's Creed: Bloodlines (PSP), Just Dance (Wii), and Pokémon Mystery Dungeon: Explorers of Sky (DS), scored poorly — 63, 48, and 53, respectively.

Below are tables with each system's top five games ranked by Metacritic scores (left) and unit sales (right).

PS3	Rank	Total Sales	Score	PS3	Rank	Total Sales	Score
Uncharted 2: Among Thieves	1	1,118,574	96	Modern Warfare 2	1	3,310,903	94
Modern Warfare 2	2	3,310,903	94	Assassin's Creed II	2	1,285,815	91
Assassin's Creed II	3	1,285,815	91	Uncharted 2: Among Thieves	3	1,118,574	96
God of War Collection	4	554,536	91	God of War Collection	4	554,536	91
FIFA Soccer 10	5	280,984	91	Dragon Age: Origins	5	455,755	87
PSP	Rank	Total Sales	Score	PSP	Rank	Total Sales	Score
GTA: Chinatown Wars	1	123,695	90	LittleBigPlanet	1	260,086	87
LittleBigPlanet	2	260,086	87	Assassin's Creed: Bloodlines	2	219,983	63
Half-Minute Hero	3	34,287	84	Star Wars Battlefront: Elite Squadron	3	149,392	63
FIFA Soccer 10	4	97,475	83	GTA: Chinatown Wars	4	123,695	90
Tekken 6	5	70,748	82	Yu-Gi-Oh! 5D's Tag Force 4	5	121,150	70
Wii	Rank	Total Sales	Score	Wii	Rank	Total Sales	Score
New Super Mario Bros. Wii	1	5,042,122	87	New Super Mario Bros. Wii	1	5,042,122	87
DJ Hero	2	240,915	87	Wii Fit Plus	2	1,967,654	80
Pro Evolution Soccer 2010	3	57,190	82	Mario & Sonic at the Olympic Winter Games	3	893,648	72
Wii Fit Plus	4	1,967,654	80	Modern Warfare: Reflex Edition	4	549,603	76
EA Sports Active: More Workouts	5	194,325	80	Just Dance	5	470,787	48
DS	Rank	Total Sales	Score	DS	Rank	Total Sales	Score
The Legend of Zelda: Spirit Tracks	1	788,519	87	The Legend of Zelda: Spirit Tracks	1	788,519	87
Might & Magic: Clash of Heroes	2	40,516	86	Mario & Sonic at the Olympic Winter Games	2	520,568	76
Space Invaders Extreme 2	3	43,989	82	Bakugan: Battle Brawlers	3	501,933	60
LEGO Rock Band	4	107,884	80	Pokémon Mystery Dungeon: Explorers of Sky	4	348,233	53
Mario & Sonic at the Olympic Winter Games	5	520,568	76	Drawn to Life: The Next Chapter	5	309,289	72
X360	Rank	Total Sales	Score	X360	Rank	Total Sales	Score
Modern Warfare 2	1	6,232,345	94	Modern Warfare 2	1	6,232,345	94
Forza Motorsport 3	2	963,155	92	Left 4 Dead 2	2	1,938,107	89
Assassin's Creed II	3	1,936,410	91	Assassin's Creed II	3	1,936,410	91
FIFA Soccer 10	4	384,540	90	Forza Motorsport 3	4	963,155	92
Left 4 Dead 2	5	1,938,107	89	Dragon Age: Origins	5	938,898	86

From this data, what can developers and publishers expect after a game's release?

Based on Metacritic’s own scoring system, I split the data into four categories: universal acclaim (a score of 90 or more), generally favorable reviews (between 89 and 75),  mixed or average reviews (between 74 and 50), and generally unfavorable reviews (49 or less). I decided not to use "overwhelming dislike" because no games scored that low in my sample.

Overall, most titles — 58 percent — received mixed reviews. Additionally, most sold less than 100,000 units (62 percent).

Even more depressing, a plurality of games — 36 percent — sold less than 100,000 copies and received mixed reviews. Twenty percent of all games studied sold between 100,000 and 500,000 fell into the same scoring category. Only three percent of all games analyzed sold more than 1 million copies and received Metacritic’s most coveted rating — universal acclaim.

Click for a larger view

Includes sales data for games studied without a Metacritic score. Click for a larger view.

---

I suppose that Kaminsky gave herself an out by qualifying her statement with “may.” If what she proposes were actually true, we’d expect to see the above scatter charts’ sales data curve upwards exponentially above a Metacritic score of 80. In fact, what we actually see is most games lingering in the top-left corner of the chart (i.e., low sales, moderate scores) with a few runaway titles garnering millions in sales.

My analysis demonstrates that no link between Metacritic scores and sales exists. So why do publishers insist on such a connection? I can’t really say. That reviews scores would influence sales does feel intuitive. Perhaps publishers also feel reassured that they've based their decisions on a quantifiable number.

But I won't pretend that I have all the answers. I don't really know why or why not a particular game sells well or poorly. My data isn't equipped to answer that question, either. What I do know is that no correlation between Metacritic scores and unit sales exists, and I think it's foolish for publishers to tie monetary compensation to such scores.