This post has not been edited by the GamesBeat staff. Opinions by GamesBeat community writers do not necessarily reflect those of the staff.
Updated at 11:30 a.m. Pacific: After internal discussion, I’ve opted to change “backpedaling” in my editor’s note to “subsequently changed” to reflect the fact that updating reviews in this fashion has been Polygon’s policy since the site’s founding. -Rob
Polygon’s review policy, titled Polygon’s Review Scores and “The Bump,” is about as toothless and cosmetic as policy documents come. There’s a pretty lengthy preamble to give the reader context as to why an outlet would have an actual policy to begin with, and this primarily relates to the evolving nature of the game as a product. There’s no definitive purpose spelled out in the document, and by that I mean that I couldn’t tell you what purpose a Polygon review is supposed to serve other than to establish and maintain the site’s “legacy and credibility.” There is an acknowledgement that reviews are valuable to the site’s readership but also a puzzling commitment to “being as fair as possible to the developers and publishers who pour time and money into the games you play.” Is a Polygon review supposed to be a purchase recommendation or an evaluation of a game under ideal conditions? From the policy document, it’s difficult to say for sure.
I intend to take Polygon to task for one statement in the policy, though, and it reads as follows:
At Polygon, we operate on the supposition that reviews are a critical evaluation of a game on the day of its release, which the score we assign reflects.
If the review is supposed to be a “critical evaluation of a game on the day of its release,” then this should imply a few things that would put it at odds with the typical games website milieu of assessing review code in the weeks leading up to release — sometimes even at events held at the publisher’s offices. Ideally, a review under Polygon’s supposition would be conducted using a retail copy of the game. I wouldn’t expect Polygon to buy it (publisher supplied would be above board), but the publication’s recent review of SimCity — along its with subsequent revisions — shows just how meaningless the policy and the concept of “The Bump” really is.
To be clear, I’m not alleging that there’s any impropriety on the part of the reviewer, Russ Pitts, or Polygon’s reviews editor, Arthur Gies. It is stated in the site’s forums, publicly on Twitter, and in the review itself that the game was trialed on “development servers.” I am, however, implying that Polygon’s review policy is for show only, with its writers being able to point to it as proof of the site’s supposed integrity rather than it actually guiding the site’s process and practices.
The employment of The Bump seems to be at odds with the grand scheme of the broken policy. To acknowledge the trouble that most consumers have endured trying to connect to publisher Electronic Arts’ servers and actually play SimCity on release (March 5), Gies employed The Bump and reduced the game’s review score from a near-perfect 9.5 to a still respectable 8.0. Just two days later, the score dropped significantly to 4.0 to recognize developer Maxis’ removal of certain features that apparently affected Pitts’ enjoyment of SimCity. Once again, to quote the policy, review updates are supposed to chart the “timeline of that game’s evolution.” I would argue that it’s not to document launch hiccups.
Polygon’s review of another always-online PC title, Diablo III, that endured issues with access at launch is proof of my assumptions regarding the intent of the policy. In a sidebar, titled “Server Issues,” Arthur Gies explains why the dungeon crawler’s score remained at a perfect 10 with the following:
Ordinarily, my position as reviews editor at Polygon is that we review a game as it exists on release day because our responsibility is to our audience. While we do all we can to maintain due diligence with regard to giving a game every opportunity to deliver, we choose your wallet and your time before the benefit of the doubt.
But Diablo III is different. It’s different because Blizzard has a track record spanning almost two decades of games that have become institutions, and it’s also run the most popular MMO [editor’s note: massively multiplayer online game] around for almost eight years. Put simply, Blizzard, more than any developer around, has earned that benefit of the doubt. I believe that the server issues will be resolved. With that in mind, it does both our audience and Diablo III a disservice to dwell on that aspect in this review.
So in this case, Blizzard, a high-profile developer that has experience delivering games with online infrastructure is spared the wrath of The Bump as “they’ve earned the benefit of the doubt.” I can’t attest to developer Maxis’ experience with online-only products, but EA has released a vast catalogue of online-dependent or otherwise enabled titles in a variety of genres. For what reason does the Polygon review team believe that EA is unable to deliver a stable experience? How have they not earned the benefit of the doubt? At least in the case SimCity, Maxis delivered the product in full and removed features (and will presumably restore them once the servers become stable), as opposed to Diablo III where the real money auction house and player-versus-player features weren’t ready for the release date and took varying amounts of time to be delivered.
This lack of consistency again raises the question of what purpose the review is supposed to serve — not just at Polygon but at games writing outlets in general. Is the review supposed to be a purchase recommendation for readers? If so, Polygon did a disservice to their audience by trialing SimCity in conditions different to those that would be available to consumers and bestowing a near-perfect score. Besides, it’s not like The Bump will carry over to Metacritic, whose ratings are the only thing arguably close to an objective measure in terms of games reviews. Surely, with the collective experience of Diablo III’s muddled launch, the Polygon team must’ve thought it prudent to test the retail servers before publishing a review for another always-online game. Then again, if the review is supposed to assess the game in ideal conditions, then The Bump should never have been applied as Pitts got to see the game at its best.
I approached both Pitts and Gies with my concerns via Twitter and received a reply from the reviewer (Pitts), who stated (across multiple tweets) that:
We did our best to ensure we were reviewing the game in conditions as approximating the user experience as closely as possible. That’s not always going to be 100 percent. And in this case, launch conditions were far more terrible than anyone assumed. Hence, the repeated updates. TL; DR: This is an extreme case of almost everything.
I responded with the allegation, as per the above, that Polygon’s policy is more for show and highlighted both the breach of policy and the inconsistency with regards to the outlet’s Diablo III review, and was told that:
It’s not that simple, but I appreciate your feedback. Definitely something to consider.
Look, I’m not naive. I know that most gaming sites live and die by their reviews and that waiting to trial the game on retail servers would have cost Polygon no small amount of hits. Furthermore, it’s possible that non-disclosure agreements were signed and delivery dates may have been promised. I don’t know all the factors that were in play, but I know that Pitts was being genuine with his response.
When we’re talking about “legacy and credibility” and integrity, however, sacrifices need to be made. I mean, why have a policy if you have no intention of honoring it? I’d point to Gamespot and senior editor Kevin Van Ord’s review of SimCity as an example of best practice: It acknowledges the broken state of the game as well as highlights promising aspects that just weren’t functional at launch. I think if you’re going to apply a bump to the review score, you should start with the product that was delivered to consumers and work up (or down) from there.
I work with well-written policies, and I work with dated, ineffective ones as well. A good policy accounts for most situations and variables while allowing sufficient flexibility through interpretation for the benefit of stakeholders. Polygon’s review policy is not a well-written policy, and while not old, has already been rendered ineffective by the business of video games writing. If you want that audience (or, better yet, to expand it), you’ve got to go to press early and be first if you can.
Promising to evaluate something close to what your readership will have in its collective hands is at odds with this business model, so Polygon has to make a choice: Do you stand with your audience, or do you postulate to them? Whatever the answers to this question and the others published above, I think the site’s management needs to revise the policy or consider withdrawing it all together.
I’m not a professional games writer. I’ve wanted to be one since I first picked up a compendium of reviews for games on the Sega Master System and Genesis, but I’ve made peace with the fact that I’ll always be a commentator more than a genuine contributor. That, and my current occupation — providing policy advice and administration services in a university context — has done a good job of putting food on my table and a roof over my head. You might see me pop up on a few different gaming sites, but rest assured that my money has and will always be earned elsewhere. I hope that with this footnote that you’ll be satisfied that this examination of Polygon’s review process and policy isn’t sour grapes; it’s me looking at games writing in the context of my day job.