Starhawk developers learned from social games that “analytics is king” (interview)

GamesBeat: That’s one of those interesting lessons that the social game companies taught everybody. Zynga and its analytics enables them to see something happen in the game and then change the game for the next day.

Jobe: Absolutely. And we learned that. It was actually incredibly eye-opening for me. As odd as this sounds, I was at an Austin Game Developers Online conference, and I really wanted to learn more about the analytic side, because I knew its direct application to shooters, especially a global online shooter, could be huge. I sat in a bunch of sessions with Nexon, Zynga, and a bunch of companies that were really steeped in backend analytics. Some of them had their own systems. Many of them kept referring to a piece of software called Tableau, which is a really great data visualizer. And I really didn’t know anything about it at the time, but I knew that this was something that I needed to know more about, that we needed to integrate with Starhawk.

I remember coming back to the studio, sitting down with our systems designer and saying, ‘You know what? This software looks like something we need to use. It’s going to be a great way for us to visualize all kinds of design stuff. Get on it.’ And that was really the start of it. Tableau software worked out great for us. Of course we use a very involved custom Sony backend, but Tableau software is a really great visualizer, and I would highly recommend that for any game studio. It’s great. And we did exactly what you said regarding the learnings that we’ve all gotten from social games.

We did that, literally on a daily basis, during our international Starhawk beta. We would look at the data, see what was happening, we would look at the forums and Twitter and Facebook. Of course, you have the passionate internet fans. They may have had just one bad game, so they immediately think that whatever weapon they were using at the time was underpowered. But you compare the passion of their response with the cold clarity of the objective data to steer the design decisions and the tuning of the game.

Math is the universal language, right? While we have a ton of players in North America who we communicate with quite frequently and clearly in English, we have players everywhere. So we were able to get essentially playtest data from all nations, whether or not we spoke their language, whether or not we could read their forum posts. It was really interesting stuff that came out of that data. Stuff that was surprising to us. I made a presentation about this at SXSW this year. It blew my mind to see that Russians playing the Starhawk beta accounted for the most proximity mine kills. That was a group of people on earth that love to plant proximity mines the most, and they accounted for the most proximity mine kills. Some of the best players on the planet, in fact the biggest percentage of great players, even though it was relatively small, but they were so good, were from Greece.

GamesBeat: I guess there aren’t many jobs in Greece? People have some time to get pretty good at games.

Jobe: Oh my god, yeah, I guess so. I actually made a joke about it, which was probably off-color at the time. I said, ‘I guess when your economy and government’s in free-fall, you get to play a lot of games.’

GamesBeat: Are you able to take some of these learnings and actually turn them around in a day? Or because it’s a console game, does it take you longer to do?

Jobe: It depends on what system the change is in. One of the things that we were really inspired by was a system that Naughty Dog fielded way back on Uncharted 2. We were captivated by this. They were able to make some relatively quick game balance changes. They had exposed some weapon systems in the game that they could make adjustments to, and I was captivated by that. So much so that we developed, at a very foundational level across the entirety of our engine in Starhawk, the notion of a “hotfix” system. Which was very much inspired by Zynga A/B testing and what Naughty Dog did with Uncharted 2. It’s basically a way for us to take all of the parameters that would otherwise be encoded into the executable itself, extract as much of it as possible, including art assets like texture maps and the placement of pickups and bunkers and walls and all that stuff, and extract that out into a layer that can, for all intents and purposes, bypass Sony’s potentially slower patching system, so that we could take advantage of the analytics.

And we did that quite frequently in the beta. Even simple things. I would see on Twitter a picture someone would take with the iPhone where he found a hole in one of the levels. So I immediately take that picture back and say, ‘Art team, find where this hole is in this picture and create a hotfix and we’re going to push it tonight.’ And we could do it. So that’s like a defect fix. But at the same time, with the analytics, we could do things like, ‘Wow, the auto-turrets are getting way too many kills. Over the course of a couple of days, let’s start to spin that back down a little bit and look at the trend of total kills that all players on the planet are getting as a function of auto-turrets.’ So we would look at the trend line, almost like judging a stock, and we could do tuning to the game to bring that down. One of the companies that does a phenomenal job using analytics in game tuning is Riot Games, with League of Legends.

GamesBeat: It sounds like this is a relatively new learning that sort of started in social and is becoming useful on the consoles.

Jobe: I think so. With other big studios, I’m sure they were probably ahead of us. But I can say, at least for us, it was a really big thing, and I know that Sony has requested of us additional examples of how we use the analytics system, how we plot different things on maps, and how we glean that data. And so I think there’s still a lot more to be done. It’s really interesting. One of the things that we will probably be rolling out in a patch in Starhawk after launch is even further analytics. Because after we got the system done and the game was essentially out for replication, manufacturing, we ended up creating a much bigger list of development analytics that we’re really interested in, and that we didn’t have time to put in. And they’re just really kind of geeky engineering stuff, like ‘How many times are people doing matchmaking versus a created game list? How long is the average time spent matchmaking by region?’ These are some really important statistics that we think we could benefit from for tuning our backend infrastructure for the global server set. But those are analytics that we’ll have to add further down the road.

GamesBeat: It sounds like these are also some lessons for the console makers and what they could do with future consoles and the updating of games and things like that.

Jobe: Oh, yeah. I mean, let’s be honest. The frequency of updates that players are expecting from contemporary games is a frequency that is often much faster than most publishers can support. And, quite frankly, if you look at what the next generation of hardware has in store for us, what are systems, at the hardware or the operating system level, that could have really great analytics in place? Whether it be in an API provided by the hardware vendor or whatever. There’s a lot of opportunities there, I think, for expansion. And I think in some respects we have to catch up a little bit to what we see coming from some of the social gaming pioneers.

GamesBeat: Do you already know some of this, about how soon you’ll patch, or update the game in some way after it launches?

Jobe: We don’t have anything rigidly scheduled. Of course, like every developer, we’ve got a day-one patch that went through, and we’ll apply a few more bits of polish and refinement and fix a few defects that we wanted to make sure got resolved for the international launch. But as far as when we deploy our next big patch, we don’t have anything scheduled right now. Our approach to downloadable content (DLC) and big patches is a little bit different from most publishers. It’s kind of sad to say, but we are not going to have DLC available on the store before the game launches. We want to take a look at a couple of weeks’ worth of player feedback on the title before we look at what kind of DLC and before we look at what kind of updates we’re going to do in the next patch. We’ve got a few things cooking right now that we know are global issues that we want to address. But as far as DLC stuff goes, we want to see what’s going to best fill a potential void or desire in our community before we go ahead and start making it.

GamesBeat: What else is interesting to mention about some of the lessons here?

Jobe: There’s a lot of stuff that we learned internally, as far as our tools go. We went from Warhawk, which was a really fun multiplayer-only game, to Starhawk, which has a story-driven single-player component, and that shed a spotlight into some areas that we could do better at for our next round of technology. Our research and development team at LightBox Interactive has already started laying the foundation and high-level planning for our next engine.

I really look forward to better hardware speed. Not just for visuals, a slightly better pixel shader or slightly more visual performance. I think it could do the industry a lot of benefit. It could do studios a lot of benefit if we took some of that performance and used it to actually make the production tools faster and more forgiving and easier. At least I know that inside LightBox we would forego some bells and whistles if we could insure that we could iterate more quickly on a map or a layout or be able to get a couple more churns of content revision. I think that’s something we’re going to be focusing on a great deal in the future — just really making sure that we further break down the barriers between artists and designers and the game engine itself.

GamesBeat: I guess you’re going to have more lessons after the game launches.

Jobe: Absolutely. As I often say, it’s not like the olden days where the game is done and it’s on store shelves. You’re going to have hundreds of thousands of players playing it worldwide. Part of the development phase, which is the post-launch development, really just started. But I think we’re really well prepared, and the systems in Starhawk and the procedures that we tested in our beta test will be able to support the community for that post-launch support.

GamesBeat 2012 is VentureBeat’s fourth annual conference on disruption in the video game market. This year we’re calling on speakers from the hottest mobile, social, PC, and console companies to debate new ways to stay on pace with changing consumer tastes and platforms. Join 500+ execs, investors, analysts, entrepreneurs, and press as we explore the gaming industry’s latest trends and newest monetization opportunities. The event takes place July 10-11 in San Francisco, and you can get your early-bird tickets here.