Starhawk developers learned from social games that “analytics is king” (interview)

It isn’t easy to move to a new city, launch a new studio, and create a new game at the same time. But LightBox Interactive has done just that, and now Sony is finally releasing LightBox’s new sci-fi western-style game, Starhawk.

The third-person shooter game debuts today on the PlayStation 3. The predecessor game, Warhawk, debuted on the PS 3 as a multiplayer-only title. But with Starhawk, LightBox has stepped up and created a single-player campaign along with multiplayer and cooperative play as well. The frenetic action game requires you to be a good shot and a strategist at the same time.

During the production, chief executive Dylan Jobe said, the company watched the game industry change and saw the rise of Zynga, which focused on learning from its social game analytics and immediately modifying its games as a result. Jobe said that even after today’s release, LightBox will have a big job ahead of it tuning the game to what the fans want. It will not only have to fix bugs but will also change the game and come out with new content to keep the fans coming back. Game design, it seems, is always changing and never done.

Here’s a transcript of our interview with Jobe.

GamesBeat: How does it feel to have gotten to this point now?

Jobe: It feels great. It’s kind of tricky when you release a multiplayer game like this. I felt the same way with Warhawk. There’s that good feeling of having it done, yet at the same time you know that when you release a multiplayer game, especially to the global community, there’s still a lot of work ahead. There’ll be tuning and fixes and all kinds of thing that we’ll have to address and take care of for the community. So in some respects it’s like we’re done, but in other respects, we’ve just started.

GamesBeat: Tell me about some of the development history here. And also about Lightbox and starting that up.

Jobe: Sure. We are a new studio. Starhawk will be our first game, even though I and my business partners have been working with Sony for many, many years. Even going back to the good old days of SingleTrac. Originally, I actually was not in the game industry at all. I worked as a product designer for AT&T and NCR, National Cash Register, of all places.

But I always really loved video games. So I eventually just thought, ‘Well, I keep making Quake mods and all these things, I want to get into the industry.’ I ended up leaving NCR and going to SingleTrac in Utah. I worked there for a little while and then worked at Incognito Entertainment, where I got to work on Twisted Metal Black and then produced and directed War of the Monsters, and then of course Warhawk, the PSN multiplayer game.

About the same time, Scott Campbell — who was at the time the president of Incognito — and Dave Jaffe wanted to start a new company, Eat Sleep Play. So they went off to create that new company. And I took 12 guys. We moved from Utah down to Austin and from there started LightBox and started developing Starhawk for Sony. We’ve been working on Starhawk for just over two and a half years.

GamesBeat: What were some of the hard parts about starting a company at the same time you’re starting your first game?

Jobe: That’s actually a good question. The hard thing is maintaining the production timeline, because you have milestones due to your publisher, yet at the same time you’re building your company. You have to contend with milestone deliveries, and you have to make sure that build gets out on time, but you also have to make sure that the actual build-out of your space gets done on time. We started doing a little bit of development while we were still in our small space in Salt Lake City. It was kind of a weird jumble, because we had some people still working in Salt Lake City, and we had a temporary space in very startup fashion, one of these rent-a-office places in Austin, while we were looking around for office space.

At the time, the economy was not super great. Credit was pretty much locked up. We were able to get LightBox started without having to take out any small business loans at all. It was tricky to find office space that could accommodate a new studio. That was one of the challenges, but one of the upsides, because it was such an economic downturn, we were able to get a really good price on some of our general contracting, furniture, some of the other costs that probably would have cost a bit more money had we started the company while the economy was doing well. So it was kind of a balance. Managing all that, working with our electricians and everything, and at the same time continuing to deliver monthly prototypes to Sony was definitely challenging. It felt like a strategy game, like a Sid Meier’s video game.

GamesBeat: It seems like it’s pretty tricky to get people to move for your company, too, down in Austin.

Jobe: It turned out way better than I thought it was going to be. So what we did was work with 12 people in Utah. I was very up front with them. I said, “You know, we’re going to be starting a new studio in Austin.” Because at the time we were still Sony employees, Incognito Entertainment was a wholly owned company of Sony’s. I wanted to make sure that I was very up front with all the new employees. I and the business partners ensured that we were able to give everyone a ton of notice, saying, “We’re going to start on this day, here are your new employee packages when you do become LightBox employees, let us know any of your thoughts.” We gave people a ton of fair warning and information.

And as it turned out, everyone moved down with us. It worked out great. So what we did was, we pulled our 12 people down from Utah, but at the same time we were doing remote hiring. So we were able to staff up with about an additional 20 headcount, by doing deferred hires. We were bringing people out to interview, having them come on board, letting them know what was going on, and then we set their start date a little bit later, about the time — and this was all very stressful — about the time that we got our studio done. The furniture was installed, the infrastructure was all in place. We had a bunch of new people start, for what amounted to a kind of overnight transition from about 12 to 30 people. I think we got lucky, actually. At the time there were some studios in Austin and in Dallas that had suffered some hardships. There were people who were looking, on the market, who were kind of fresh blood injected into the Austin scene, and we seemed to be an attractive place to start.

GamesBeat: So how many did you wind up with altogether?

Jobe: Right now we’re at 43. LightBox Interactive is a small team compared to other big shooters. Just like all game developers, we put in long hours.

GamesBeat: What would you say is the best sales pitch you’ve got for people who don’t know what Starhawk is about, who are considering getting into it?

Jobe: Weird as it sounds, one of the main reasons we designed Warhawk the way we did was a direct response to what we felt was happening in the shooter genre. We felt there was a stagnation. We felt that the shooter genre had just basically turned into a game of one-upsmanship when it comes to the linear consumption of showpiece moments, running down the hall, see a showpiece, shoot a guy in the face, run down the hall, over and over again. So we took a lot of risks with Starhawk, with our space Western vibe, the build mechanics, and the battle mechanics.

I think all of that being said, the high level sales pitch to someone who’s a shooter fan but not familiar with Starhawk is, ‘If you’re a shooter fan and you want to try something new, truly new, take a look at Starhawk. Because it’s not your average shooter, and it’s not your average shooter because of the build and battle system. Basically, instead of having you go through a series of linear missions, making weapon selections and cover selections, you’re actually creating your own cover because you can build your own choke points with connecting walls, or build your own towers, build your own bunkers, summon your own artificial intelligence (AI) allies. You can create Razorback 4-by-4s and other vehicles in garages that you create. So it really kind of blends a shooter and realtime strategy together in a way that still feels really fast and visceral.

GamesBeat: My joke for you is, maybe you ought to try to market Starhawk to the people who played Rage.

Jobe: Ha ha ha. Maybe?

GamesBeat: They didn’t have such a great multiplayer component to their game.

Jobe: Yeah. That’s true. I must admit, I was surprised at the car combat aspect.

GamesBeat: There’s some similarity, so people who liked that game might also like this too.

Jobe: We certainly put a lot of our effort into the multiplayer. One thing is for certain: We agree that in the contemporary shooter market, multiplayer matters. What keeps people playing the shooter games for thousands of hours is the sticky multiplayer gaming, the community tools, and all of that stuff. So we definitely, even in Starhawk, put an emphasis on multiplayer. But our single-player campaign is good. Some people thought it might just be a glorified multiplayer tutorial where you play capture the flag with bots. That’s not it at all. There’s a really cool story there about Emmett Graves in our space western universe. It’s an analog to the Gold Rush in the early American West, except that in our universe the human colonists are after what we call “blue gold,” these geysers of blue energy. That’s ultimately the MacGuffin, the catalyst for the conflict in the universe.

GamesBeat: How about the comparisons to Warhawk? What did you learn from that, and how have you made this different?

Jobe: We learned a lot from Warhawk. Warhawk was a pioneer in some respects, because if you remember back at that time, which feels like forever ago, the whole notion that you would be able to download a big-budget game was kind of crazy. People at the time thought, “Oh, it’s a game that you download so it must be a small game. It can’t be a big game, it can’t be fun, I can’t be playing it for hundreds of hours.”

But in fact that was totally not the case. And now when you think about downloading a game, whether it be off Xbox Live Arcade or Steam or whatever, you don’t even think twice about it. We learned a lot of stuff on the multiplayer side in Warhawk. But I think, in many respects, one of the key multiplayer learnings was that single-player really does matter. Above and beyond the desire to steep players in the universe and the fiction and all of that stuff. It actually gives players a very safe place to get acclimated to the game systems.

Because adversarial multiplayer is an unforgiving place. Making sure that single-player is a good compelling experience, but also making sure that it gives them an opportunity to understand the game and get good with it before they go online, ended up being key. In Warhawk of course we didn’t have a single-player campaign, so players were essentially thrown to the wolves right off the bat, which made the experience very unforgiving. For those players who kind of stumbled through it, they were able to find some really exciting multiplayer experiences, but that learning curve was really steep in Warhawk. I think that’s one of the things that we changed significantly in Starhawk.

One of the other big changes that we did, to help make more informed design decisions about balance, was putting in place a really massively robust backend system for game analytics. We’re able to track far more statistics than we ever could. We’re a game studio in Austin, right? In the heart of a lot of social gaming and indie games and Facebook apps and all of that. Analytics is king. With it, you can make some really amazing business decisions, and also really smart, well-informed design decisions. We invested a lot of time putting in place a great telemetry system in Starhawk. We track every single event for every player in every game on the planet every single day. We know what skill you activated, when, how long you had that skill activated, when you switched.

We know every kill that every player made, where they were, how they made it. We know how many structures are being built, where they’re built. Down to very specific game design and tuning parameters, like, for every missile that is fired at another aircraft, we know how their controller was configured. Did they attempt to dodge it? If they attempted to dodge it, were they successful or were they not? So that we can make sure that the different control schemes have balance and are not providing advantages one way or another.

GamesBeat: That’s one of those interesting lessons that the social game companies taught everybody. Zynga and its analytics enables them to see something happen in the game and then change the game for the next day.

Jobe: Absolutely. And we learned that. It was actually incredibly eye-opening for me. As odd as this sounds, I was at an Austin Game Developers Online conference, and I really wanted to learn more about the analytic side, because I knew its direct application to shooters, especially a global online shooter, could be huge. I sat in a bunch of sessions with Nexon, Zynga, and a bunch of companies that were really steeped in backend analytics. Some of them had their own systems. Many of them kept referring to a piece of software called Tableau, which is a really great data visualizer. And I really didn’t know anything about it at the time, but I knew that this was something that I needed to know more about, that we needed to integrate with Starhawk.

I remember coming back to the studio, sitting down with our systems designer and saying, ‘You know what? This software looks like something we need to use. It’s going to be a great way for us to visualize all kinds of design stuff. Get on it.’ And that was really the start of it. Tableau software worked out great for us. Of course we use a very involved custom Sony backend, but Tableau software is a really great visualizer, and I would highly recommend that for any game studio. It’s great. And we did exactly what you said regarding the learnings that we’ve all gotten from social games.

We did that, literally on a daily basis, during our international Starhawk beta. We would look at the data, see what was happening, we would look at the forums and Twitter and Facebook. Of course, you have the passionate internet fans. They may have had just one bad game, so they immediately think that whatever weapon they were using at the time was underpowered. But you compare the passion of their response with the cold clarity of the objective data to steer the design decisions and the tuning of the game.

Math is the universal language, right? While we have a ton of players in North America who we communicate with quite frequently and clearly in English, we have players everywhere. So we were able to get essentially playtest data from all nations, whether or not we spoke their language, whether or not we could read their forum posts. It was really interesting stuff that came out of that data. Stuff that was surprising to us. I made a presentation about this at SXSW this year. It blew my mind to see that Russians playing the Starhawk beta accounted for the most proximity mine kills. That was a group of people on earth that love to plant proximity mines the most, and they accounted for the most proximity mine kills. Some of the best players on the planet, in fact the biggest percentage of great players, even though it was relatively small, but they were so good, were from Greece.

GamesBeat: I guess there aren’t many jobs in Greece? People have some time to get pretty good at games.

Jobe: Oh my god, yeah, I guess so. I actually made a joke about it, which was probably off-color at the time. I said, ‘I guess when your economy and government’s in free-fall, you get to play a lot of games.’

GamesBeat: Are you able to take some of these learnings and actually turn them around in a day? Or because it’s a console game, does it take you longer to do?

Jobe: It depends on what system the change is in. One of the things that we were really inspired by was a system that Naughty Dog fielded way back on Uncharted 2. We were captivated by this. They were able to make some relatively quick game balance changes. They had exposed some weapon systems in the game that they could make adjustments to, and I was captivated by that. So much so that we developed, at a very foundational level across the entirety of our engine in Starhawk, the notion of a “hotfix” system. Which was very much inspired by Zynga A/B testing and what Naughty Dog did with Uncharted 2. It’s basically a way for us to take all of the parameters that would otherwise be encoded into the executable itself, extract as much of it as possible, including art assets like texture maps and the placement of pickups and bunkers and walls and all that stuff, and extract that out into a layer that can, for all intents and purposes, bypass Sony’s potentially slower patching system, so that we could take advantage of the analytics.

And we did that quite frequently in the beta. Even simple things. I would see on Twitter a picture someone would take with the iPhone where he found a hole in one of the levels. So I immediately take that picture back and say, ‘Art team, find where this hole is in this picture and create a hotfix and we’re going to push it tonight.’ And we could do it. So that’s like a defect fix. But at the same time, with the analytics, we could do things like, ‘Wow, the auto-turrets are getting way too many kills. Over the course of a couple of days, let’s start to spin that back down a little bit and look at the trend of total kills that all players on the planet are getting as a function of auto-turrets.’ So we would look at the trend line, almost like judging a stock, and we could do tuning to the game to bring that down. One of the companies that does a phenomenal job using analytics in game tuning is Riot Games, with League of Legends.

GamesBeat: It sounds like this is a relatively new learning that sort of started in social and is becoming useful on the consoles.

Jobe: I think so. With other big studios, I’m sure they were probably ahead of us. But I can say, at least for us, it was a really big thing, and I know that Sony has requested of us additional examples of how we use the analytics system, how we plot different things on maps, and how we glean that data. And so I think there’s still a lot more to be done. It’s really interesting. One of the things that we will probably be rolling out in a patch in Starhawk after launch is even further analytics. Because after we got the system done and the game was essentially out for replication, manufacturing, we ended up creating a much bigger list of development analytics that we’re really interested in, and that we didn’t have time to put in. And they’re just really kind of geeky engineering stuff, like ‘How many times are people doing matchmaking versus a created game list? How long is the average time spent matchmaking by region?’ These are some really important statistics that we think we could benefit from for tuning our backend infrastructure for the global server set. But those are analytics that we’ll have to add further down the road.

GamesBeat: It sounds like these are also some lessons for the console makers and what they could do with future consoles and the updating of games and things like that.

Jobe: Oh, yeah. I mean, let’s be honest. The frequency of updates that players are expecting from contemporary games is a frequency that is often much faster than most publishers can support. And, quite frankly, if you look at what the next generation of hardware has in store for us, what are systems, at the hardware or the operating system level, that could have really great analytics in place? Whether it be in an API provided by the hardware vendor or whatever. There’s a lot of opportunities there, I think, for expansion. And I think in some respects we have to catch up a little bit to what we see coming from some of the social gaming pioneers.

GamesBeat: Do you already know some of this, about how soon you’ll patch, or update the game in some way after it launches?

Jobe: We don’t have anything rigidly scheduled. Of course, like every developer, we’ve got a day-one patch that went through, and we’ll apply a few more bits of polish and refinement and fix a few defects that we wanted to make sure got resolved for the international launch. But as far as when we deploy our next big patch, we don’t have anything scheduled right now. Our approach to downloadable content (DLC) and big patches is a little bit different from most publishers. It’s kind of sad to say, but we are not going to have DLC available on the store before the game launches. We want to take a look at a couple of weeks’ worth of player feedback on the title before we look at what kind of DLC and before we look at what kind of updates we’re going to do in the next patch. We’ve got a few things cooking right now that we know are global issues that we want to address. But as far as DLC stuff goes, we want to see what’s going to best fill a potential void or desire in our community before we go ahead and start making it.

GamesBeat: What else is interesting to mention about some of the lessons here?

Jobe: There’s a lot of stuff that we learned internally, as far as our tools go. We went from Warhawk, which was a really fun multiplayer-only game, to Starhawk, which has a story-driven single-player component, and that shed a spotlight into some areas that we could do better at for our next round of technology. Our research and development team at LightBox Interactive has already started laying the foundation and high-level planning for our next engine.

I really look forward to better hardware speed. Not just for visuals, a slightly better pixel shader or slightly more visual performance. I think it could do the industry a lot of benefit. It could do studios a lot of benefit if we took some of that performance and used it to actually make the production tools faster and more forgiving and easier. At least I know that inside LightBox we would forego some bells and whistles if we could insure that we could iterate more quickly on a map or a layout or be able to get a couple more churns of content revision. I think that’s something we’re going to be focusing on a great deal in the future — just really making sure that we further break down the barriers between artists and designers and the game engine itself.

GamesBeat: I guess you’re going to have more lessons after the game launches.

Jobe: Absolutely. As I often say, it’s not like the olden days where the game is done and it’s on store shelves. You’re going to have hundreds of thousands of players playing it worldwide. Part of the development phase, which is the post-launch development, really just started. But I think we’re really well prepared, and the systems in Starhawk and the procedures that we tested in our beta test will be able to support the community for that post-launch support.

GamesBeat 2012 is VentureBeat’s fourth annual conference on disruption in the video game market. This year we’re calling on speakers from the hottest mobile, social, PC, and console companies to debate new ways to stay on pace with changing consumer tastes and platforms. Join 500+ execs, investors, analysts, entrepreneurs, and press as we explore the gaming industry’s latest trends and newest monetization opportunities. The event takes place July 10-11 in San Francisco, and you can get your early-bird tickets here.