EA’s Lucy Bradshaw acknowledges SimCity ‘stumbled out of the gate with our service’ (interview)

When a game sells too many copies at its launch, it’s usually not a bad thing. But with SimCity, it turned into a nightmare for Electronic Arts, because the city-building simulation was designed as an always-connected single-player game. When players overwhelmed the servers on the March 5 launch day, just about no one could play the game for which they had paid at least $60.

EA said that the point of making SimCity always-connected was so that players could build cities in regions with their friends and establish interactions between the cities, such as one city buying garbage services from another. But skeptical players figured the real reason was that EA wanted to stop game pirates. And that skepticism turned into anger after the launch, when the overload locked out hundreds of thousands of players around the world. EA had to slow down its marketing while it brought more servers to bear.

EA’s Maxis Label, which made the game, responded with an emergency server overhaul. Lucy Bradshaw, a senior vice president at Electronic Arts and the general manager of Maxis, issued regular blog updates on the server demand, which was double EA’s initial expectations. She eventually promised a free game download to SimCity players. [Update: And, in a blog post today, Bradshaw said that the always-connected feature did not come as a mandate from corporate].

Could EA have avoid this PR disaster? What lessons has EA learned? Was technology unable to handle a tsunami of demand?

Bradshaw candidly acknowledged that EA was “dumb” to underestimate demand. But she is grateful for the support of fans, who are playing the game in huge numbers without connection problems now, and she says her team has moved as fast as possible to fix the situation. Eleven days after the launch, EA has put in place enough computing hardware to handle the traffic, and it is beginning to sort through the lessons it has learned.

Bradshaw has been heads-down trying to rectify the situation, but we sent her some questions via email and she got back to us. Here are her answers.

GamesBeat: As technically and as clearly as possible, can you describe the problems you encountered with the launch?

Bradshaw: The problem was a combination of issues: an early surge in sales and playership and our servers not handling that load to our expectations. This left us with higher playership and lower capacity. We reacted quickly to improve the situation by adapting our server setup, improving it and deploying new servers.

GamesBeat: Maxis ran a beta test not long before the launch. Why didn’t you discover the problem during that test?

Bradshaw: The beta was helpful in addressing some of the early game issues, and it helped us improve the player journey to get into the game. However, we have experienced issues at launch that were not seen during either our beta or our load testing. One lesson is that we needed longer, more comprehensive beta tests. And another is that a sandbox game like SimCity is not entirely predictable, so our load scenarios needed to be better informed by this and test a wider variety of player variables.

GamesBeat: Also, why was the beta test so limited? Was it really just an hour?

Bradshaw: At that point we had completed numerous internal betas and server load testing and nothing indicated that we would have any issues. We leaned on load testing and the combination of results were positive. As I’ve noted, the mix of our approach to understanding the stresses our systems would undergo was not adequate. This and the early surge put us on our heels, but since then we have adapted our server configuration, enhanced performance of our servers and code, made client side improvements, and deployed and upgraded new and existing servers

GamesBeat: Would a longer beta test have revealed the problems?

Bradshaw: I do think that a different approach to our betas would have better informed our automated load testing and helped us to better analyze the results from both of these means of testing. So, yes, if we had it to over again, I would change how we approached our betas and I would have used this information to improve our load testing..

GamesBeat: Given the changes in server technology from the days of The Sims Online, what are the advantages of servers, and what are the lingering disadvantages?

Bradshaw: I don’t think of the challenge as one of servers today vs. yesterday necessarily. Basically, technology has changed since we put SimCity 4 out in 2003. We envisioned a design for this SimCity that takes full advantage of these technological advances. We see PCs today as inherently connected and social in use. The idea of creating a connected SimCity where players can connect within an region and the choices that they make in their city can affect those surrounding them. That felt like a huge leap for SimCity. It allowed us to think of how cities often specialize, how they have a symbiotic relationship. And we set about making a connected, social SimCity that reflects our dynamic, social, global world that we live in today. We think this is the right leap for the franchise, and players who are in the game today playing are experiencing this connected game and giving us great feedback.

So I see it as today’s technology enabled us to envision and create a SimCity that really innovates the game from a solo-city-in-isolation experience that was fun and creative to a connected, dynamic and social game that makes you think differently about your choices, gives you a familiar but new experience and challenges, and reflects our world of today.

GamesBeat: In other words, if you had a dream server, what would it be?

Bradshaw: Again, I think the challenge is not the servers. We configure our servers to meet the needs of our service. The challenge we face as game makers is that today’s gaming behavior has changed as have our ambitions. We came up with a SimCity vision for a connected and social game that is dynamically evolved by its players, with all playing in the same context. We fully embraced and believed in this vision. It’s an ambitious vision that takes advantage of the technology of today, high bandwidth connections, social play behavior, and cloud servers so we could have our game easily played form multiple devices and saves that are accessible anywhere.

This vision implied new design, new ways that our simulation needed to work to support multiplayer interactions and world events and challenges, ways to keep a persistent state of the world for players to interact in – each player’s city updating, trades and information passing between cities, etc. We have made both a new SimCity but also took on the challenge of a service and all of the systems that go with that. We clearly did not deliver on our service at launch and have worked to quickly rectify that situation.

GamesBeat: Why aren’t you able to offload a lot of the server demand to other providers, such as Amazon Web Services? Don’t you already use them or some similar outsourcing vendor for part of the demand?

Bradshaw: We do use AWS quite extensively on SimCity, and very satisfactorily. We basically had to reconfigure things to perform better. Our server availability is no longer a problem. Players are logging on and playing in the hundreds of thousands.

GamesBeat: How did user behavior exacerbate the problem of having too much demand and too few servers? What did you anticipate in the design of the game on that front, and what came as a surprise?

Bradshaw: What we did not expect out of the gate was the length of time each player played: seven hours average at launch. Which is cool but again meant we had players on for long amounts of time and this affected our peaks. The ways in which they navigated the game, which being a sandbox style game is not predictive. These just led to results that we did not experience in our Betas.

GamesBeat: When you fixed the problem and rebuilt servers, what did you actually do? How did you quickly redeploy them?

Bradshaw: A combination of optimizing our server architecture and improving our response times, deploying enhancements on both a series of new and the original servers and issuing a few critical client updates has greatly reduced the number of connection issues, wait time and improved the overall stability of the game. From the moment we realized we had load issues to the last deployment of our new and updated servers, it took us less than a week.

GamesBeat: How is the infrastructure partitioned on a worldwide basis? If I log into my home server, I can see my city. But if I log into another server, I can’t. Why is that?

Bradshaw: Each server is unique. If you log in to a specific server, and build and save a city, then you will need to return to that server to play that city again. You can save up to 10 regions on each server. We let you know when you choose a server if you have cities saved there already. We also make sure that if you have played on a specific server that we have space reserved for you, new players are guided to servers that have capacity. And we also default your server selection to the last server you played on.

GamesBeat: Six or seven days is a relatively short time to tackle a really big technology problem. How did you solve it quickly?

Bradshaw: It was heartbreaking to see something we poured so much into stumble in such a visible way, but on the other hand the support from across all of EA has been entirely heartwarming. The whole EA organization really rallied together behind our team at Maxis. We worked 24/7 in very long shifts to do what we needed to do to deliver on our promise to the SimCity fans. We’ve definitely got servers that are available, players are logging in, playing for great lengths of time. As of today, 5.6 million cities have been created and 12.9 million total hours of actual play time has taken place in the game. Wow!

GamesBeat: How did you determine the number of servers to have on launch day?

Bradshaw: We base these decisions on internal benchmarks, demand metrics and beta participation. What we underestimated was a huge surge in pre-orders within the last week and player behavior once in the game, for example the longer game sessions that I referenced before.

GamesBeat: If I so choose, I can start a private game and play completely by myself, which would basically turn SimCity into a single-player experience. You’ve given players that choice, so why take away the choice to play offline?

Bradshaw: Even in a private game where you own the entire region and have elected to play solo, we still do a great deal of activity on our servers to keep the state of your region up to date for all of the cities you play to have the same information. Events, trades, collaboration between cities to build Great Works, world challenges, global leaderboards, and global achievements are all systems that we have created to support this concept of a connected world where cities interact and affect one another. Additionally, we save your cities to the cloud. This allows you to play from any computer and access them easily.

GamesBeat: Was there a way to create multiplayer interactivity in the game without requiring the always-on connection?

Bradshaw: We set about a design vision that we fully embraced and it threaded through every decision we made. It was a huge effort to address the specifics of a new design, an all-new simulation that both gives you the most detailed simulation in the franchise yet but also the regional simulation, events, challenges, the social support to engage players, with chat, region wall, leaderboards, achievements and great works. The design we embarked upon relies upon that connection.

GamesBeat: Will EA reconsider doing “always connected” games in the future in favor of restoring offline single-player games?

Bradshaw: I can’t speak on behalf of other creative teams at EA. For Maxis, we will always try to create games that are new and exciting, redefining the simulation genres. Connected or unconnected isn’t a business decision for Maxis. It’s a creative decision that many of our players are enjoying right now. Our troubles exist because we stumbled out of the gate on our service — and for that we are sorry — but we are working hard to make it right. We stand by our design and players who are experiencing it are giving us great feedback.

GamesBeat: Did EA take anything from the server queue issues of Star Wars: The Old Republic into account with SimCity’s launch?

Bradshaw: We did end up mirroring some of their pre-launch preparation and activities. They had an early access program, but they also conducted a worldwide launch that let North Americans and Europeans into the game at the same time. We executed a similar plan with SimCity where we let players trickle in from around the world over the course of a week. Americans got in on March 5, Australians and Japanese on March 6, parts of Europe on March 7 and the U.K. on March 7. We felt this was an effective plan, but the issue was that our servers had issues from day one, and that snowballed throughout the course of the week and compounded our difficulties.

GamesBeat: How are the resources required for SimCity different?

Bradshaw: We have many of the same needs in terms of roles and expertise that we always have: great designers, amazing artists, and expert engineers. With the addition of our online services we add to our team DBA’s, online engineers, java scripters, experts in web services and more. We also take a different approach to launching a game in that we need a live ops team who monitors our service to assure it is performing well, that players are having a great time and to continually improving it as a service, while we also continue to support it with new content and capabilities.
At EA we have a host of teams who work on capabilities that we can draw upon as we make our games as well.

GamesBeat: Does EA feel it’s lost customers thanks to these issues?

Bradshaw: I like to think that if anything, we had a few potential players who just pressed pause before jumping in. But in reality, we’re seeing very high usage across the board, some really cool things that players are doing inside the game, and we love the passion and support we’re seeing from our community. People are loving the game, we knew they would love it, and we hope that even more keep jumping in to give to give it a go.

GamesBeat: How many people can play at once now?

Bradshaw: Our servers can now support several hundred thousand peak concurrent players, and we’ve reached peaks just in the first week that were literally 100 percent above our early estimations.

GamesBeat: How do you feel about the whole debate about changing review scores? While many critics enjoy the game, they’ve lowered their
ratings because of the outages. If the outages disappear, is it only fair to raise review scores?

Bradshaw: Of course I’m going to say yes, they should raise their scores :). The reviews are up the reviewers. The most we can do is build something we think is truly special, that will excite and exhilarate players. We know we did that here, I think many of the reviewers see that too. Look, it would have been easy for more reviewers to just tank us with the connectivity issues we had. But they didn’t. They want a chance to see the game in action, and I thank them for that, thank them for giving us a chance. I hope they’ll be so happy with what they see that they’ll want to give it the high score it deserves.