EA’s Lucy Bradshaw acknowledges SimCity ‘stumbled out of the gate with our service’ (interview)

When a game sells too many copies at its launch, it’s usually not a bad thing. But with SimCity, it turned into a nightmare for Electronic Arts, because the city-building simulation was designed as an always-connected single-player game. When players overwhelmed the servers on the March 5 launch day, just about no one could play the game for which they had paid at least $60.

EA said that the point of making SimCity always-connected was so that players could build cities in regions with their friends and establish interactions between the cities, such as one city buying garbage services from another. But skeptical players figured the real reason was that EA wanted to stop game pirates. And that skepticism turned into anger after the launch, when the overload locked out hundreds of thousands of players around the world. EA had to slow down its marketing while it brought more servers to bear.

EA’s Maxis Label, which made the game, responded with an emergency server overhaul. Lucy Bradshaw, a senior vice president at Electronic Arts and the general manager of Maxis, issued regular blog updates on the server demand, which was double EA’s initial expectations. She eventually promised a free game download to SimCity players. [Update: And, in a blog post today, Bradshaw said that the always-connected feature did not come as a mandate from corporate].

Could EA have avoid this PR disaster? What lessons has EA learned? Was technology unable to handle a tsunami of demand?

Bradshaw candidly acknowledged that EA was “dumb” to underestimate demand. But she is grateful for the support of fans, who are playing the game in huge numbers without connection problems now, and she says her team has moved as fast as possible to fix the situation. Eleven days after the launch, EA has put in place enough computing hardware to handle the traffic, and it is beginning to sort through the lessons it has learned.

Bradshaw has been heads-down trying to rectify the situation, but we sent her some questions via email and she got back to us. Here are her answers.

GamesBeat: As technically and as clearly as possible, can you describe the problems you encountered with the launch?

Bradshaw: The problem was a combination of issues: an early surge in sales and playership and our servers not handling that load to our expectations. This left us with higher playership and lower capacity. We reacted quickly to improve the situation by adapting our server setup, improving it and deploying new servers.

GamesBeat: Maxis ran a beta test not long before the launch. Why didn’t you discover the problem during that test?

Bradshaw: The beta was helpful in addressing some of the early game issues, and it helped us improve the player journey to get into the game. However, we have experienced issues at launch that were not seen during either our beta or our load testing. One lesson is that we needed longer, more comprehensive beta tests. And another is that a sandbox game like SimCity is not entirely predictable, so our load scenarios needed to be better informed by this and test a wider variety of player variables.

GamesBeat: Also, why was the beta test so limited? Was it really just an hour?

Bradshaw: At that point we had completed numerous internal betas and server load testing and nothing indicated that we would have any issues. We leaned on load testing and the combination of results were positive. As I’ve noted, the mix of our approach to understanding the stresses our systems would undergo was not adequate. This and the early surge put us on our heels, but since then we have adapted our server configuration, enhanced performance of our servers and code, made client side improvements, and deployed and upgraded new and existing servers

GamesBeat: Would a longer beta test have revealed the problems?

Bradshaw: I do think that a different approach to our betas would have better informed our automated load testing and helped us to better analyze the results from both of these means of testing. So, yes, if we had it to over again, I would change how we approached our betas and I would have used this information to improve our load testing..

GamesBeat: Given the changes in server technology from the days of The Sims Online, what are the advantages of servers, and what are the lingering disadvantages?

Bradshaw: I don’t think of the challenge as one of servers today vs. yesterday necessarily. Basically, technology has changed since we put SimCity 4 out in 2003. We envisioned a design for this SimCity that takes full advantage of these technological advances. We see PCs today as inherently connected and social in use. The idea of creating a connected SimCity where players can connect within an region and the choices that they make in their city can affect those surrounding them. That felt like a huge leap for SimCity. It allowed us to think of how cities often specialize, how they have a symbiotic relationship. And we set about making a connected, social SimCity that reflects our dynamic, social, global world that we live in today. We think this is the right leap for the franchise, and players who are in the game today playing are experiencing this connected game and giving us great feedback.

So I see it as today’s technology enabled us to envision and create a SimCity that really innovates the game from a solo-city-in-isolation experience that was fun and creative to a connected, dynamic and social game that makes you think differently about your choices, gives you a familiar but new experience and challenges, and reflects our world of today.

GamesBeat: In other words, if you had a dream server, what would it be?

Bradshaw: Again, I think the challenge is not the servers. We configure our servers to meet the needs of our service. The challenge we face as game makers is that today’s gaming behavior has changed as have our ambitions. We came up with a SimCity vision for a connected and social game that is dynamically evolved by its players, with all playing in the same context. We fully embraced and believed in this vision. It’s an ambitious vision that takes advantage of the technology of today, high bandwidth connections, social play behavior, and cloud servers so we could have our game easily played form multiple devices and saves that are accessible anywhere.

This vision implied new design, new ways that our simulation needed to work to support multiplayer interactions and world events and challenges, ways to keep a persistent state of the world for players to interact in – each player’s city updating, trades and information passing between cities, etc. We have made both a new SimCity but also took on the challenge of a service and all of the systems that go with that. We clearly did not deliver on our service at launch and have worked to quickly rectify that situation.

GamesBeat: Why aren’t you able to offload a lot of the server demand to other providers, such as Amazon Web Services? Don’t you already use them or some similar outsourcing vendor for part of the demand?

Bradshaw: We do use AWS quite extensively on SimCity, and very satisfactorily. We basically had to reconfigure things to perform better. Our server availability is no longer a problem. Players are logging on and playing in the hundreds of thousands.

GamesBeat: How did user behavior exacerbate the problem of having too much demand and too few servers? What did you anticipate in the design of the game on that front, and what came as a surprise?

Bradshaw: What we did not expect out of the gate was the length of time each player played: seven hours average at launch. Which is cool but again meant we had players on for long amounts of time and this affected our peaks. The ways in which they navigated the game, which being a sandbox style game is not predictive. These just led to results that we did not experience in our Betas.

GamesBeat: When you fixed the problem and rebuilt servers, what did you actually do? How did you quickly redeploy them?

Bradshaw: A combination of optimizing our server architecture and improving our response times, deploying enhancements on both a series of new and the original servers and issuing a few critical client updates has greatly reduced the number of connection issues, wait time and improved the overall stability of the game. From the moment we realized we had load issues to the last deployment of our new and updated servers, it took us less than a week.

GamesBeat: How is the infrastructure partitioned on a worldwide basis? If I log into my home server, I can see my city. But if I log into another server, I can’t. Why is that?

Bradshaw: Each server is unique. If you log in to a specific server, and build and save a city, then you will need to return to that server to play that city again. You can save up to 10 regions on each server. We let you know when you choose a server if you have cities saved there already. We also make sure that if you have played on a specific server that we have space reserved for you, new players are guided to servers that have capacity. And we also default your server selection to the last server you played on.

GamesBeat: Six or seven days is a relatively short time to tackle a really big technology problem. How did you solve it quickly?

Bradshaw: It was heartbreaking to see something we poured so much into stumble in such a visible way, but on the other hand the support from across all of EA has been entirely heartwarming. The whole EA organization really rallied together behind our team at Maxis. We worked 24/7 in very long shifts to do what we needed to do to deliver on our promise to the SimCity fans. We’ve definitely got servers that are available, players are logging in, playing for great lengths of time. As of today, 5.6 million cities have been created and 12.9 million total hours of actual play time has taken place in the game. Wow!