How The Weather Company survived a 1,000% traffic spike during Hurricane Sandy

Maintaining service during a massive and destructive storm can be difficult. But The Weather Company, a source for weather forecasts and data, found itself in the middle of its own storm during Hurricane Sandy, as panicked people pushed its website traffic levels literally 1,000 percent higher than normal.

Sandy’s timing couldn’t have been worse, as the Weather Company is in the early stages of a massive transformation, transitioning from 13 global data centers to managed hosting in Amazon’s cloud.

Perhaps it was lucky number 13, however, because in spite of a Boston data center being caught in the storm, the company’s 100-strong technical team maintained the site, apps, and services at a 99% uptime rating throughout the storm.

I talked to the Weather Company’s chief information officer, Bryson Koehler, about how they did it.

“We saw double the traffic that we’ve ever seen before on our digital platforms,” Koehler told me. “We had to handle 10X the traffic we’ve ever had before.”

The Weather Company delivers news on basically any medium you can imagine: the original TV show, smartphone apps, tablet apps, live streaming video online, print, radio, and APIs for partners all over the globe. On a typical day, the company serves the equivalent of about 40 million pageviews on its digital platforms, and it has previously seen service interruptions when the number approached only 120-140 million.

But on Monday, the company was serving 110 gigabytes of data per second, almost without a hiccup, not counting the livestreams — 170 thousand simultaneous streams of live video, pushed out over the Internet.

The question is: how?

While the Amazon Web Services integration is still mainly in the future, The Weather Company did have several hundred instances running at 100% capacity, Koehler said, telling me that the scalability was “exactly what you would expect,” as was the cost efficiency.

But the company had significant help from three additional critical partners.

“We relied very heavily on three partners that helped us deal with the sudden surge: Verizon, Akamai, Google were great throughout the storm.”

Akamai, of course, accelerates and distributes site load. Verizon manages some of the company’s data centers and much of its networking infrastructure, adding gigabytes of capacity literally in real time during the course of the hurricane. Google manages the livestreams, as The Weather Company delivers live video via a partnership with YouTube.

But the 13 data centers were key to the service’s resilience. Like any large organization that has been built over time and acquired other companies, Koehler said, there’s a mix of technologies.

“You end up with a good bunch of technologies that come together, but for the most part, we run Linux servers, Apache for web serving, Tomcat for our app server, and VMware for virtualization. MySQL is our core weather database.”

More than the technology, though, Koehler said it’s about the team. The Weather Company put together a 100-person hurricane team: technicians, system administrators, and networking specialists. “We were very prepared,” he said.

photo credit: cab234 via photopin cc