How Facebook’s cold storage preserves your old photos

Facebook has done a whole lot of work in order to keep your oldest and least popular photos neatly tucked away and ready to be served up whenever people want to look at them. Today Facebook is opening up about many of the ideas its infrastructure team has implemented in the two cold storage facilities it maintains at its data centers in Oregon and North Carolina.

People share 2 billion photos on Facebook’s services every day, so Facebook needs to store them efficiently. And Facebook has made improvements on hardware and software to deal with that scale.

Each big room, or data hall, inside the cold storage buildings, can currently handle 1 exabyte — a thousand petabytes, or a million terabytes, or a billion gigabytes, and every rack can take two petabytes, Facebook’s Krish Bandaru and Kestutis Patiejunas wrote in a blog post today.

Other web companies, like Google and Microsoft, have made extensive customizations on their infrastructure over the years, but Facebook has been the most transparent, launching the Open Compute Project to share its hardware designs with others in 2011. Facebook’s cold storage systems are one of many components of Open Compute.

But not every facet of the cold storage systems — which now “protect” hundreds of petabytes of data, according to the blog post — has been public until now.

Today’s blog post has several highlights:

• These facilities use just a quarter of power that traditional storage servers do.
• Facebook updated the firmware in the hard drive controller in order to make sure that only one drive gets turned on at a time.
• Facebook dropped the number of fans in each storage node from six to four.
• The number of power shelves decreased from three to one.
• The number of power supplies per shelf went down from seven to five.
• Facebook ran into a problem — the racks were too heavy for the system Facebook had to get them to the place where they needed to be. Here’s Bandaru and Patiejunas: “For example, one of our test production runs hit a complete standstill when we realized that the data center personnel simply could not move the racks. Since these racks were a modification of the OpenVault system, we used the same rack castors that allowed us to easily roll the racks into place. But the inclusion of 480 4 TB drives drove the weight to over 1,100 kg, effectively crushing the rubber wheels. This was the first time we’d ever deployed a rack that heavy, and it wasn’t something we’d originally baked into the development plan!”
• Facebook wanted to store less than two copies of any given piece of data while still preventing data loss. The company ultimately devised a “re-encoding service that could reoptimize the data to support the next generation of cheap storage media, whether it was more reliable or less so,” Bandaru and Patiejunas wrote. “As a result, we can now store the backups for 1GB of data in 1.4GB of space — in previous models, those backups would take up larger amounts of data, housed across multiple disks. This process creates data efficiency while greatly increasing the durability of what’s being stored.”
• Facebook also created a system for rebalancing data across hardware.

Read the full blog post to learn much more about Facebook’s cold storage facilities.