LinkedIn open-sources Ambry, an object store for media files

LinkedIn today announced that it has open-sourced Ambry, a piece of software it built to store and serve up media files like photos, videos, and PDFs. The system is available on GitHub under an open source Apache license.

LinkedIn previously relied on a complex architecture involving closed source technology that was not cheap to scale, even as user numbers and data have both kept increasing. It wasn’t easy to expand, either.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":1952306,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,cloud,dev,","session":"C"}']

“This convinced us that we needed to find a better approach,” LinkedIn engineering manager Sriram Subramanian wrote in a blog post. “We took it as an opportunity to look at different end-to-end solutions. We looked at many possible solutions including distributed file systems, storage appliances, cloud, and an in-house implementation. We thought deeply about our design goals, understood the tradeoffs and were convinced to build an in-house solution that fit our needs better.”

The system has been in production for more than a year, wrote Subramanian, who has previously worked on Kafka and Samza open source software during his time at LinkedIn.

Of course, other big technology companies, like Facebook and Google, regularly open-source different types of software. But object storage isn’t the biggest area of innovation in the open source world.

Many popular Web services, like Airbnb, Pinterest, and Spotify, look to cloud services like Amazon Web Services’ (AWS) S3 for object storage. It’s not trivial. But LinkedIn is more like Facebook and Google in the sense that it operates its own infrastructure for its services. Hence the need for full software that can operate across multiple LinkedIn data centers. Of course, there are representational state transfer (REST) application programming interfaces (APIs) for working with Ambry, which is written completely in Java.

The system is complex. For a full rundown of its functionality, check out the full blog post and the Ambry wiki.