Former Airbnb engineers simplify Mesos to manage data jobs in the cloud

SAN FRANCISCO — As companies store and analyze more data in clouds to improve products and adjust operations, management of the underlying infrastructure can become complex, and that infrastructure can sprawl out. An analytics tool runs on one group of virtual machines, while part of a web application runs on another group, and so on. Engineers at data-rich companies such as Twitter have bumped up against this sprawl problem and developed tools to solve it.

But the tools themselves aren’t always easy to implement, and a months-old startup named Mesosphere is looking to change that.

Mesosphere focuses on simplifying the deployment of Apache Mesos, an open-source system that figures out the best way to perform computing jobs in a data center in order to ensure that servers get used in the most efficient way.

One perk is that increasing utilization of computing and memory resources could lead to the use of fewer servers, which in turn could mean paying less money for power. Another benefit is that developers can focus more on building applications and less on making sure those applications scale out correctly.

Today, Mesosphere made it easier for companies to see what Mesos can do. It launched a free service called Elastic Mesos, which lets companies set up and run Mesos quickly and easily in the popular Amazon Web Services public cloud. The Mesosphere service is free.

To further increase adoption of Mesos, Mesosphere is publishing straightforward tutorials on how to use the tool. In addition, the company will post tutorials on using related frameworks that run in conjunction with Mesos: Chronos, for scheduling heavy-duty batch-processing jobs, and Marathon, for making sure less resource-intensive ongoing tasks continue to run even when certain hardware fails.

Looking to Mesosphere for help with Mesos makes sense. The startup’s founders, Florian Leibert and Tobi Knaup, used it often while they were at Airbnb. Leibert, who build out the company’s data-infrastructure team, wrote Chronos. He and Knaup made major contributions to Marathon. The two left Airbnb in May to form their startup, which now has eight employees.

“Ultimately, we really want to help build the ecosystem for Mesos, or around Mesos,” Leibert said.

He wasn’t as excited about how Mesosphere will make money as he was about just growing the Mesos user community. Which is understandable given that the company is looking to provide professional services for Mesos, Chronos, Marathon, and Hadoop on top of Mesos, according to a page on its website.

But thinking about services, or trying to figure out which vendors might want to compete with Mesosphere by supporting Mesos, is a little premature. Right now Mesosphere just wants to expand the Mesos user base, and Elastic Mesosphere looks like it could have that effect.

Editor’s note: Our upcoming DataBeat/Data Science Summit, Dec. 4-Dec. 5 in Redwood City, will focus on the most compelling opportunities for businesses in the area of big data analytics and data science. Register today!