Twitter engineer joins Mesosphere to push the open-source project he helped make

Inside an academic lab, Ben Hindman created key parts of Apache Mesos, an open-source tool for efficiently running lots of applications in data centers. He found himself at Twitter for four years, putting the system in place. But now he is doing what he arguably was destined to do: working on the project for a living to make it a standard everywhere.

Last year a startup called Mesosphere launched, with the goal of promoting and running a business based on the Mesos project. Now Hindman has left Twitter, where he was a member of the social networking company’s technical staff, to lead the design of the operating system Mesosphere is building for data centers, with Apache Mesos as the core kernel.

Hindman’s move shows that more of the brainpower behind Mesos are ready to advance it. Which makes Mesosphere worth watching even more than it was a few months ago, alongside startups with other game-changing data-center technology, like CoreOS and Docker.

Hindman sees value in the work those other startups are doing. After all, Mesos now works with Docker’s container technology, and CoreOS can do the hard work of upgrading an operating-system kernel in a way that’s complementary to Mesos, Hindman told VentureBeat.

“It’s like we’re defining and describing the future of building software for most of the people out there that are working in a lot of these (big web) companies,” said Hindman, who described himself as a founding advisor to the startup. “That’s a blessing. That’s, like, a gift. It’s so much fun to do that.”

Hindman got to know Flo Leibert, Mesosphere’s chief executive, in high school in Colorado. Leibert was a foreign-exchange student from Germany, and the two became great friends. They’ve been in touch ever since.

Once Leibert heard about the research project Hindman was working on as a Ph.D. student at the University of California, Berkeley — simply put, Mesos draws on all available computing resources and runs a variety of programs in the most efficient way — Leibert and Abdur Chowdhury, then Twitter’s chief scientist, got Hindman to give a talk about the work at Twitter.

“They were like, ‘You gotta come to Twitter,'” Hindman said. “‘You gotta help us use something like Mesos. This is amazing. This is something we’d be really interested in growing.'”

That was in 2010. Sure enough, that same year Hindman joined Twitter.

“It was a lot of work to take something that was a research prototype really and turn it into a production-quality piece of software that we could run at Twitter,” Hindman said. But he and Leibert managed to do it, with the help of a few other Twitter engineers along the way.

Small programs went into production on top of Mesos in 2011. And each year after that, it became more of a standard. Last year, a Twitter engineering head declared that any piece of software going forward would have to run on Mesos. That, Hindman said, was “a huge deal.”

Also last year, Leibert and Tobias Knaup left Airbnb, where they had run applications atop Mesos and started Mesosphere in a nondescript office on the outskirts of San Francisco’s Mission neighborhood.

Since then, Mesosphere has taken on a $10.5 million round of funding and even made an acquisition. In essence, the startup is getting more mature. And now it’s time for Hindman to take his rightful place at the startup commercializing the software he helped create.

Sure, he’s leaving one of the top social companies in the world. But he sounded veritably thrilled to be joining up with his old friend and focusing his energy on the development and implementation of Mesos all over the place, not just at Twitter.

“I want to be a part of that,” Hindman said. “I want to be a big part of that. I want to participate in it, and no matter how cushy my life could have been, I don’t want to miss this opportunity, miss this chance.”