Hortonworks gains ground as its Hadoop management tool gets adoption at Pivotal

Cloudera might lead in the business of selling a distribution of Hadoop open-source software for storing, processing and analyzing lots of different kinds of data. But competitors are gaining ground and consolidating.

Today there was more news on that front. Two companies in the Hadoop business, Hortonworks and Pivotal, announced that they will collaborate on the Hortonworks-backed Apache Ambari tool for managing Hadoop clusters. The news doesn’t mean Pivotal will drop its Command Center cluster management software, but it does point to the power of using existing open-source tools.

What’s more, it suggests not everyone will standardize around Cloudera’s tools, even though chip giant Intel has sunk multiple millions of dollars into Cloudera, giving Cloudera more visibility than Hortonworks and others in the Hadoop distribution business. Cloudera has the closed-source Cloudera Manager, but that’s not what Pivotal chose to standardize on.

The question worth asking is what comes next. Perhaps Hortonworks could become more widely used through additional and deeper partnerships. Perhaps Pivotal will put aside its variations on basic Hadoop tools and adopt more of the original open-source Hadoop projects. For instance, Pivotal could drop its Hawq query engine for pulling up data with standard SQL queries and throw its support behind the Apache Hive project. If that happens, Hortonworks could benefit.

Then again, Pivotal could keep opting for its own programs as it pushes Hadoop alongside other tools for dealing with data.

Here’s Gartner analyst Merv Adrian riffing in an email on the way things might play out:

As for Ambari, most of the megavendors in information management, who become more important as something like this goes mainstream, tend to have their own management consoles (e.g. Oracle’s OEM, IBM’s Tivoli.) As the market firms up, the likelihood is they figure out how to subsume Hadoop management into the stuff they have, and they may build or partner — or both. Pivotal didn’t have as much existing revenue to consider/leverage/protect so the decision made more sense for them.

In any case, Hortonworks does have more money to start influencing what other technology vendors do, after its $50 million investment from HP, following a $100 million round earlier this year.