Hadoop looks more promising than ever after Intel’s big Cloudera deal

Hadoop’s software for storing, cleaning up, and analyzing lots of different kinds of data may not cost you much, but that doesn’t mean it’s second-rate.

It’s becoming more and more pervasive in IT stacks at financial services companies, health care providers, retailers, device manufacturers, and software startups. It’s not as commercially successful as Oracle database software, but it’s heading in that direction.

“I’d say every Fortune 5000 or Global 5000 enterprise already has a Hadoop cluster,” Tom Reilly, chief executive of Hadoop distribution vendor Cloudera, told VentureBeat last month.

And that was before news hit of Cloudera’s monstrous $900 million funding round, which included a hefty investment from Intel.

With deals like that, Hadoop is becoming more visible.

This week Barron’s reported on financial analysts’ predictions of acquisitions of companies dealing in big data and Hadoop, like Platfora, ClearStory Data, Talend, Think Big Analytics, and Paxata. And yesterday the Wall Street Journal took a look at Hadoop, too, concluding that these are still early days for the technology.

But the software has made significant progress since the open-source Apache Hadoop project started in 2006.

Big web shops like Facebook, LinkedIn, and Pinterest depend heavily on their Hadoop clusters. And that position should become more common in the future, partly because Hadoop can run fine and expand easily on commodity servers, without the need for expensive specialty hardware. And it can handle messy unstructured data, without filling up precious space inside data warehouses where different kinds of data can converge.

Which is why Hadoop could become the kind of thing companies look to when they want to decrease their expenditures — an ambition VentureBeat will be exploring at our DataBeat conference coming up next month in San Francisco.

The Journal quotes a Babson College IT and management professor as saying many companies only use Hadoop in proof-of-concept work, and the paper calls out security as one issue. And Pacific Crest Securities analysts estimate that the Hadoop market has just 500-700 customers, Barron’s reports.

But, if anything, that spells opportunity for the software.

Engineers have been contributing code under open-source licenses to bolster authorization, authentication, and auditing capabilities for Hadoop, and they’ve made it possible to operate multiple analysis tools with a single cluster of data center infrastructure.

Hadoop could represent an excellent choice for analysis of data from wearable devices and other gadgets in the burgeoning Internet of things market.

And as more companies sign up for deals with Hadoop vendors like Pivotal, Hortonworks, MapR, and Cloudera, more money will become available for further Hadoop development.

Plus, Intel’s recent deal could inspire other top technology vendors to consider investing in or buying Hadoop companies.

In essence, then, the Hadoop business is well worth following. And companies that enable analytics of data sitting in Hadoop should be in for further growth, too.