Cloudera is building a new open-source storage engine called Kudu, sources say

Big data company Cloudera is preparing to launch major new open-source software for storing and serving lots of different kinds of unstructured data, with an eye toward challenging heavyweights in the database business, VentureBeat has learned.

The storage engine, Kudu, is meant as an alternative to the widely used Hadoop Distributed File System and the Hadoop-oriented HBase NoSQL database, borrowing characteristics from both, according to a copy of a slide deck on Kudu’s design goals that VentureBeat has obtained. The technology will be released as Apache-licensed open-source software, the slides show.

Cloudera has had one of its early employees leading a small team to work on Kudu for the past two years, and the company has begun pitching the software to customers before an open-source release at the end of this month, a source familiar with the matter told VentureBeat.

That source and others believe Kudu could present a new threat to data warehouses from Teradata and IBM’s PureData (formerly Netezza), and other vendors. It may also be used as a highly scalable in-memory database that can handle massively parallel processing (MPP) workloads, not unlike HP’s Vertica and VoltDB, the sources say. And one day Kudu — which works across multiple data centers with RAM and fast solid-state drives (SSDs) — could even play a part in backup and disaster recovery.

(Update on September 28: Cloudera today officially announced Kudu, days after declining to comment for this story. The full press release is here. A blog post on Kudu is here. A dedicated Kudu website is here. An academic paper on it is here. And finally, a public beta of Kudu is live now on GitHub.)

However Cloudera chooses to market Kudu, it’s clear that the software is a big step forward for the company, not only in the company’s efforts to outdo other Hadoop vendors, but also in its quest to become a prominent player in enterprise software.

Not that Cloudera is a nobody. It’s worth almost $5 billion, according to one recent estimate, it has considerable backing from Intel, and it’s been positioning itself as a competitor to much larger database companies, like IBM and Oracle. But the fact is, fellow Hadoop vendor Hortonworks has gained credibility after it went public last year, and Hadoop company MapR is still around, too.

Cloudera recently doubled down on the rising Apache Spark open-source big data processing framework, but Spark is something Cloudera has been working on for years. And a few months ago, Cloudera brought new Python capability to Hadoop, following its acquisition of DataPad last year. Those are important efforts, but Kudu is something entirely new, something that can give the company freshness as it grows toward an initial public offering.

So what is Kudu, then?

It’s “nearly as fast as raw HDFS for scans” and, at the same time, “nearly as fast as HBase for random access,” according to one slide from a presentation on Kudu’s design goals. But Kudu is not meant to be a drop-in substitute for HDFS or HBase. “There are still places where these systems will be optimal, and Cloudera will continue to support and invest in them,” a slide said.

Kudu could be used for time-series data, or real-time reporting, or model building, according to another slide.

And it’s important to note that Kudu isn’t a SQL query engine for pulling up specific data. Cloudera has Impala for that, and others have Hive for that. Kudu has an “early integration” with Impala, and Spark support is coming, according to a slide.

The Kudu application programming interface (API) works with Java — the common language of Hadoop — as well as C++. Kudu’s architecture allows for operation across sites, according to one slide. That makes it comparable to Google’s Spanner and the Spanner-inspired CockroachDB. That could make Kudu a great choice for big companies looking to store their big data around the world.

Is Kudu well adopted, though? No, not yet.

“Looking for beta customers,” a slide said.