What do you do when Google Analytics is too slow? Build a new damn database

SAN FRANCISCO — Imagine hundreds of billions of crumb-sized bits of data flying around in a room. Imagine they’re all somehow related, but they’re not yet sorted. You need to pull out specific crumbs, but it’s like a needles-only haystack, and finding the bits you need takes precision and time.

Lots of time.

New Relic CEO Lew Cirne banged his head against that problem for the majority of 2012. He finally had his “aha!” moment on Dec. 30 while on a family vacation in Tahoe: He needed to build a new kind of database, one designed to catch crumbs, and he knew exactly how to do it.

So he sent his family home without him and got his hands dirty.

“I spent a month up there coding, 8 a.m. to 10 p.m., just banging it out. I had a ‘Hello, world’ database up and running in two weeks,” Cirne said in an interview with VentureBeat this week.

“In the summer, I started working on the interface and making it so anyone can run these queries and get visualizations.”

The result is Rubicon, a big data-crunching piece of software that works like a warhorse and feels like a butterfly. By that we mean that using it is light and almost playful — it doesn’t require too much thought or effort, and the output is beautiful and easy to understand.

What’s most impressive, though, is that warhorse and the incredible speed and dexterity with which it handles the aforementioned hundreds of billions of data points. (And, by the way, the data have been gathered from New Relic’s customers and end users since June, so there’s plenty of info to sort.)

“We had to totally change how we collect data to do this,” said Cirne.

“If we measure 10,000 pageviews in one second, we can distill it down to 100 pieces of data and therefore do it at high scale. But you lose the ability to do these histograms and percentiles. To do this, we created an entirely new analytics product for New Relic.”

Rubicon, which is slated to ship in the first half of 2014, can break out anything – like, any damn thing — in terms of data. You want to see users of your paid product over the past three days from Japan on Android devices with two or more kids and a family history of eczema? Well, almost, but you get the point. You can quickly get deep into a very specific niche of your app’s userbase and drill down to look for code problems, business problems, interface problems, etc.

Here’s a screenshot. Notice the query in the top bar:

And everything Rubicon will also be available as an API. “I think third parties are going to want to integrate with Rubicon for business analytics,” Cirne said.

“Rubicon is helping New Relic service the business. The APM product is about helping developers keep the app running. … This will let developers really become heroes to the business.

“I’ve spent two years thinking on how to make business value based on the data we collect. And I’ve looked at all kinds of databases to do that, but I determined we needed to build one from the ground up to handle billions of time-stamped events … lots of little events where the strings are often repeated.”

In other words, sales people can see where the conversion funnel gets too tight. Marketing folks can see which ads are most effective in specific regions. Designers can finally figure out whether to use the red button or the blue button on that mobile website.

And of course, it’s all easy for the developer.

“We designed this for the common case of whipping through billions of data points,” said Cirne.

“It makes server malfunctions a nonevent. It’s built for resiliency and cloud hosting. We can do incredibly high speed on a small investment. What might cost a million from a pure-play analytics vendor will cost a tiny fraction of this at New Relic.”

We’ve got more coming up from FutureStack, New Relic’s first-ever developer conference being held today and tomorrow in San Francisco, so stay tuned.