Big data, what have you done for me lately?

In gearing up to attend this week’s DataBeat, I found myself thinking about what I want out of the conference. I’ve dreamed about being able to analyze all the data that’s relevant to my company and what that could do for me as a CMO.

Imagine the insights: how customers use a product, how they consume content, and how social media data and brand sentiment come together to tell hidden stories. Imagine connecting all the data around the deals you’ve closed, with all the contacts in your database, and immediately seeing who looks like a possible customer? Or – for those of you that say you don’t need marketing (by which you probably mean advertising and are confusing the two terms) – imagine the ability to create emails and content that are targeted so accurately that people find the material helpful vs. spammy, and actually want more?

There is so much talk about big data, and yet it is rare to see in companies that aren’t drowning in cash like Google, a business that can really get to the dream of N=ALL; a way to see all of the data in your organization and find connections and insights so fully unexpected that they radically change how you operate.

Why isn’t this dream more of a reality?

1. Too many incremental, disparate solutions

One reason is that there’s still such a mishmash of companies each contributing their small part to the big data puzzle, and nothing is pulling it all together in one solution. Storage, search, dashboards and graphs – both open source and proprietary – are just parts of the overall solution. But to a CMO like me, they don’t get me to my dream of answering all my questions about our business. And all of these insights are incredibly complex to stitch together.

Earlier in my career, before I (or anyone other than engineers) understood Hadoop, my company embarked on a Hadoop deployment. Our engineers told us “Yes, this will solve our analytics problem and we will be able to understand our users better.” We all got very excited. But, as we all now know, a Hadoop deployment doesn’t equate to insights. It’s only step one: store a ton of data. And only after you achieve this first step can you start looking at all the other software you need to deploy before you can actually get to the insights.

2. Siloed data has led to siloed thinking

Another reason is that business executives haven’t yet centralized their thinking about their business data. There are tools like Splunk to look at machine-generated data through log analysis, and tools like HootSuite that look at human-generated data by monitoring social media. And then there’s Marketo and Salesforce that focus on the silo of marketing and sales. But where is the centralized “data lake” that will allow for connections between these silos? Is it possible that there are more insights we could garner by connecting more events together?

3. Painfully slow time-to-insight

While Hadoop has solved so much of the scale problem for big data, it does not yet solve the dream of instant insight gratification that most of us have. How long does it take to analyze the data stored in Hadoop, and how easy is it for anyone from the company to review it and discern actionable insights? The true dream of big data is being able to learn something new and immediately change what you’re doing. And not just one person in an organization – everyone in a business should have insights into what’s happening and be able to take action right away. Imagine the competitive advantage of being able to do that.

So at DataBeat, I’m looking at how companies are harnessing their data. That’s why I’m so excited for Graham Tackley’s talk later this morning about what The Guardian has built with its internal tool “Ophan.” The Guardian team has actually achieved the dream of putting insights into the hands of all the people who can make decisions and take actions that have a massive impact on the business, from the C-Suite to the editors on the newsroom floor. The Guardian is simply a great example of where big data could and should go. And that’s what a CMO like me wants to discover.

Jen Grant is the CMO of Elasticsearch, an open source end-to-end search and analytics platform that delivers actionable insights in real time.