5 big data trends to watch out for in 2015

What a difference a year makes. At the end of 2013, the industry was still debating whether Hadoop and related big data technologies were going to become mainstream or were just niche technologies for Internet companies.

A year later, the answer is clear — Hadoop is without question the foundation of the new data stack, the first of the Hadoop distributions (Hortonworks) is now a public company, with others sure to follow. This is putting a spotlight on the next layer up the stack — big data analytics — and the use cases that will be unlocked and transformed by collecting and connecting vast quantities of raw data and empowering business analysts with new capabilities. In 2015, we’ll see the impact of big data across almost every industry sector, and there will be a multitude of proof points that go beyond vague claims often seen today.

Here are five trends to watch for over the next 12 months:

1. Big data analytics fills security void

Big data analytics will emerge as a powerful complement to traditional security software.

Massive breaches at Target and Home Depot in 2014 highlighted the degree to which the dizzying array and vast quantities of data that corporations collect has quickly outgrown the capabilities of traditional security software. The increased complexity and varieties and volumes of data enterprises currently manage make it nearly impossible to analyze everything, especially given the fact that most security platforms don’t talk to each other, and the cost of processing huge volumes and varieties of data is both cost and time-prohibitive.

But big data analytics can be used to identify suspicious behaviors and patterns that might indicate a network — even one protected by disparate security solutions that don’t talk to each other — is under attack. Over the past six months, CISOs have started to look at big data analytics as an important new tool that will enable their security infrastructure to identify and remediate “low and slow” advanced-persistent-threat attacks that might otherwise be undetectable. Look for this trend to accelerate and expand to other sectors in 2015.

2. Faking it with big data will no longer cut it

We’re in a historic technology replacement cycle that has only just begun to affect the enterprise sector. Many experienced vendors are trying to sell rebranded technologies that cannot handle the volume or variety of data that new platforms and technologies can. While it’s taken a few years for big data analytics to emerge as a fundamentally distinct solution from traditional business intelligence and data warehouse technologies that came before it, the next year will see less buyer confusion about what characterizes next-generation analytics. Buyers will come to understand what is possible with a modern stack and what to look for when choosing a solution.

3. The Internet of things comes around to big data

The hundreds of device makers building the Internet of things will get serious about big data — or get left out in the cold if they don’t.

Data generated by the household devices, wearables and industrial devices that comprise today’s Internet of things is growing exponentially, but almost all of it sits in silos created by different device makers. The real value is in connecting the dots across these silos to understand patterns of behavior, interaction and failure modes. This is critical to understanding how devices and services are operating, understanding the flow of communication that could constitute security threats, and weaving together dependencies and interdependent failure modes. Using Hadoop as a data lake to store Internet of things data is a required first step, and a range of analytical approaches follow.

For example, Opower uses big data analytics to help customers at more than 100 utility companies better understand their usage patterns and suggests how they could save money by consuming less power.

In 2015, look for the discussion to shift to how the device makers are turning to data analytics to help the nascent Internet of things market reach its full potential.

4. Companies reorganize around big data analytics

According to a recent Gartner report, 85 percent of Fortune 500 companies are ill prepared to exploit big data. Companies of all stripes and sizes will increasingly adopt a data centric approach as they set strategy, encourage collaboration among colleagues, and interact with customers.

A few years ago, big data tools were available only to corporate giants that could invest in proprietary and largely inflexible infrastructure. The advent of the Hadoop data lake is democratizing — and quickly accelerating — the game, forcing companies of all sizes to react if they want to remain competitive. In 2015, big data’s ability to connect dots in ways never before possible will prompt companies to rethink the way their employees collaborate and use data to identify and quickly adapt to opportunities and challenges.

For example, Riot Games, the 1,500-person gaming company most famous for League of Legends, has a task force of employees who are responsible for making their company more data-driven. They use big data to analyze behavioral data of 30 million users to find bugs in the software and to make data-driven decisions on which new features to introduce into the gaming environment.

5. The winning architecture emerges, leading to vendor consolidation

After more than six years of development, testing and early-stage roll outs, the big data landscape is ripe for a shake out.

Countless startups have either taken the wrong architectural road or are focused on niche products that cannot meet their customers’ overall needs. In 2015, look for some of these small players to fail, or become acquisition targets as larger vendors look to cobble together end-to-end solutions.

With big data indicators all pointing north, we’ll see enterprises making bigger strategic bets in 2015 as they look to vendors that offer end-to-end solutions built to handle massive amounts of disparate data without requiring data scientists. Look for both Cloudera and MapR to go public in 2015, and plenty of discussion about those companies (both public and private) that can thrive in this new world of big data — and those that are on a fast decline. It’s shakeout time.

Ben Werther launched Platfora to transform the way businesses use big data analytics. Before founding Platfora, Ben was vice president of products for DataStax and head of product at Greenplum.