The elephant in the room: Are you ready to face big data?

Our use of social media, smart devices and the Internet all create huge volumes of data. Behind the scenes, machine-to-machine interactions, sensors, recommendation engines and APIs drive a proliferation of information. Much of this data — in some studies as much as 80 percent — has little to no structure and much of it is generated at machine speed; at high velocity. This kind of “big data” — extremely large data sets that have little to no structure and are growing every minute — is hard for databases and analysis tools to handle just in time. But it also contains a wealth of valuable information waiting to be found.

Often, the clues that lie in big data can be the key to an enterprise’s future success, offering insights to optimize supply chains, uncover consumer behavior patterns, identify traffic and energy patterns, and much more. Rather than be a burden, all of this newfound knowledge can allow businesses to make effective long-term and real-time business decisions.

How can businesses grapple with the task of not only managing this data explosion but more importantly, extracting value from it and turning it into an asset? How do you remove the noise from this high velocity, highly variable data and discover key insights while they are still relevant?

That’s a question I and and other panelists broached at VentureBeat’s recent CloudBeat session on “The New Social Layer.” The session examined how companies are using social media to gain insight into what customers think.

Those of us on the panel reached a surprising agreement: Much of the big data tools and technology needed for businesses to begin exploring this data is already in place. So why are so few companies making use of this data? What they’re missing is the right culture and people.

What struck me about this agreement was that we had executives from a startup, a major media company, an SMB and a Fortune 500 firm all agreeing on this catalyst to succeed with big data.

Organizations will always need individuals with strong traditional data management skills, but today business and IT staff need to collaborate in order for enterprises and SMBs to evolve. The individuals who will be highly sought by organizations will be people who can serve as change agents, pushing organizational collaboration and information innovation. I have been speaking extensively this year about the need for data scientists and big data skills and plan on continuing to do so.

At IBM, part of my job is to help develop new big data products and partner with customers to figure out the cultural shift needed. One company I’ve worked with, Vestas Wind Systems, a Danish energy company, is using our big data analytics technology to determine the best locations for wind turbine placement. Vestas is able to analyze everything from weather reports, tidal phases and satellite images to geospatial and sensor data, and to use that analysis to select locations that will generate optimal return on investment. This type of analysis used to take weeks but can now be done in less than an hour. Once the turbine has been sited and assembled, engineers can use these big data solutions to predict turbine performance and determine the best times to schedule maintenance.

That’s just one example of what is possible today. In the future, advanced and predictive analytics will help business users gain powerful insights into data so they can forecast future scenarios. One form of predictive analytics currently underway is analyzing consumer sentiment, a new phenomenon where companies can mine the countless insights gained from streaming human interactions across social networks and the Internet to establish consumer behavior or corporate best practices.

Dutch entertainment company RTL Nederland, for example, is using predictive analytics to mine viewer sentiment and feedback from blogs and social media sites, allowing the company to make real-time adjustments to improve viewer satisfaction for shows like “The X Factor” and “So You Think You Can Dance.” For “The X Factor” alone, RTL analyzed over 70,000 online conversations, enabling it to address viewer likes and dislikes so it could effectively tweak its programming.

Lastly, whether we were aware of it at the time or not, many of us witnessed big data analytics in action on the TV show Jeopardy! this past February, when IBM’s Watson, faced off against Jeopardy champions Brad Rutter and Ken Jennings. Zipping through category after category (with a few stumbles along the way), Watson demonstrated its ability to search roughly 200 million pages of text and buzz in once it reached a “confidence threshold.” Being able to understand Natural Language – the way humans speak to each other – including puns, double entendres and pop culture references, was a groundbreaking achievement for technology. Being able to do it all in about three seconds blew the human competition away. The reach of this exciting new big data analytics technology will be broad and will certainly play a role in many areas of our future.

Yes, the amount of big data that exists can be intimidating, but it’s exciting to envision the countless possibilities it can achieve for businesses. In order to capitalize on big data, businesses need to recast their views of their data problem. Instead of looking for quick fixes to handle the vast amounts of incoming data, they should look at it as an opportunity to make a significant impact on their business. If they have people on staff dedicated to determining the types of insights they’d like to uncover from their data, the technology exists to help them find it.

Anjul Bhambhri is vice president of Big Data Products at IBM.

[Image via Eric Isselée/Shutterstock]