The 3 big problems in big data (hint: They all involve people)

Gurjeet Singh is a cofounder of Palo Alto-based big data analytics company Ayasdi. 

Analysts estimate that enterprises will spend $34 billion on big data investments in 2013. Clearly, we believe that data holds immense potential. Unfortunately, we are nowhere near ready to unlock its potential as most enterprises are simply applying the same thinking and technologies in newer packages to solve their problems. We most certainly have advanced our abilities to store and process data. What is holding us back are people problems.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":868230,"post_type":"guest","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,","session":"B"}']

There are three cultural issues that we must solve to unlock the latent potential of data: data silos, a lack of data scientists, and broken communication channels between data scientists and business users.

1. Data Silos

In most enterprises, the data generated by a functional area ends up being the property of that group. This leads to two problems. First, it’s difficult to get a “complete” view of the data. Consider all the silos and systems that hold data: CRM, ticketing, bug tracking, fulfillment, etc. Getting all the relevant systems to even talk to each other is a huge challenge. Second, there’s significant cultural dissonance within organizations. Typically, each group controlling a data silo ends up caring more about their power and place in a department rather than the success of the organization as a whole. Organizations need to pool their data to find the answers to and get a complete view of their data.

2. Data Scientists

A typical enterprise generally has 10x more IT employees than analysts or data scientists. The process of analysis starts with a line of business request. IT collects data from various databases and transfers it to data scientists. Large teams of data scientists are deployed who spend months (or sometimes years) querying the data. Hiring data scientists (with advanced background in statistics, computer science, and some functional expertise) to accelerate the process is difficult because people with these skills are extremely scarce. The demand and interest in data scientists is skyrocketing, as Google Trends can attest, while we are producing fewer of them. What we need is a new class of technologies that amplify the impact data scientists and allow more people to become data scientists.

3. Communication

Finally, the biggest hurdle to fully realize the potential of data science is the lack of communication between data scientists and business users. Said another way, the analytical gap between a data scientist and a business user is so wide that even communicating insights poses a problem. Anything that does not make intuitive sense is often regarded with skepticism, or not fully understood, by business users, which can lead to missed opportunities. Data scientists and business users need to align, work more closely together, and build trust to solve business problems.

Conclusion

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":868230,"post_type":"guest","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,","session":"B"}']

It is imperative that we solve these three problems. We need to:

1. Open up data to everyone and break down data silos,
2. Amplify the productivity of data scientists and do more through automation, and
3. Develop a more collaborative culture, which allow business users and data scientists to communicate more effectively.

The results will be worth it.

[Editor’s note: Gurjeet Singh is speaking on stage today at VentureBeat’s DataBeat conference in Redwood City. DataBeat runs today and tomorrow; it’s not too late to catch the rest of the conference.]