The 5 stages of big data grief

Big data got glamorous in 2012, and vendors rushed to slap the words on their products. Project managers begged to be assigned to big data projects. Investors and board members asked management teams what their big data strategy was. It was a big data lovefest. “What Would Big Data Do?” bumper stickers were seen on Teslas in the Valley.

But then a funny thing happened: A backlash erupted. Negative press started to exceed positive press. Pundits questioned the value of big data. We had entered the stage Gartner likes to call “The Trough of Disillusionment.” Turns out, most people didn’t really understand what big data was, much less its value. Good business cases for big data projects were elusive, and in some enterprises, big data projects became something to be avoided.

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":888778,"post_type":"guest","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,","session":"C"}']

You may have heard of the Five Stages of Grief, a popular psychological theory positing that when a person is faced with or recovering from an extremely traumatic event, he or she will experience a series of emotional stages: Denial, Anger, Bargaining, Depression, and Acceptance. In our work helping Fortune 500 companies with advanced analytics work, we are seeing this same phenomenon. Executives who once so exuberantly embraced the concept of big data are shying away from it, and distancing themselves from big data projects. Executive sponsors have become harder to reach.

So in that spirit, I present to you the Five Stages of Big Data Grief. If you recognize yourself or your organization in any of these five stages, you may already be well on your way to recovery.

1. Denial

“There’s nothing in that big data that we don’t already know.” This is a defense mechanism, to help people justify the fact that they haven’t yet tackled any big data projects. The fact is, you don’t know what’s hiding in big data until you take some inventory. Maybe there’s nothing there. But it’s much more likely that something is there, and if you don’t explore, you’ll never know.

This attitude has often been observed by us in IT departments reluctant to move away from change requests and data models. On a larger canvas, big data signals changes in the entire underlying process of how enterprises leverage data to arrive at smarter decisions by promising a far more agile, ‘self-service’ model for deploying and consuming analytics: an innately threatening proposition for IT departments fearful of losing control over data.

2. Anger

“There’s nothing in that big data that we don’t already know!” Yes, the same as above, but with an exclamation point for emphasis. This may apply to organizations that have done some early exploration with big data and been disappointed. As with any expedition, you may not strike gold on your first attempt. That doesn’t mean you stop looking. Two years into the big data era, there are enough best practices out there that you can find some that apply to your area. Take some time to adjust your strategy, and try again.

This attitude has often been observed by us in IT departments that have dabbled obligatorily and uninterestedly in big data. It is in the interest of IT departments to encourage teams to experiment with big data, however, so they can understand where it does and doesn’t work. Those test runs will also enable them to anticipate future uses of their data, and the data management and security best practices that such uses would require.

3. Bargaining

“If we could just get the budget to expand the project…” It’s easy to blame outside factors for a disappointing outcome. We counsel our clients to begin at the beginning: Work with what you have available. You may find it’s necessary to add more data to a data set in order to have a successful outcome, but that’s rare. More often than not, we find that the data on hand is a treasure trove on its own.

This attitude has often been observed by us in one or more teams that have experimented with big data but have not necessarily identified the best tools. This can often happen when teams dabble in big data without a clear understanding of what it is. While big data technologies essentially offer faster processing of larger, more diverse datasets, many people tend to confuse the message by focusing excessively on the former and forgetting the latter. This leads to disappointments about performance and demands for larger budgets for better infrastructure.

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":888778,"post_type":"guest","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,","session":"C"}']

4. Depression

“These data sets are just overwhelming. There’s no way we can do this.” The simple truth: Yes, you can do this. Thousands of other organizations are doing this every day. Put a plan in place. Start small, with some easy wins that will excite the team and show management that investments in big data analysis can pay off.

This attitude has often been observed by us in one or more teams that have experimented with big data but have not necessarily identified the best use-cases. It’s important to understand that big data technologies don’t help in every situation. For example, processes which are inherently sequential, in terms of how they read and compute over an ordered data set, cannot necessarily benefit from distributed computing. Successfully identifying the low-hanging fruits is essential to realize the value that big data offers.

5. Acceptance

“This isn’t going to happen overnight. We need to be strategic about how and when to undertake big data analysis.” If this is your organization’s current attitude, you’ve come to terms with the challenge of big data, and reached acceptance.

This is the collective attitude of enlightened businesses which have seen the long-term value in big data. They take a holistic approach to strategically adopting it within their organizations, one step at a time. While still somewhat rare, these businesses tend to be absolutely committed to a coordinated, long-term plan not just for leveraging big data now, but also for competing and thriving in the information economy of tomorrow.

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":888778,"post_type":"guest","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,","session":"C"}']

In our view, big data analytics can indeed bring tremendous value to organizations if they understand how to leverage it. The value isn’t in the number crunching; it’s in how you use the results to impact the business.

In our experience working with clients, a lack of clear understanding of what big data is and what it can do (by any stakeholder) often leads to troubling results: either to minimal support from a threatened IT, or disappointment and disillusionment for hopeful business teams trying to draw greater insights from it — and sometimes, both.

Companies that take the time to understand what big data can offer, however, are also usually the ones that have successfully navigated through the five stages of grief. It is also these same clients that often end up collaborating with upper management to successfully develop a clear strategy for capitalizing on big data across their entire organization.

Zubin Dowlaty is head of innovation and development at Mu Sigma, a big data solutions company. He works closely with Fortune 500 organizations to help develop new products and services that will enable them to make better business decisions that are grounded in real data. Prior experience includes VP Decision Sciences at InterContinental hotels group and Statistical Modeling Manager at United Parcel Service.

[aditude-amp id="medium3" targeting='{"env":"staging","page_type":"article","post_id":888778,"post_type":"guest","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,","session":"C"}']