The stupidity of Business Intelligence and why this ‘hot’ sector needs an A.I. overhaul

Several years ago, I was working as the CTO of a major tech company with significant operations around the world. One morning, while reviewing our daily collected figures, I noticed something odd. I determined that we were facing a significant sales drop in one of our most important target regions. We immediately scrambled all the relevant people in our department and began combing through our data to try to identify what the problem was and how to fix it. In a matter of days, we had identified the source and created an effective way to counter the problem, yet the scars of the experience were lasting and deep.

In the weeks that followed, we pored over our protocol to identify what we could have done better to address the issue faster. We had been investing heavily in business intelligence solutions, we had action plans in case of a crisis, and as soon as we identified the problem we jumped into action, but our company had still suffered substantial losses in potential revenue.

The frustrating truth was that, apart from someone having miraculously stumbled upon the issue earlier, we were doomed to lose money that day. Simply put, the BI and analytics tools we were using were slow, reactive, and ultimately not very good at anything other than visualizing data. And while a graph certainly makes it easier to see patterns, it became clear that all our solutions were incredibly limited. This led us to the very difficult realization that the $16.9 billion BI industry is full of companies that aren’t all that intelligent and, in many cases, are actually making us dumber and slower.

Battle for incremental improvements

When Business Intelligence solutions were initially introduced, the excitement was palpable and well-deserved. New tools were making the massive amounts of data sitting on servers around the world actionable, giving businesses a powerful ability to dig deeper into their processes and pull out critical insights.

Yet it seems that the competition for innovation in the period following the initial launch has failed to deliver on the industry’s promise. The last few years have seen a major plateau in the pace and scale of innovation in the sector, and the impact is being increasingly evident.

Today, rivals are competing to add source integrations and create simpler user interfaces and more beautiful visualizations, while losing sight of the wider goal of BI, which is to help businesses channel their data to make better decisions. The result is an increasingly frustrated market of CTOs and data analysts who are spending obscene amounts of time tweaking dashboards and standard deviation functions to try to access the types of information that they actually need.

While BI providers are focusing on making incremental advances, the users of the products are suffering from the lack of foresight in the industry.

The next Nokia?

The shame of the current BI ecosystem is that the promise the sector once offered is now being overlooked, as products consistently disappoint and fail to meet expectations. The most common sentiment being heard from the range of BI experts is that companies are increasingly looking to create their own systems to overcome the vast limitations of what is available on the market.

The curse of incremental progress is pushing businesses away from the leading solutions, and the impact of this will have a massive reach. Firstly, it will cause several leading tech brands to go the way of Nokia, as they fail to keep up with the inevitable innovation that will come to replace them. Secondly, an entirely new wave of BI tools will hit the market, creating a huge amount of unrest in the sector as legacy players struggle to keep up. Vast amounts of capital will be invested, and the M&A market will likely see a major influx. Many of the leading players — some of whom are publicly traded – will face intense scrutiny.

The only question that remains is which technologies will drive this next wave?

Autonomous BI

Whether you prefer machine learning, deep learning, or Artificial Intelligence, the terminology is far less important than the capacity to teach systems how to analyze information autonomously. Returning to my own individual crisis, why couldn’t my solution have alerted me that a major decrease had occurred in my sales in a given region in real time? Why was it incapable of effectively narrowing down the range of potential sources for the issue — based on the time and manner in which the incident occurred?

All advances in A.I. are built on the premise that if we can teach machines to learn from their “experiences,” then they will be able to more effectively sort through new information and help us flag the pieces that we need to know about immediately. Obvious steps forward, like the capacity to more effectively recognize seasonality or expect “unexpecteds,” will help lower the number of false positives and enable a far greater reliance on BI.

Beyond that, the capacity to more quickly link a cause with an effect could drastically reduce the time it takes to prevent or handle a crisis, thereby limiting massive losses. And as BI solutions are able to dig deeper, they will empower businesses to uncover unforeseen opportunities, creating new means of driving revenue and enabling far more insightful decisions.