Big data? I’d settle for any data at all

One of the hottest marketing terms right now is “big data.”

[aditude-amp id="flyingcarpet" targeting='{"env":"staging","page_type":"article","post_id":525148,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,","session":"B"}']

Like “the cloud” last year, it’s ubiquitous: Every tech company seems to be working on some kind of pitch to show how it can handle huge volumes of data and turn it into a strategic advantage for you.

I don’t blame them. Nobody wants to be accused of having “small data,” after all. That would just be embarrassing.

The truth is, though, most companies would be better off with small data — or really any data — than they are today. An embarrassing number of business decisions are made without reference to real data.

I realized this point over the weekend, while reading Daniel Kahneman’s eye-opening 2011 book, Thinking, Fast and Slow. Kahneman, who won the Nobel Prize for economics in 2002, has spent his career studying how people make decisions. He and his many research partners have found that most of our decisions are based on quick, intuitive responses, rather than on our brain’s more deliberative, analytic capabilities. The intuitive part of your brain is amazingly powerful at coming to rapid, synthetic judgments based on a large amount of information. Unfortunately, it’s terrible at making judgements about anything where there is a statistical uncertainty about the outcome, and even trained statisticians have a poor intuitive sense of statistical probabilities.

Kahneman details the many examples where supposed experts are provably inept at predicting the future. Political analysts can’t reliably predict the outcome of elections. Sports fans can’t predict who’s going to win a game. Individual investors are terrible at picking stocks, and hedge fund managers aren’t much better, doing only marginally better than random chance.

Once I’d read Kahneman’s book, it became obvious that much of the tech industry suffers from the same problems he describes.

• When a venture capitalist decides to invest in a startup, it’s often based on hunches and on “pattern matching,” the VC term for betting on things that, in their opinion, look like something that’s been successful before.
• When a startup launches a new product, it’s usually done without any real data. And that’s fine, except that startups don’t usually set themselves up to collect usage data and act on it rapidly, a model espoused by Eric Ries and others in the “lean startup” movement.
• When companies hire people, their decisions about whom to hire often come down to personal chemistry between the candidate and the hiring manager.
• When a large company decides on a marketing strategy, it’s often based on the hunches of senior marketing managers or on the advice of marketing consultants.
• When journalists — myself included — decide on an angle for a story about a new company, it’s usually based on some kind of hunch about the company rather than on exhaustive analysis of the company’s statistically likely outcomes.

There are exceptions: In environments that are relatively predictable, where repeated experience provides immediate feedback about the validity of your judgments, you can become expert enough to make valid predictions. That’s how chess masters get so good at analyzing positions at a single glance, and why experienced firemen have a “sixth sense” about when a floor is about to collapse.

Sadly, the tech world is not very predictable, nor does it provide immediate feedback. In situations like this, Kahneman advises, any kind of algorithm — even one based on common sense and scribbled on the back of an envelope — has more predictive power than an expert judgment.

[aditude-amp id="medium1" targeting='{"env":"staging","page_type":"article","post_id":525148,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,","session":"B"}']

That’s why I’m inclined to believe Vinod Khosla when he says that software can ultimately replace 80 percent of doctors. Predictive algorithms, well-designed checklists, and caring nurses can probably take care of people better than doctors can in many cases — leaving doctors to focus on the complex situations where their expertise provides real value.

So why aren’t people using algorithms more often to make business decisions? One problem is that they simply don’t have access to enough data about the outcomes of previous decisions. That’s where so-called big data companies could make a real difference.

For VentureBeat’s part, we’ve been making an effort to get more systematic about collecting data on the companies and products we cover, so we can make more accurate judgments about them — or give you the data to make your own judgments. For instance, our news team contact form, which lets you send news alerts to our reporters, is more structured than a typical email. That’s because it feeds into a database that, over time, will become a valuable resource for VentureBeat and its readers.

But we’re still only a tiny part of the way along this journey towards more algorithmic decision-making. Ditto for most of the tech industry.

[aditude-amp id="medium2" targeting='{"env":"staging","page_type":"article","post_id":525148,"post_type":"story","post_chan":"none","tags":null,"ai":false,"category":"none","all_categories":"big-data,business,","session":"B"}']

How are you using data and algorithms? Let me know in the comments below or by email. I’d like to hear from you.

Photo credit: saturn ♄ via photo pin