Here's what big data can't do: Chess, weather, and politics

REDWOOD CITY, Calif. — Do you design by data, or design by your own intuition? Or somewhere in between?

At today’s DataBeat/Data Science Summit, Quid CTO and Oxford physics Ph.D Sean Gourley touched on the balance between and dangers of data-based and intuition-based decision-making.

“When we think about what data can do, it’s important to help people understand the world around them,” Gourley said. “If you’re building systems to make changes in the world, understanding is really core to that.”

In terms of algorithms versus intuition, he continued, “Where does big data leave us as humans? Are we out of the game?

“Biological limitations mean we can’t think quickly enough … but we also know that if you look at a system like weather prediction, we can track data from the last few years, and computers can play a massive role in prediction. But meteorologists adjust the outcomes of the models and give it a 16 percent lift by interacting with that model.”

The conclusion Gourley has come to, he said, is, “Humans and machines perform better than machines alone. … Big data can’t predict everything.”

When it comes to chess, Twitter, or world peace, Gourley said it’s “naive” to imagine that algorithms don’t have any limitations. For example, there’s the issue of data quality.

“Data is messy. You start to understand that the information out there is more than any organization can collect on its own. … There’s huge amounts of information owned by publishers hidden behind paywalls. But if you make all data free, who’s producing it?”

Case in point: Wikipedia.

On the other side, human biases also get in the way of complex problem-solving, especially for topics like geopolitics or casualties of war.

Ultimately, he concluded, practice with arranging that balance will lead to near-perfection.

“The technology we create will modify us. And we have to be okay with that. But we should design it to change us in a way that’s beneficial and will make us smarter, designing it with humans in mind. How do you project this data in a way that can be comprehended?”

Stay tuned for more from the Data Science Summit today in Redwood City, Calif.