Humanizing big data: The everyday impact of crowd science

In my recent VentureBeat post, I explored the emerging field of crowd science, how this sub-niche of data science is at the intersection of analytics and psychology, and how we apply it to our customers’ innovation processes at Mindjet. Since then, I’ve stumbled upon numerous articles and news events that involve concepts from crowd science, leading me to believe it is even more of a ubiquitous focus area than I had previously realized.

All of us — data scientists or not — are directly and indirectly contributing to this blossoming field.

Defining crowd science and data

Using data science techniques — such as data mining, algorithm development, and statistical modeling — on social network and crowdsourced data, crowd science invites psychological and behavioral elements that are not necessarily present in traditional data science. Basically, what we’re seeing is that new elements of behavior are affecting data, such as politics, opinions, and agents interacting with and influencing each other. So, in addition to looking at data in the traditional way, we must now consider political and social structures, and how people learn from and influence each other; we must consider how ideas flow through social networks, what motivates people to contribute to discussions, and the consequences of engagement.

With the interconnectivity of today’s businesses and communities, this level of synergy is exponentially more frequent. This much larger store of data on social structures and phenomena gives us the ability to study and understand these networks and how they evolve. Whereas traditionally scientific data analysis involved careful measurement and often cumbersome collection of data, we now have a vast and expanding resource of data via the Internet that’s enabling us to analyze highly complex systems, like social structures and hierarchal behavior, much easier than ever before.

Dr. Alex Pentland of MIT has coined a term similar to crowd science: social physics. Pentland defines it as “a quantitative social science that describes reliable, mathematical connections between information and idea flow on the one hand and people’s behavior on the other. Social physics helps us understand how ideas flow from person to person through the mechanism of social learning and how this flow of ideas ends up shaping the norms, productivity, and creative output of our companies, cities, and societies.”

Crowd science, day to day

In the digital age, crowd science is everywhere, from shaping user experience in online platforms, to aiding in the reorganization of governments via political revolutions. Amazon uses crowd science techniques to recommend items to users based on their purchase history and search behaviors by comparing individual actions to those of their entire user base, or “crowd.” Twitter, too, is a prime example of a social network that facilitates idea flow; it’s commonplace to hear about a topic “trending on Twitter,” whether in conversation or the national media. Twitter employed hashtags as a form of metadata, making it easier to query the data and see topics that are currently on people’s minds and spreading across the world. Researchers have studied how this spread of information facilitated powerful events such as the Arab Spring and how we can use that information to influence and shape future networks and events.

And then there’s Facebook. Facebook’s news feed is a constant flow of information and opinions, which demonstrably facilitates interactions and the exchange of knowledge. Facebook was recently involved in a controversy in which their scientists manipulated the users’ data feeds to understand how it affects the tone of those users’ posts. In the end, it backfired — users who had been unaware of the experiment felt that Facebook was using them as lab rats and unethically altering their mental well-being. This is the darker side of crowd science, a field in which it’s typically necessary to use real people as subjects in order to advance.

Experimenting with the user experience is not a new concept, however. Techniques such as A/B testing have been employed for years to measure and optimize desired outcomes, such as conversion rates on an e-commerce site. The reality is that without the use of human subjects for testing and analysis, research would have to fill in the blanks with imprecise speculation. Still, these experiments have social implications that must be considered, particularly when the pool of data at hand is so unequivocally huge.

The impact of crowd science on innovation

At Mindjet, we’re interested in studying our customers’ networks in order to drive efficiency and effectiveness in productivity and collaboration, and to prescribe best practices for repeatable innovation. Our goal is to uncover the variables that lead to more engagement, and understand how we can encourage collaboration and high-value idea flow in our customers’ innovation networks. Our networks may not be influencing a political revolution, but they can shape the direction of a company, whether through the development of a new product, or a change in the overall working culture. This helps our customers make major strides in things like employee engagement, cross-regional collaboration, productivity, and valuable ideation.

As Dr. Alex Pentland writes in his book “Social Physics”: “For the first time, we will have the data required to really know ourselves and understand how our society evolves. By better understanding ourselves, we can potentially build a world without war or financial crashes, in which infectious disease is quickly detected and stopped, in which energy, water, and other resources are no longer wasted, and in which governments are part of the solution rather than part of the problem.”

We are at the cusp of a cultural revolution. It’s happening on a global scale, as civilian groups thirsty for change overthrow established governments; similarly and on a smaller scale, traditions and expectations are being tossed out and revamped within company-specific innovation communities, which are allowing employees to collaborate effectively and achieve real change. It is, in fact, reminiscent of the American Revolution, wherein a “crowd” of colonists came together with a unified view, declared and won their independence, and transcribed the rules of the society in which they lived. Much like a political democracy, crowd science and managed-innovation processes allow people to have equal input, rather than a single person or party driving the direction of the group without opposition.

It’s this shift from top-down power to communal ideation that makes crowd science so valuable for industries dependent on meeting changing market needs. As the repository of data — social, behavioral, demographic, and otherwise — continues to inflate, it’s the humanization of what we learn that will bring concrete meaning to the why, what, and how of where we’re trying to go – and give us the motivation to get there.

Anna Gordon is a data scientist at Mindjet, provider of the world’s leading enterprise innovation management platform. She completed her Ph.D. in Statistics and specializes in machine learning, data mining, business/military intelligence, risk and reliability theory, and Bayesian inference.