Why big data needs human insight

What can we accomplish by combining the best of human intellect with the expanding processing capacity of machines?
Written by Mari Silbey, Contributor on

In finance, space exploration, climate science, and a multitude of other industries, computers are taking on greater responsibility for conducting business and solving the world's problems. So what role will humans play in our own future? How much control should we cede? And most important, what can we accomplish by combining the best of human intellect with the expanding processing capacity of machines?

At the recent Structure: Data conference put on by GigaOM, the question of how humans and machines should work together was not just a philosophical musing but an issue of practicality. The presentations covered real-world big data and computing applications, as well as the murky future of marrying machine-based quantitative power with humans' creative and qualitative analytical skills.

Searching for signals

"We have a deal with UPS, and we are now about to start underwriting solely on UPS data that we gather," noted Robert Frohwein, CEO of financing corporation Kabbage during one panel at the event. Then he clarified, "When I say solely, that's a primary underwriting source."

Kabbage relies on computer algorithms to help determine when a business is a worthwhile credit risk, and how much money should go into a loan. It turns out UPS shipping information provides a powerful predictive metric in investment evaluation. It wasn't a computer program that discovered that fact: people at Kabbage inferred a possible connection and theorized that shipping information could prove valuable as a data signal.

During a different session, Eric Berlow, founder of Vibrant Data Labs, an organization committed to data for social good, emphasized the same point. "A problem well defined is a problem well solved," Berlow said. He was referring to the idea that computers can calculate equations with a huge number of variables, but without humans to define the parameters, there's no way for a computer to distinguish what is signal and what is noise.

Throughout the event, a consensus emerged that when it comes to problem solving, people are particularly good at two things: constraining a data set to the context of a particular situation and generating new avenues of analytical exploration. Humans have intuition and creativity. Machines, meanwhile, are good at processing large volumes of information at great speed.

Synthesizing strengths

The complementary strengths of humans and machines should lead to better problem solving. Yet, it can be difficult to combine the two. Most organizations doing machine-based quantitative analysis focus on problems that can be solved primarily with math, such as how to create a better search agorithm or how to optimize bidding on financial stocks.

Sean Gourley, co-founder and chief technology officer of data analytics company Quid, however, underscored the importance of overlaying a human perspective on a machine's computational outlook. The easiest problems to solve are ones that can be easily quantified. But, Gourley asked in his presentation, should we really only focus on the easiest problems?

Gourley gave the example of deciding when and how to reduce U.S. troops in Iraq. Equations can help determine a possible inflection point for troop reduction, but those equations are informed by human knowledge of things such as the psychology of an insurgent group. In the case of evaluating troop strength, you don't end up with a precise model, said Gourley, but one that includes qualitative inputs. Still, "you wouldn't want to do this without the data at your hands."

Indeed, the unifying theme of the conference was that the combined powers of human and machine offer the best hope for dealing with the world's biggest challenges. Big data can solve big problems, but only with the help of human insight.

Image courtesy of kosheahan on Flickr

This post was originally published on Smartplanet.com

Editorial standards