Humans are the weak link in Big Data
Inherent in using Big Data are the tools we use to make sense of them: statistics.
In recent posts I've looked at how untested software can skew results, and dishonest scientists who manufacture data. But supply side issues aren't the only problem with Big Data. We have to look at how humans deal with information we've never had before.
Take climate change. We know the planet is getting warmer. Almost all competent scientists agree that increased atmospheric CO2, a byproduct of fossil fuel consumption, is the cause. Other theories - such as sunspots, cloud cover, solar variations, and a world-wide conspiracy of grant-hungry scientists - have been examined and found wanting.
The impact promises to be catastrophic: hundreds of millions of people displaced, dwarfing the current refugee tragedy in Syria, trillions of dollars in property losses, tropical diseases spreading into formerly temperate climes, and more.
Yet we have the spectacle of an US Senator throwing a snowball on the Senate floor as if that made any kind of sense. And a candidate for President of the United States, the world's leading economy, called global warming BS because it got cold in New York in the winter.
Statistical ignorance
Inherent in using Big Data are the tools we use to make sense of them: statistics. Before the 2012 election, people claimed that Mitt Romney was ahead by "deskewing" poll results. There was widespread shock when the deskewed results were wrong.
There's a similar effort going on for this election, where Trump's dismal polls are dismissed by true believers because "look at his crowds!" The statistically illiterate often complain that a sample size of ≈1,000 can't possibly be accurate in a nation of 330 million.
While there are ways pollsters can screw up - look at the Brexit polls - there's no evidence that's happening here, especially when you combine the results of many polls.
Similarly, climate change denial has fed on ignorance of statistics and the scientific method. Beyond the fact that wealthy industries are lobbying hard, there's a bigger problem: most lawmakers are lawyers, not scientists.
Lawyers are trained to win arguments, not evaluate data. That's a huge systemic problem when Senator Snowball denies science with a catchy, but stupid, stunt.
The Storage Bits take
Most Big Data is going to be used to sell you something, not solve the world's problems. And that's OK.
The bigger issue is whether or not we are smart enough as a country and a global community, to ask the right questions and use these powerful tools to act on the answers. Despite the obvious idiocy among our elected officials, I'm hopeful that, eventually, we will see a return to facts, evidence and a focus on the long-term good of the nation.
Courteous comments welcome, of course.