Wildly inaccurate election forecasts highlight Big Data challenges

For several weeks I was receiving daily messages on my phone from the New York Times' Upshot column, which confidently predicted Trumps chances of winning at around 5 per cent most days.

See also

The Power of IoT and Big Data

We delve into where IoT will have the biggest impact and what it means for the future of big data analytics.

Read now

The Upshot data was crunched from many different polls and fed into a special algorithm based on historical and other relevant data. Other organizations also used reams of Big Data to feed their analytical models and were coming to similar predictions: Trump would lose.

So how was it that all these sophisticated analytical models with access to high quality data got the election forecast so wrong?

Jim Rutenberg in the New York Times writes that there was a cultural bias.

Journalists didn't question the polling data when it confirmed their gut feeling that Mr. Trump could never in a million years pull it off. They portrayed Trump supporters who still believed he had a shot as being out of touch with reality. In the end, it was the other way around.

There's an important lesson for enterprises here is that simply getting access to all your Big Data is not enough. It won't result in valuable business predictions unless the analysis is the right one.

Domain knowledge counts for a tremendous amount of success with Big Data because analysis matters and knowing the right questions to ask comes from experience. The right analytical model is vital but being aware of cultural bias in those models is key.

See also: Big data: Three ways to make your project a success | The IoT security doomsday is lurking, but we cannot talk about it properly | How to stop people being the weakest link in enterprise security

However, the inclusion of a cultural bias into analytical models is not something to avoid because if a company knows the culture of its customers it can uncover emerging markets or changes in buying habits more rapidly than others.

The Big Data industry promises businesses that they can uncover new sources of revenues -- and they can. But the election has shown that Big Data is useless if the analysis is flawed.

It's the hidden cultural bias that's dangerous and can lead to wildly inaccurate predictions. Knowing that there will always be some hidden cultural bias means better design of analytical models.

Fortunately, analytical models can learn and adapt, and can be run against each other, to give management a good understanding of their business and their options for future performance.

It's too late for the pollsters but their analytical tools will certainly be a lot sharper next time.