Information overload: when is too much data too much?

Summary:Is the process of ingesting petabytes of data about every online transaction, every tweet, and every 'like' button clicked making companies any smarter about the people they seek to serve?

Readers of this blogsite as well as the rest of SmartPlanet will know that we have been tireless proponents of analytics to solve all kinds of problems -- understanding customers, markets, cities, health issues, transportation issues, and even searching for extraterrestrial life , to name a few.

But, particularly for businesses, is the process of ingesting petabytes of data about every online transaction, every tweet, and every "like" button clicked making companies any smarter about the people they seek to serve?

Two points here: First, as discussed before on these pages, there's a school of thought that suggests that spontaneous decisions often are better than decisions arrived at after hours or days of intense analysis.

Then, there's that old maxim of quality over quantity. Do attempts to collect and analyze piles and piles of data about customers really lead to a full understanding of customers? Or just piles and piles of data, some of it potentially intrusive?

A new report in Knowledge@Wharton talks about the privacy issues in tracking customers so closely, but also brings out another conclusion: that while technology has made it possible to collect massive amounts of data on customer habits, it also has resulted in "sloppiness" in the way the data is handled, rendering it  meaningless in many situations.

As Wharton marketing professor Peter Fader expressed it:

"In the past, data was scarce, so companies carefully analyzed what they had, Fader notes. Firms like Pillsbury and Procter & Gamble used in-house experts to track repeat purchases, augmenting what they found with demographic data and results from expensive surveys. Direct marketers compiled lists that they tested and re-tested. Insurance companies learned how to classify their customers into high and low-risk groups. 'Companies were very shrewd with their data because it was so hard to get,' says Fader. 'Today, it's drinking from a fire hose. We have everything, a million variables ... and the data miners are trying to use all of it.' In the rush to collect and connect, he adds, the data itself has become the focus -- not the customer."

Fader points out that many online purchases are too random to draw meaningful conclusions. "How granular should our messaging get? The one-to-one is going too far. Just because we have two or three data points on a customer doesn't mean we've nailed them down." Often, he adds, "taking the data too literally draws misleading conclusions and puts out messages that are not only ineffective but almost creepy."

The takeaway here is that all the information in the world is available to businesses that want it; but the ability to be selective, careful and smart with data will separate a company from others that are buried in piles of bytes.

This post was originally published on

Topics: Innovation


Joe McKendrick is an author and independent analyst who tracks the impact of information technology on management and markets. Joe is co-author, along with 16 leading industry leaders and thinkers, of the SOA Manifesto, which outlines the values and guiding principles of service orientation. He speaks frequently on cloud, SOA, data, and... Full Bio

zdnet_core.socialButton.googleLabel Contact Disclosure

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Related Stories

The best of ZDNet, delivered

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.