Do companies really get big data?

Our Big Debate panel discussion last week, where we asked "Big deal about big data, but is Singapore benefiting yet", threw up several interesting talking points such as the importance of reviewing analytics tools as well as privacy concerns.

The consensus was data, if properly tapped, would yield tremendous benefits for businesses and the general population to gain deeper insights and make better decisions.

Panelist Tan Eng Pheng from Singapore's Infocomm Development Authority noted many companies currently have a lot of automation, but may not necessarily be getting all the value from the data they've harvested. The ability to manage all this data that's coming out of IT systems will present great opportunities for organizations, he said.

An audience member, however, pointed out financial services providers in Singapore are huge adopters of big data technology and have been using it for a long time.

I'm not sure if I agree with that. Yes, telcos and banks for the last decade at least have been using business intelligence tools to analyze the multitude of customer data they already have to better understand the kind of services their consumers prefer.

But these are datasets which are typically quite structured. Customer credit card bills, for instance, or user demographics, call history, and shopping habits. They can easily be collected and tagged according to pre-determined parameters set by the bank or telco. This is not the case today.

Ten years ago, they didn't have to deal with the plethora of data coming from social media networks and user-generated videos, and from a wide array of devices such as smartphones and tablets.

Just think about these stats... In just one minute, Twitter users average over 100,000 tweets, Google generates 2 million searches, Facebook users post over 680,000 pieces of content, YouTube users upload 48 hours worth of new video, and 204.1 million e-mail messages are sent.

In fact, 90 percent of data in the world today was created in the last two years alone. Yes, there have long been tools to help companies tap the value of data, such as business intelligence and analytics, but it's the oft-cited "three Vs"--velocity, variety, and volume--which have made big data a business challenge today.

Big Debate panelist from the National University of Singapore, Jude Yew, explained: "The name itself implies a problem. Big data, there's so much of it, how do you make sense of it and you're capturing all this data in real-time."

He added there were meaningful insights behind actions but if not properly analyzed, organizations could end up making spurious conclusions.

The good news is IT vendors already have been coming up with more advanced analytic tools to extract intelligence from the era of big data. It also helps that public opinion and attitude toward privacy is changing.

Yew added: "People are becoming more open. It's quite normal to see people 'check in' when they enter a restaurant, whereas 20 years ago I don't think this would have been an acceptable norm. That's a good thing. People are more open to their data being used, and companies also are more open with their data via APIs (application programming interfaces), for instance."

He cautioned, however, about the need to tread the privacy line with care. If abused, public attitude can change very drastically. "Whenever Facebook rolls out a change that people are suspicious of, there will be backlash. We see people closing their accounts and stopping their usage of Facebook," he noted.

"Likewise, over here, it's something that needs to be managed. That's the challenge. How do we balance the insights we can gain from the data but, on the other hand, how do we protect their privacy and ensure an ethical use of the data," Yew said.

That's timely reminder for any organization eager to deploy big data technology today. One wrong move and they could lose the customers they had spent the last decade trying to understand.