'Big data as a service' is here, but is anybody ready?

Is "big data as a service" — BDaaS — a real badass idea or what?

From a technical standpoint, BDaaS is perfectly doable, wrote Philip Wik, database administrator for Redflex in Service Technology. Sure, the last thing we need is another "something as a service" term. But the real question is: What would the business do with it?

It's inevitable that big data — defined as volumes of data, in new varieties, moving through organizations at close to real-time speeds — will be driving all technology decisions in the near future, Wik believes. "Big data, in combination with clouds, EDA, and SOA, is defining the future of information technology," he said. "Although third-form normalized operational data stores will retain their value, big data will not only supplement, but will eventually replace, data warehousing star schemas, as infrastructure costs decline and presentation level analytical tools increase in both number and in sophistication."

The ingredients necessary for BDaaS include a high-functioning service-oriented architecture, cloud virtualization capabilities, complex event-driven processing, Hadoop, and business intelligence tools than provide deep analytics. The pieces are already falling into place, Wik observed: "As big data software continues to improve, changes can be made in the user interface, communications, data storage, and task processing layers without having to rebuild the entire architecture," said Wik. "BDaaS can be regarded as an asynchronous SOA, with a complementary relationship between services and events."

Again, the question is, is everybody ready for this? Wik says the immediate, as-a-Service availability of big data analytics creates some troubling questions, not only from a business standpoint, but from an ethics standpoint as well. The power of analysis that big data provides can be abused. "Rarely in the technical literature do the words ethics and big data appear in the same sentence," he stated. "We can define what big data is, but do we understand what big data means? As a matter of commercial self-interest in the context of our universal rights, we must address the ethics of big data."

As Wik puts it:

Vast databases that talk to other vast databases could erode our sphere of privacy to the point that privacy will cease to exist, even for those who believe that they are off the grid. Because of the ubiquity of sensors and cameras, the grid is our existence itself.

The ethics of big data is one concern, and another should be the efficacy of big data. Business leaders may not have grasped its potential as of yet, and even those working with it have only begun to dip their toes in it. Unfortunately, things being what they are, "big data" is being sold as the next transformative panacea for all business ills. But as with anything technology related, applying big data analytics will not in and of itself deliver profitability or growth. What is needed is enlightened management that understands its potential.

Indeed, as Renee Boucher Ferguson pointed out in a post at MIT Sloan Management Review, there's even a danger of management becoming blindsided by too much reliance on the promises made around big data. As Richard Haass, president of the Council on Foreign Relations, asked IBM CEO Ginni Rometty at a recent meeting:

You don’t worry at all that there's a danger in data? I was sitting there listening, and I was thinking of what Wayne Gretzky says about, "you don’t skate to where the puck is; you skate to where the puck is going to be." Can't reams of data get in the way? Doesn't data at some point almost force you inside the box and towards averages?

Ferguson also discussed the risk of biases and gaps in the data that organizations are coming to rely on — "getting drawn into particular kinds of algorithmic illusions".

The bottom line is we have the tools to build BDaaS. But what we do with it is another, perplexing question.