Big data: An overview

Big data: An overview

Summary: Data is being generated about the activities of people and inanimate objects on a massive and increasing scale. We examine how much data is involved, how much might be useful, what tools and techniques are available to analyse it, and whether businesses are actually getting to grips with big data.


Big data vendors

If you're looking to exploit big data in your business, who are the vendors you should be considering? As might be expected, there's a great deal of activity in this area, with many startups, a few emerging 'star' companies, and established database vendors working hard to adapt to the latest developments in data management, analysis and visualisation.

Earlier this year, Wikibon published an illuminating analysis of the big data market (Big Data Vendor Revenue and Market Forecast 2012-2017), covering 69 vendors of hardware, software and services — including both 'pure play' big data companies and enterprises that have multiple revenue streams.

Current and future 'stars' of the big-data world are likely to be found among the 'pure play' vendors who derive 100 percent of their revenue from this market. These are graphed below, along with MarkLogic, whose big data revenue Wikibon estimates to be 88 percent of its total. This established company (founded in 2001) is the leader (in revenue terms) among those that specialise in Hadoop or NoSQL solutions (highlighted in red). Also prominent in the Hadoop/NoSQL community are ClouderaMongoDB (formerly 10gen), MapR and Hortonworks:

Source: Big Data Vendor Revenue and Market Forecast 2012-2017 (Wikibon, 2013). Hadoop and NoSQL specialists are highlighted in red.

None of Wikibon's top four pure-play big data vendors are Hadoop/NoSQL specialists: CIA-funded Palantir initially concentrated on data mining for US intelligence and law enforcement agencies, but its software is increasingly widely used in mainstream business; fast-growing Splunk majors on searching for, capturing, indexing, analysing and visualising machine-generated data; Opera Solutions offers big data analytics as a service in a number of business sectors; and Mu Sigma integrates a variety of commercial and open-source tools and technologies into a 'decision support ecosystem', placing much emphasis on training data scientists in its own internal 'university'.

When we look at all big data vendors in Wikibon's analysis (excluding those in which hardware accounts for more than 50% of their big data revenue), we find several classes of company heading the revenue chart: broad-portfolio tech giants (IBM, HP, Oracle, EMC); leading software houses (Teradata, SAP, Microsoft); and professional services companies (PwC, Accenture):

Source: Big Data Vendor Revenue and Market Forecast 2012-2017 (Wikibon, 2013)

Not surprisingly, a number of leading business intelligence and analytics vendors are present here. Those that also appear in Gartner's latest (February 2013) Magic Quadrant are: IBM, Oracle, SAP, Microsoft, SAS, GoodData, Alteryx, TIBCO, QlikTech, Tableau Software, Microstrategy, Pentaho, Jaspersoft and Acuate.

Also represented on the all-vendors chart are web behemoths like Amazon and Google. Big data analytics is part of these companies' internal DNA, and they have turned their expertise and infrastructure into products and services such as Elastic MapReduce and Redshift (Amazon), and BigQuery (Google).

The sheer number of companies involved in big data and the revenues being generated show that it's definitely not all hype. As ever in a developing market, we can expect plenty of future merger-and-acquisition activity as established companies cherry-pick the startups and growing firms jostle for position.

Outlook: Big, and getting Bigger

The size of the 'digital universe' is growing apace, as is the number companies involved in developing tools and techniques for managing, analysing and visualising big data. Many companies (especially large enterprises, which by definition routinely deal with 'big' data) are already exploiting big data, but despite widespread awareness of the potential benefits, it has yet to achieve mainstream adoption.

The database world now has two camps: the internet-centric, open-source-based world of scalable distributed databases, where much of the recent big data innovation has occurred; and the enterprise-centric world of traditional, heavily siloed, relational database management systems, where much of the expertise needed to actually run businesses resides. Finding ways to get the best from both worlds, creating a new generation of 'data scientists', will be key to big data's journey from hype to the mainstream.

Big data may well spend some time in Gartner's 'Trough of Disillusionment' as the various barriers to mainstream adoption are dismantled, but there's just too much valuable data out there for it to remain there for long.

Topics: Going Deep on Big Data, Big Data


Charles has been in tech publishing since the late 1980s, starting with Reed's Practical Computing, then moving to Ziff-Davis to help launch the UK version of PC Magazine in 1992. ZDNet came looking for a Reviews Editor in 2000, and he's been here ever since.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Riding high in the Hype

    I certainly have no problem agreeing that Big Data is coming close to the Peak of Inflated Expectations. That is why businesses need to accurately plan and research before they leap into a half-baked project.

    I wanted to share a video that I think can be helpful for your readers that deals with planning and executing a Big Data program. ( This video is based off of TEKsystems research and delivers the message in a cute way through multiple sci-fi references. It gives a more realistic expectations of how to begin to approach a Big Data initiative, backed up by research from leaders in the industry.
  • was the America's Cup the first use-case of big data in sport?

    This is an interesting well-researched article with good data points. On the subject of how big data will be used and what value it will add, there is an interesting theory that Oracle just won the America's Cup by using big data. Below is a blog post on this topic (which i contributed to).
  • What the Gartner hype cycle misses out

    Of course certain hyped things prove to be nothing but hype and after the trough of disillusionment disappear without trace.

    Big data is probably one of those.

    Of course no one likes to talk about the computer industry getting something absolutely and fundamentally wrong.
  • What the Gartner hype cycle misses out

    Of course no one likes to talk about the computer industry getting something absolutely and fundamentally wrong.
    Big data is probably one of those.
  • Big Data - Businesses are not ready yet

    considering that most business are not even ready for traditional style BI, I think Big Data and open source technologies orbiting it have a long way to go. However the path is set and sooner or later Big Data will become industry norm.

    Our company helps small and medium size organizations to take first steps into BI with confidence. We provide consultancy in proprietary & open source technologies.
  • Big Data

    I guess these are merely favorable..I am just astounded by their services and functions too..Check 'em