Why a big brain may be the main requirement for big data
It's an engrossing thought for an idle moment: if you had all the computing power in the world at your disposal, what task would you want accomplished? Would you put it to work on a cure for cancer? Or perhaps the power could be used to discover whether there's an absolute square root of pi.
Big data: An overview
There are plenty of issues that computers could address — and what's becoming apparent is the virtually limitless nature of computing resources at the world's disposal. Where 30 years ago an 8-bit computer and 256KB of RAM was a big deal, now we have quad-core phones storing gigabytes of information.
The huge repositories of information around the world, which get bigger every day, have given rise to the term 'big data' — and although few people seem to like the term, no-one has yet come up with a better one.
When talking about big data, it's never long before Google and Amazon enter the conversation. These companies represent two of the biggest data stores in the world — data stores that are measured in exabytes and petabytes. But Google and Amazon are typical of today's large enterprises in another way: we think of them as two very large companies, but in reality they comprise multiple smaller entities.
Take Amazon as an example. Within Amazon is a second-hand book company (AbeBooks), a download company (Audible), a publisher (Kindle Publishing), the IMDb movie database company, four shoe companies and many, many more. Amazon also recently bought The Washington Post.
Look at Google and the picture is similar. Again, the 'search giant' actually comprises many different entities doing many different things.
Amazon and Google measure their files on individuals and companies in the multiple millions, but how do they keep tabs on everything? Big databases help. I am an Amazon customer, and have been since 2002 when I bought two Star Trek DVD box sets for £122. I know that because all the details of that transaction are still in Amazon's database. So are the details of every other transaction by all the millions of Amazon customers around the world. That's big data in action. Now companies and individuals can keep details of everything because the world has enough storage space for everything.
The big question is: now we have all these petabytes of data, what can we do with it?
CSC survey
The IT outsourcing company CSC completed a report on Big Data and Analytics in the UK and interviewed 130 CFOs and CIOs to see how they viewed the issue from their respective sides of the business. All were agreed that the IT industry is at a crucial stage because of the opportunities offered by big data. The problem is, nobody really knows what to do with it yet.
The survey did come up with some crucial insights though:
• Data is the competitive tool for any organisation. But it has to be the right data, correctly managed, suitably arranged, and come from a mixture of sources, including external sources. If the data is clean, accurate and appropriate then the correct use of the data will put the organisation in a strong, powerful position. Such a company would most definitely be an Emerging Expert rather than an Eager Enthusiast (see below).
• But having the right data is only the first step. Organisations need to be able to interrogate data in order to turn it into intelligence — that is, actionable business insights. The CIO not only needs to be able to carry out these actions, but must also be able to create the business case for them in the first place. CSC's report should go some way towards helping the CIO create such a case, so that they can get down to manipulating the data as soon as possible. After all, you don't want your competitors to reach big data enlightenment before you.
• Eager Enthusiasts tend to use data to look back, to measure past success or failure. Early Adopters have taken the time to collect and treasure a lot of data, but aren't quite sure how to use it appropriately, while the Positive Pioneers and Emerging Experts use data to project, forecast, strategise and think about the future.
• Around half of the researched organisations recognise that big data could have as huge an impact on their company as the creation of the web. This is the survey's biggest finding: not that big data will happen, but rather the recognition of the huge impact that big data (when correctly applied) will have on organisations.
• The majority of CFOs say the quality of data to which they have access, and the speed at which it's delivered, is adequate at best. On the other hand, 10 percent of CFOs say that the data they receive is brilliant and that this enables them to achieve optimum business performance. Although a quarter of CIOs suggest that their CFO doesn't know what big data is, only 3 percent of CFOs offer this perspective. In a similar vein, over half of CFOs say that big data is on the balance sheet, whereas only 9 percent of CIOs thought this would be the case.
• Most CFOs see IT as a cost — a necessary cost, but a cost all the same. According to many CFOs, IT is delivered as a service or a utility, which positions the CIO as someone who ensures that such a service is always available. However, a significant minority of CFOs see IT as one of the key elements in growing the business, and they also say that the head of technology enables such growth.
• The advantage that can be delivered by successful big data exploitation can only be achieved if the organisation has IT and the business working together in synergy. Only the Positive Pioneers and the Emerging Experts will have these two sides working together to deliver such an outcome.
• Data scientists, mathematicians or pattern trackers are employed by over half of the researched organisations to manipulate information. Most of these work within the IT department, but just under half work within the business. In 2013 these people should become more prevalent if organisations are to reap the big data rewards on offer.
• The impact of big data on the organisation could be the introduction of a new driver for IT and business co-operation — the Chief Information Marketing Officer (CIMO). This is someone who can successfully bring together large amounts of data from a variety of sources and ensure that their organisation makes appropriate, considered, strategic decisions, becoming the Emerging Experts in their markets in 2013.
The last point is perhaps the most telling. big data is not too difficult an idea for people to comprehend. The principle is that all organisations have a lot of data, much of which is potentially — perhaps enormously — useful. But how do you determine what's useful and what isn't, what tools you need to analyse the data, how you analyse the data and so on?
The right kind of brains
Every organisation needs someone who can successfully orchestrate large amounts of data from a variety of sources and ensure that appropriate, considered and strategic decisions flow from that data. But who are these people, and where can they be found?
According to CSC, 57 percent of organisations maintain that they have "data scientists, mathematicians or pattern trackers within their IT department or throughout their organisation". That's good news — IT organisations believe they have the right people. However, only 23 percent of CIOs say that IT helps the business to grow "like water grows flowers", as CSC puts it.
According to analyst firm McKinsey, what the world needs is more brains — or "deep analytical talent", as they put it. Two years ago McKinsey looked at what was needed in terms of talent. Their report looked at industries from aerospace to wholesale trade and asked what roles they had that needed filling. In among the actuaries, epidemiologists, mathematicians and industrial engineers there was one role that all industries needed to fill all the time: operational research analysts. What the world needs is people who can think, the analysts concluded.
Big Data: Book review
But there's a problem. Everybody is agreed that in the UK, as elsewhere, we don't just need people with brains — we need people with the right kind of brains. But everybody is also agreed that we have a shortage of the right analytical skills and nobody is agreed on how to go about solving that problem.
But again, according to current analyst opinion, this is by no means the only problem. Earlier this month, NewVantage Partners produced a report on big data that claims that this whole issue will force a major shake-up to the IT industry — despite the claims of some more cynical souls that big data is a big dud.
According to NewVantage, "Big data differs from traditional approaches in the quantum leap in affordability, scale, and variety of analytics it can support". The survey showed that 91 percent of executives believed that their organisation has a big data initiative planned or in progress. Of these, 60 percent reported that at least one big data initiative has been implemented, with 32 percent claiming to have a fully operational initiative in progress.
But what do the surveyed executives believe big data to be? Most describe it as "collections of data so large, complex, or requiring such rapid processing, that [it] becomes too difficult or impossible to work with using standard database management or analytical solutions".
As expected, executives believe that big data is on the rise. According to the survey, 68 percent expect that their organizations will invest more than a $1m on big data in 2013, but that this will quickly rise with 88 percent believing it will hit that figure by 2016.
A year ago, most executives cited the need to integrate a greater variety of data sources as their primary requirement. This year's survey indicated a near-equal distribution among integrating more data sources, analysing larger volumes of data, and analytical velocity — the speed with which organisations can obtain answers to critical business questions — among the main requirements.
But above all, the NewVantage survey said, speed was the primary requirement — or, as the analysts put it, accelerating the Time-to-Answer (TTA). "The biggest factor in ensuring business success, cited repeatedly by executive respondents, is the ability to make better, fact-based decisions, and to realize this ability by accelerating the speed with which organizations can gain insight and answer critical business questions," the survey said. "The ability to accelerate the TTA is a metric for business success."
Also, according to the survey, organisations are looking at, "new roles and processes to ensure successful adoption". One result will be the rise of the Chief Data Officer (CDO), the survey suggests, with 48 percent having established one or considering one, and implementing new processes and organisational structures including CDOs to ensure successful business adoption.
Analyst firm Gartner has conducted a lot of research into big data and feels that while the term is all-encompassing it is actually misleading. "We are not happy with the term 'big data'," says Gartner vice-president, Frank Buytendijk. "It is misleading and it is not helpful. There are many different strands withing that area. IT organisations are used to dealing with their everyday IT infrastructure or structures, but to understand big data they may need to rethink these."
The people that are needed come under many different headings and classes but are, when it comes down to it, people who can think — and especially can think in a structured way. It could be one of the issues that defines the next decade.
Gartner sums up the issue under four headings, the V's: Volume, Variety, Velocity and Visualisation. Volume is all about scaling systems to optimise their capacity. Variety is ensuring that systems can cope with the huge variety of data models in use today thanks to the internet. Velocity is gearing up an organisation's various structures — organisational as well as physical — to be able to deal with situations promptly. Visualisation acknowledges the enormity of the input information used today, which in turn puts pressure on organisations to produce prompt, concise information.
"It used to be that organisations dealt with information in a structured way because that was how organisations worked," says Buytendijk. "Now they have to deal with structured and unstructured information from a huge variety of sources."
But one of the biggest problems for the IT industry is a shortage of the right skills, says Buytendijk. "Everybody knows this. There are various estimates, but we think the IT and related industries are short by about a million-plus people."
The people that are needed come under many different headings and classes but are, when it comes down to it, people who can think — and especially can think in a structured way. It could be one of the issues that defines the next decade. We need more and more people, and they need to be cleverer people — and not just clever, but clever in the right sorts of ways.
We are already seeing this in the occasional story about a system glitch that should have been solved in less than a day now taking several days as companies wrestle with systems that have become hugely complex. How the indstry deals with that will be an unfolding story.