Wanted: Big data unicorns to wrangle unruly data

Wanted: Big data unicorns to wrangle unruly data

Summary: Big data skills remain hard to find, and confusion about how to define what a data scientist does is just adding to the headaches.

Looking for skilled data scientists can be akin to 'chasing unicorns'. Image: Shutterstock

The skills needed to wade through the rising tide of big data are in short supply, according to employers, although their unrealistic expectations may be part of the problem of finding the right staff.

UK companies involved in big data and analytics warn there is a "severe shortage" of local data talent with the right skills, according to research. All the companies surveyed said they were looking for analysts with a heady mix of competencies — "a data scientist profile, including a mix of analytical and coding skills, and creativity and business know–how".

Big data: An overview

Big data: An overview

Big data: An overview

The report said this particular skillset was needed because of the increasing availability of bigger and messier data in all industries. Four in five of the companies interviewed said they were struggling to find the talent they need, with the skills shortage appearing even more severe outside of London.

Not only are there are too few candidates with the skills demanded, but many candidates lack experience, the report said: "Seasoned data analysts are very expensive, and junior people require extensive training."

It added employers had complained that "good analysts often cannot code, and good coders often cannot analyse. Data analysts with the commercial nous to create business impacts are very rare."

On top of this, managers said they found it hard to explain their needs to HR managers and recruitment agencies, and to assess the skills of candidates.

The long-term outlook is uncertain with some firms expecting better tools for data analysis to alleviate the pressures on the market for data talent, while others worry that competition for data talent between sectors will heat up the market further. Some companies are considering — or have already started — offshoring their data analysis capabilities outside of the UK.

As a result, the research — by UK 'innovation foundation' Nesta, Creative Skillset, and The Royal Statistical Society — said politicians need to take "urgent action" to address the problems by for example encouraging more training, ensuring that UK employers can recruit overseas data talent to remedy current skills gaps, and create better links between employers and universities to improve the supply of talent.

The report described data scientists as workers with a particularly sophisticated skillset: "Programming and database skills to access and 'wrangle' unruly data, statistical skills to extract insights from it, and the business knowledge to transform those insights into impacts."

However, it admits that this is a rare set of abilities. "Some say that looking for them is like 'chasing unicorns', because people able to master all those skills are so rare, while others complain that the data scientist label has become too vague a description of an occupation to be useful."

The report warns that confusion around definitions matters because it makes it difficult for employers to recruit the right talent, for talent to find what jobs to apply for, and for universities to design programmes that address the needs of industry.

Read more on big data

Topics: Enterprise Software, Big Data

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • "Seasoned data analysts are very expensive"

    Law of supply and demand... high demand, low supply, prices go up. But the tech business being what it is will beg congress for more H-1B visas to bring in only slightly qualified people from offshore who need less training to dilute the talent pool to drive the prices down. Businesses will say they need the "talent" when they can find similar talent in the United States. They will also say they want diversity but what they really want is that cheap labor because they think it will save money.
  • You're right about unrealistic expectations

    I'm a data statistical analyst.

    I mine databases, and I apply statistical techniques to the data extracted.

    The key to data mining is refining and drilling down to what the requester wants in as concise and exact questions as possible, and designing queries to match, no more, and no less. Unfortunately, those asking the questions rarely know what they want to know until after at least the first data pull. And those are the management people who claim that a "more experienced 'data scientist'" will solve that problem. They need to look in the mirror to see where the problem really lies.

    As far as statistical analysis is concerned, 90% (or more) of the needs of business, even with big data, can be solved with the basic statistical tools you can find in any college intro to statistics course.
  • Don't forget the data!

    The worry I have is that the discussion around "data scientists" tends to focus on the technical skills required to design and build models, rather than the communication skills to interpret the results and articulate the underlying narrative.

    The questions of data quality, data governance and meta data management also come into play. In order to have information that maps to the business requirement, good data stewardship is vital, as I discussed in this post: