Wanted: Big data unicorns to wrangle unruly data

Big data skills remain hard to find, and confusion about how to define what a data scientist does is just adding to the headaches.

Looking for skilled data scientists can be akin to 'chasing unicorns'. Image: Shutterstock

The skills needed to wade through the rising tide of big data are in short supply, according to employers, although their unrealistic expectations may be part of the problem of finding the right staff.

UK companies involved in big data and analytics warn there is a "severe shortage" of local data talent with the right skills, according to research. All the companies surveyed said they were looking for analysts with a heady mix of competencies — "a data scientist profile, including a mix of analytical and coding skills, and creativity and business know–how".

Big data: An overview

Big data: An overview

Data is being generated about the activities of people and inanimate objects on a massive and increasing scale. We examine how much data is involved, how much might be useful, what tools and techniques are available to analyse it, and whether businesses are actually getting to grips with big data.

Read More

The report said this particular skillset was needed because of the increasing availability of bigger and messier data in all industries. Four in five of the companies interviewed said they were struggling to find the talent they need , with the skills shortage appearing even more severe outside of London .

Not only are there are too few candidates with the skills demanded, but many candidates lack experience, the report said: "Seasoned data analysts are very expensive, and junior people require extensive training."

It added employers had complained that "good analysts often cannot code, and good coders often cannot analyse. Data analysts with the commercial nous to create business impacts are very rare."

On top of this, managers said they found it hard to explain their needs to HR managers and recruitment agencies, and to assess the skills of candidates.

The long-term outlook is uncertain with some firms expecting better tools for data analysis to alleviate the pressures on the market for data talent, while others worry that competition for data talent between sectors will heat up the market further. Some companies are considering — or have already started — offshoring their data analysis capabilities outside of the UK.

As a result, the research — by UK 'innovation foundation' Nesta, Creative Skillset, and The Royal Statistical Society — said politicians need to take "urgent action" to address the problems by for example encouraging more training, ensuring that UK employers can recruit overseas data talent to remedy current skills gaps, and create better links between employers and universities to improve the supply of talent.

The report described data scientists as workers with a particularly sophisticated skillset: "Programming and database skills to access and 'wrangle' unruly data, statistical skills to extract insights from it, and the business knowledge to transform those insights into impacts."

However, it admits that this is a rare set of abilities. "Some say that looking for them is like 'chasing unicorns', because people able to master all those skills are so rare, while others complain that the data scientist label has become too vague a description of an occupation to be useful."

The report warns that confusion around definitions matters because it makes it difficult for employers to recruit the right talent, for talent to find what jobs to apply for, and for universities to design programmes that address the needs of industry.

Read more on big data