Wanted: Big data unicorns to wrangle unruly data

Summary:Big data skills remain hard to find, and confusion about how to define what a data scientist does is just adding to the headaches.

unicorn
Looking for skilled data scientists can be akin to 'chasing unicorns'. Image: Shutterstock

The skills needed to wade through the rising tide of big data are in short supply, according to employers, although their unrealistic expectations may be part of the problem of finding the right staff.

UK companies involved in big data and analytics warn there is a "severe shortage" of local data talent with the right skills, according to research. All the companies surveyed said they were looking for analysts with a heady mix of competencies — "a data scientist profile, including a mix of analytical and coding skills, and creativity and business know–how".

Big data: An overview

Big data: An overview

The report said this particular skillset was needed because of the increasing availability of bigger and messier data in all industries. Four in five of the companies interviewed said they were struggling to find the talent they need , with the skills shortage appearing even more severe outside of London .

Not only are there are too few candidates with the skills demanded, but many candidates lack experience, the report said: "Seasoned data analysts are very expensive, and junior people require extensive training."

It added employers had complained that "good analysts often cannot code, and good coders often cannot analyse. Data analysts with the commercial nous to create business impacts are very rare."

On top of this, managers said they found it hard to explain their needs to HR managers and recruitment agencies, and to assess the skills of candidates.

The long-term outlook is uncertain with some firms expecting better tools for data analysis to alleviate the pressures on the market for data talent, while others worry that competition for data talent between sectors will heat up the market further. Some companies are considering — or have already started — offshoring their data analysis capabilities outside of the UK.

As a result, the research — by UK 'innovation foundation' Nesta, Creative Skillset, and The Royal Statistical Society — said politicians need to take "urgent action" to address the problems by for example encouraging more training, ensuring that UK employers can recruit overseas data talent to remedy current skills gaps, and create better links between employers and universities to improve the supply of talent.

The report described data scientists as workers with a particularly sophisticated skillset: "Programming and database skills to access and 'wrangle' unruly data, statistical skills to extract insights from it, and the business knowledge to transform those insights into impacts."

However, it admits that this is a rare set of abilities. "Some say that looking for them is like 'chasing unicorns', because people able to master all those skills are so rare, while others complain that the data scientist label has become too vague a description of an occupation to be useful."

The report warns that confusion around definitions matters because it makes it difficult for employers to recruit the right talent, for talent to find what jobs to apply for, and for universities to design programmes that address the needs of industry.

Read more on big data

Topics: Enterprise Software, Big Data

About

Steve Ranger is the UK editor-in-chief of ZDNet and TechRepublic, and has been writing about technology, business and culture for more than a decade. Previously he was the editor of silicon.com.

zdnet_core.socialButton.googleLabel Contact Disclosure

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Related Stories

The best of ZDNet, delivered

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.