Big data skills demand low but upsurge expected
Growth in the big data industry has not seen an accompanying spike in demand for IT professionals skilled in big data analysis due to companies' lack of understanding over what the technology can do and skills needed to mine the data, observers noted. However, this is expected to change as the market and technology matures.
Philip Carter, associate vice president at IDC, said companies in Asia, for instance, have a very low level of understanding of what big data is and how IT departments should approach it. This is generally the case in high-end analytics, and explains why the demand for skilled data practitioners "is not quite there yet", he said in his e-mail.
Even for IT heads with more knowledge of the technology, the analyst noted that they are not sure what types of skills are required to harness the information collated. Their strategy is to simply put a couple of people in their enterprise architecture team to "experiment" with the technology.
According to Carter, big data refers to a new generation of technologies and architectures designed to economically extract value from very large volumes of data by enabling high-velocity capture, discovery or analysis.
He went on to point out that the traditional structured data component within big data is not new. What is new is the introduction of unstructured data from sources such as sensors and social media. This, together with increased velocity as these data needs to be processed and analyzed in real time, is making big data analytics more complex, the IDC analyst pointed out.
Moreover, new types of infrastructure capabilities and tools to help businesses mine and analyze the data such as Hadoop continue to enter the market, making it tougher for companies to pick the right tools to adopt, he noted.
Demand will be "fierce"
Despite the current market situation, the analyst expects an upsurge in demand for big data skills. Business analytics is becoming an integral part of industry so demand for big data skills will soon increase, he explained.
On the supply side, the number of skilled IT professionals will also pick up, said Carter. He highlighted that in Singapore, for example, more tertiary institutions are including business analytics to cater to the industry's needs.
Remus Lim, country leader, information management, IBM Singapore, concurred. He said the current shortage of big data skills means companies are having difficulties managing and converting data to create "new intelligence" to achieve their business goals in an optimal, cost-effective manner.
That said, large companies are recognizing the challenges and opportunities that come with big data so demand for skilled workers is "sure to grow", Lim surmised.
Michael Driscoll, founder and chairman of Dataspora, a San Francisco-based predictive analytics company, added that demand will be strongest in verticals whose core businesses tend to consume and generate the highest volumes of data. These verticals include the financial services, pharmaceutical and telecommunications, he noted in an e-mail.
As a result, the competition to hire data scientists will be "fierce", he added.
Driscoll explained that data scientists are IT professionals who possess the three skills of "munging, modeling and visualizing". He defined munging to involve the process of cleaning, parsing and proofing the original data before it is suitable for analysis, while modeling refers to building either aggregated or targeted data sets. Visualizing is the process of presenting the insights gleaned from analyzing the data to the company's decision-maker.
Technical attributes aside, Carter suggested that data practitioners also need to move away from traditional methods of data analysis and rethink how they should mine the information, which may or may not be relevant to the company's needs.