Enlist big-data rock stars to shift analytics ownership from IT to business

If you’re introducing big-data analytics into your organisation, don’t be afraid to push it hard – and enlist the help of rock stars.

That was the advice from Venugopal Adooparambil, director of big data consulting with Optum, who this week spoke at a Teradata analytics forum to share his experiences implementing a massive data-analysis system for 3400-store supermarket chain SuperValu.

Adooparambil became involved in the process in a consulting capacity as the company looked for ways to get more value out of its extensive data on customer purchases and behaviour. The company had over 13 years’ worth of historical data on tape – “they didn’t have any other space to keep it,” he said – but had run into a wall in figuring out what to do with it.

Recognising that the Apache Hadoop data-analytics platform offered a way to get the data into a state where it could be more readily analysed, Adooparambil and his team proposed a six-month process by which the data would be reworked into an analytics platform to help the company better understand customer behaviour.

Executives loved the idea, but gave the team just two months to complete the data migration – and that’s when the rock stars were enlisted.

“The short deadline threw us into a scramble,” Adooparambil explained. “We knew if we were going to finish it in two months we would need a rock-star team with defined roles and responsibilities, independent vendors, and clear business use cases.”

Where have all the rock stars gone?

Finding those ‘rock stars’, however, proved difficult – not the least because SuperValu is based in the northern US city of Minneapolis, whose frigid weather and distance from tech hotspots makes it a less than high-demand destination for skilled IT practitioners.

“We didn’t have data scientists and people who were big-data and Hadoop savvy,” Adooparambil said. “They all want to go to work at Google. But what we did have was people who were great at SQL.”

That led the team to carefully consider its path forward: rather than bringing in big-data experts, it was decided to direct the company’s existing experts and use the Teradata-owned Aster Data analytics platform to make use of the data platform in the long term.

“If you’re going to succeed in big data, you need a thick skin, because you’ll get a lot of pushback. You need to define clear roles and responsibilities if you’re going to do it.”

That ETL process was only the beginning, however: the Aster Data implementation took around three months, during which a major effort was undertaken to encourage business leaders to take on the responsibility for utilising the new data platform.

Making this happen, unsurprisingly, was less than simple and required strong support from the company’s senior executives.

“You have to try to get leadership on board,” Adooparambil advised. “It’s a new paradigm for everybody, and when you’re talking about ‘everybody’ you’ve got to find people in IT, architecture, and databases.”

“If you’re going to succeed in big data, you need a thick skin,” he continued, “because you’ll get a lot of pushback. You need to define clear roles and responsibilities if you’re going to do it.”

Customer behaviour, by the numbers

The business value of the platform soon became clear, as reports run against the massive historical data set turned up stock-analysis results that were orders of magnitude faster than was possible in the past.

One proof-of-value exercise, designed to process historical sales of a particular product in order to project future demand across the company’s stores, delivered “phenomenal” speed improvements.

One report had previously taken 4 hours to run against just 13 weeks’ worth of data, but with the Hadoop and Aster Data platform in place the same report completed in just 2.2 minutes.

One report had previously taken 4 hours to run against just 13 weeks’ worth of data, but with the Hadoop and Aster Data platform in place the same report completed in just 2.2 minutes.

Proofs of concept proved that the technology demonstrated the system’s value, but the IT team quickly stepped out of the way and handed over responsibility for driving the platform to its new roster of big-data ‘rock stars’ and business leaders.

By empowering those leaders with powerful tools that let them “fail fast” and try new methods without ridiculous time delays, the IT team was able to built a big-data culture that has helped it build a culture of shared responsibility.

“The data scientists should be able to manage their system,” Adooparambil said. “IT provide the transportation layer to help them load data fast, but we worked to build best practices, standards and governance within a business partnership to maximise business self-service.”

Adooparambil’s comments about analytics skills challenges echoed those of Teradata CTO Stephen Brobst, who in his earlier presentation noted that “getting the right analytics cluster running requires a different set of skill sets than most people have in their organisations today. Hadoop’s licensing cost is zero but it’s still important to get the right balance to understand where the value is being created.”

By working hard to delineate responsibilities and promote internal capabilities, the SuperValu implementation team was able strike that balance, building up an internal skills base that effectively gave the company a new team of big-data rock stars.

The Yammer social-media platform was used to encourage group thinking within the company, while an external portal provided support when things got sticky.

“It is always an uphill battle when you do something new, and if you want the business side to work you want a sponsor that can evangelise that data,” Adooparambil said. “We talked to some of the rock stars and said ‘this is something that will be useful for your career, and here’s your value. You’ve got to sell them on it first.”