Coles Online finds paradise by the dashboard lite

Coles Online finds paradise by the dashboard lite

Summary: The online retailing arm of Coles has dramatically improved the visibility of its operations with a growing roster of real-time dashboards that have given it unprecedented control over its server loads and customer experience.

TOPICS: Big Data, Australia

Coles Online has been live since 1998, and has been running on its current IT platform since 2008. Yet, despite having installed "heaps of monitoring tools" in the past, staff members were finding it ponderous to find the information they needed.

(Image: David Braue/ZDNet)

"Generating new reports and additional metrics usually required some specialised skills," Coles Online web ecommerce manager Merric Reese told this week's SplunkLive conference in Melbourne. "The tools were very good at providing specific information about the component they were monitoring, but I had to sift through multiple tools and interfaces to find what I was looking for.

"Trying to get an overall view of system performance in real time was extremely challenging. The time required to create new dashboards, or update them, was usually measured in days, and sometimes even in weeks."

Early experiments with the Splunk web analytics tool quickly breathed new energy into the process of monitoring performance, with some tentative new dashboards able to collate data from multiple systems and present it through a series of interactive dashboards.

As a monitoring and diagnostic tool, the new environment provided a one-stop shop for Reese and his team — but it was the cultural change it engendered that got him most excited.

"Being able to do a couple of searches, and get the answer myself in about five minutes, was truly awesome," he said. "The team got really busy creating new dashboards, I got an understanding of the syntax within a couple of hours, and after that there was no stopping us."

The dashboards comb through around 3GB per day of new performance data.

Better visibility of site metrics even proved to be useful in security monitoring and picking up on behavioural anomalies, with real-time visibility of site errors making it easy to spot "a huge number of errors" that were coming into the site. Queries were requesting pages that didn't exist, and after watching the behaviour for some time, Reese twigged to what was going on.

"After doing some particular searches and mapping customer transactions through the process they were doing, it was pretty clear and quite easy to recognise that users were trying to scrape data from the Coles Online website," he said. "We needed a solution to try and prevent this."

Reese set up a regular alert that would email him whenever such behaviour was detected, but this became ponderous, as regular alerts needed to be tended to manually. The development team built a number of scripts that would automatically scan access logs and actively block users who were engaging in that type of suspicious activity. Scripts also track block IP addresses to ensure they are released after a period of time.

By surfacing its performance metrics, Coles Online has been able to turn necessary site monitoring from a chore into something that is proactively helping Reese and his team manage the performance of the site — so much so that the team has been championing its analytics dashboards throughout the Coles Group, and is starting to apply the system to its picking system. Reese expects that additional systems will push the amount of data being processed toward 10GB per day.

"We're getting some great insights out of that, and we're starting to push those reports and data to our business users for store leader boards and those kinds of things," he said. "We've learned that the key is not to let your data be locked away. Even though there are gigabytes of it being generated every day, having a tool that you can use to explore your data in real time is very empowering."

Topics: Big Data, Australia


Australia’s first-world economy relies on first-rate IT and telecommunications innovation. David Braue, an award-winning IT journalist and former Macworld editor, covers its challenges, successes and lessons learned as it uses ICT to assert its leadership in the developing Asia-Pacific region – and strengthen its reputation on the world stage.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • How embarassing

    I'll agree that splunk is a nice product, but you don't need splunk to catch 404's or identify scrapers and 10Gb per day is not "big data".

    While this sounds like a sucess story, anyone that comes from an online services or e-commerce background should be cringing at the state Coles let it's online store get to, and how it was forced to buy an expensive big data tool to do what any pro-active analyst or engineer would have already done with some shell or perl scripts from the handful of logs.

    "It was a cultural change" - exactly.

    Also - graylog2
  • Maybe now Coles can see what the public do for a change.

    Wonder if their dashboard shows how Coles have jacked the price of bread 17c every couple of months for two years.

    Perhaps they can see their $2 milk that's always fixed price going up 30c, then being lowered in two increments to show people they are reducing prices.

    Perhaps they can now calculate their $10 meals actually costing $30 to make, and how it's ruined Kurt and Kathie Freeman's reputation in the community.

    Normal 'non-coles' bread is now up around the $5-6 dollar mark today PER LOAF. When asked why the doubling of the price in 1 year, I was told 'the drivers have to pay their morgagage and are struggling'. Yeah, first time Coles ever cared about drivers. What a lie, even if that was true the drivers wouldn't be struggling at 10k+ per day for deliveries based on the $2 per loaf increase.

    It's time for these 'foreigner executives' to be removed from Coles management. They just care about their paychecks and not the jobs of everyday Australians. The only thing going down down, is the reputation of a once ethically sound company, that now is trying to kill brands that have existed for years so they can put their dirty coles logo on it.

    Don't shop at Coles, you'll pay 15-20% more than other smaller supermarkets on most items.