How this retailer uses machine learning and computer vision to keep its shelves full
The Home Depot staff use computer vision to find items on shelves.
When you're a home improvement specialist with thousands of outlets throughout the US, it can be tough to keep track of products across stores and warehouses. Add in the complication of Black Friday and a busy holiday period and the challenge seems almost intractable.
Yet The Home Depot is meeting this test head-on by using a mix of machine learning (ML) and computer vision technology to help staff find products for customers quickly and effectively.
Also: Two breakthroughs made 2023 tech's most innovative year in over a decade
Hari Ramamurthy, technology fellow at The Home Depot, explains to ZDNET in a video interview how this deployment of emerging technology is very much par for the course for the retail giant.
"We are very much a technology-focused company," he says. "We look for ways we can leverage the latest and best technologies to materially improve the experience for our staff and ultimately our customers."
Ramamurthy says The Home Depot has developed an ML-powered app, known as Sidekick, to boost staff productivity.
The app, which also uses computer vision, is installed on "hdPhones", which are mobile devices used by The Home Depot's staff. These devices have been developed in collaboration with Zebra Technologies, HPE, and Aruba.
Sidekick went live at the beginning of 2023 and Ramamurthy says the app is just the latest stage in a range of data-led initiatives across the business.
"Technologies like machine learning or artificial intelligence clearly have tremendous potential in terms of unlocking the right outcomes for our associates and customers," he says.
Also: Generative AI in commerce: 5 ways industries are changing how they do business
When it comes to the development of Sidekick, The Home Depot created a bespoke system that uses a cloud-enabled ML algorithm to allow staff -- whom Ramamurthy refers to as associates --to prioritize important tasks.
The app ensures that associates focus their attention on the most in-demand products and helps them locate items in hard-to-find locations, such as overhead shelves.
"We wanted to make sure our associates were always given the highest value task related to where they are, so they can be productive in the tasks they perform," he says. "We're using multiple signals generated from internal data sources to inform our algorithm."
The ML model takes data from transactional systems, including point-of-sale technologies and inventory management platforms.
However, the model goes beyond traditional structured sources of retail data and draws insight from semi-structured sources, such as video camera feeds that demonstrate the flow of shoppers within stores.
The app also uses computer vision, where images are captured by associates in the Sidekick app on their hdPhones.
Also: These 5 major tech advances of 2023 were the biggest game-changers
Staff members take pictures of locations across the store. The Home Depot uses the data to discover more details about which products are available on the shelves.
"Computer vision is a good example of data that's coming from a non-transactional system and informing our algorithms," says Ramamurthy.
"It's a very exciting technique because we can see there's a lot of information that comes through this stream to augment our data sources. It means we can build a more complete set of signals, and get the appropriate tasks generated and delivered to our associates."
While the app is a data-heavy tool that requires input from staff to work effectively and productively, Ramamurthy says the aim has been to ensure that any demands on staff are not overly onerous -- and that their inputs produce big benefits in terms of outputs.
"Our goal is to make the technologies fade into the background and to be as seamless as possible," he says. "The associates don't really have to understand all the factors that went into play in ensuring that a task was generated. Our objective is simply to try and prioritize the appropriate tasks."
Also: Machine learning helps this company deliver a better online shopping experience
In his position as technology fellow at The Home Depot, Ramamurthy is always looking for ways to both hone the Sidekick app and find other sources of data-led innovation.
"My role is to bridge across our various product teams, business partners, and our technology team," he says. "We're constantly looking for ways to optimize how we perform certain tasks, as well as challenge the way we are thinking. That means considering the introduction of technologies and experimenting in many cases to develop next-generation experiences that make a dent in our customers' problems."
The Home Depot has experimented with various ML and artificial intelligence (AI) techniques for several years, including the home-grown Sidekick app.
Going down the bespoke development route for emerging technology might seem like a significant risk to some digital leaders.
Avivah Litan, distinguished VP analyst at Gartner, has previously told ZDNET that emerging technologies, such as ML and AI, promise big productivity increases, yet there are significant challenges to be overcome before the tools can reap big rewards in an enterprise context.
Also: Nearly 1 in 5 dollars of digital sales during cyber week was influenced by AI
In the case of The Home Depot, Ramamurthy says the company had the in-house talent and proof-of-concept studies to show that ML and computer vision could make a big difference.
The message for other digital and business leaders when it comes to exploiting emerging technology is to focus on testing and honing your approach.
"Our experience has been very iterative. Internally, we think of this as a 'crawl, walk, run approach' to delivering value. We've made tactical improvements and had challenges that we've had to overcome along the way," he says.
"But the iterative approach that we have taken has really helped us ensure we are able to deliver on the expectations. And at this point, we're happy with the results in terms of the performance and the overall experience for the associates."
Ramamurthy and this team continue to look for small iterations that will create big improvements to the Sidekick app.
He believes there's a lot more the company can do to not only ensure it generates the appropriate tasks for staff, but to focus on factors across every facet of the store, whether that's analyzing data from sales locations or considering the layout of the sales floor.
Also: AI in 2023: A year of breakthroughs that left no human thing unchanged
"Those are all areas for further exploration," he says. "In addition, we continue to look at how we can improve our statistical ML models and the quality of some of the tasks that we generate, especially when they're augmented with other signals that come through."
Ramamurthy says he's also keen to use the insight they glean from the Sidekick app to ensure associates have the right skills and resources as they complete their tasks.
"I think those are areas, both in terms of task generation and in terms of task delivery, where there's opportunity for further refinement," he says.