Facebook develops new AI techniques to one day make "anything shoppable"
Facebook on Tuesday shared details of new artificial intelligence techniques it has developed to ultimately make "anything shoppable" -- advances in item segmentation, detection and classification to improve the way people buy, sell and discover items across Facebook platforms. The long-term goal, Facebook said in a pair of blog posts, is to create a holistic AI-powered system to enable frictionless consumerism.
"Our long-term vision is to build an all-in-one Al lifestyle assistant that can accurately search and rank billions of products, while personalizing to individual tastes," one blog post says. "That same system would make online shopping just as social as shopping with friends in real life. Going one step further, it would advance visual search to make your real-world environment shoppable. If you see something you like (clothing, furniture, electronics, etc.), you could snap a photo of it and the system would find that exact item, as well as several similar ones to purchase right then and there."
Universal product recognition
In pursuit of that vision, Facebook is deploying a new, universal computer vision system called GrokNet. This new product recognition model was designed with the intent to make "virtually any photo shoppable."
GrokNet currently powering features for buyers and sellers in Marketplace, Facebook's peer-to-peer shopping platform. When a seller uploads a photo to Marketplace, the system automatically suggests attributes to list, such as the item's color or material. For buyers, the system allows you to conduct specific searches -- such as "black leather sectional sofa" -- and find what you're looking for, even if your search terms didn't match the seller's product description.
Facebook explains that it needs a universal system for its platforms, since millions of users are posting listings across dozens of categories. Most product recognition models are designed for specific product verticals, such as furniture or fashion. Adding to the challenge, Facebook wanted to build a system that serves all countries and users, regardless of language, cultural differences, age, socioeconomic class or other factors.
The new system can identify fine-grained product attributes across billions of photos, Facebook says, across several categories. It's 2x more accurate than Facebook's previous product recognition systems.
In addition to deploying it in Marketplace, Facebook is using it to test automatic product tagging on Facebook Pages. This should help small businesses market their products more easily while making it easier for consumers to find products they like. Facebook also plans to use it to power Shops, so customers can get personalized suggestions from businesses on the products most relevant to them.
Advances in segmentation
To build a unified model, Facebook had to advance segmentation -- the computer vision process of identifying which pixels belong to which objects. Clothing can be especially hard to identify, since it's often layered or hidden behind hair.
Facebook developed a new clothing parsing technique called Instance Mask Projection, which it says is the first system that can predict obstructed or layered objects in photos, like a shirt beneath a jacket. The company says it's achieved more than 80 percent accuracy.
To identify an object, the system first predicts a box around each item of clothing. Once it has that rough segmentation, it precisely labels each pixel.
3D views on Marketplace
Facebook is also improving the quality of images on Marketplace with a new feature that allows any seller with a camera phone to turn a 2D video into a 3D-like interactive view. The company says Marketplace is the first person-to-person commerce platform to offer this automatic video stabilization technique. Currently, Facebook is testing it on Marketplace for iOS.
The feature allows viewers of the image to spin it and move it up to 360 degrees to get a complete view of the object for sale. It's designed to work without requiring the seller to edit or reformat their video, and it should work in low lighting or even if the object is partially obscured.
To create the camera angles in a 3D space, Facebook uses the visual-inertial simultaneous localization and mapping (SLAM) framework, which is commonly used in virtual reality settings.
Moving into e-commerce with Facebook Shops
The advances in AI fit in with Facebook's new push into e-commerce. Also on Tuesday, the company announced Facebook Shops -- a way for businesses to set up a digital storefront where customers can browse and purchase items directly in Facebook or Instagram.
Users can find Facebook Shops stores on a business' Facebook Page or Instagram profile, or discover them through stories or ads. In the future, the tool will be extended to other Facebook platforms, enabling customers to browse and make purchases from within WhatsApp, Messenger or Instagram Direct.
Facebook is also adding new promotional tools for brands, such as the ability to tag products and feature them at the bottom of live videos on Facebook or Instagram. The company is also helping businesses build customer loyalty programs from their Facebook accounts.
Last month, as Facebook released its first quarter financial results, CEO Mark Zuckerberg highlighted how the company has taken steps to support small businesses during the COVID-19 pandemic, as well as the products it's developing for SMBs. Small businesses that have already invested in their digital presence, he said, "are increasingly viewing them as their primary storefronts. So we're working on a number of ways to deepen this experience, helping people buy items and services directly within our apps... Overall, though, our business depends on the success of small businesses, so this is a moment where we feel that we're well positioned to be champions for small business' interests and supporters of important infrastructure that they're going to need in order to move online."