Yahoo, Flickr share how social is improving image recognition, search

Yahoo is using one of its most prominent acquisitions to date as it develops and experiments with image recognition and search.
Written by Rachel King, Contributor

SAN FRANCISCO---Image recognition could be the key to the future of search, but it is proving to be one of the most arduous computing problems to solve.

Yahoo, one of the most storied and popular search engines worldwide, is utilizing one of its more prominent acquisitions, photo sharing service Flickr, to try to get ahead in this race against the likes of Google and Bing.

Senior Yahoo Labs research scientists and Flickr engineers offered a deep dive at the technology company's San Francisco offices on Friday about how Yahoo is experimenting with image recognition.

Flickr itself just celebrated its 10th anniversary. Pierre Garrigues, a senior principal and research engineer for Flickr, posited that while Flickr is now an "old site" and household name in technology.

Nevertheless, how we're taking and sharing digital photos has shifted dramatically in recent history, according to David Amyan Shamma, senior research manager at Yahoo Labs.

Garrigues concurred, pointing toward the vast troves (and value) of the metadata and social interactions increasingly attached to these objects.

"It's not just to surface people with $3,000 digital SLRs," Shamma chimed in.

Yahoo Labs senior research scientist Jia Li asserted Flickr is one of the main destinations to collect data to improve computer vision algorithms.

"When we try to tackle an interesting problem at Yahoo, we're not only using visual signals but multiple kinds of signals to understand images and problems," Li insisted.

Shamma highlighted the Yahoo Weather mobile app, which was given a graphically-rich revamp not long after CEO Marissa Mayer commenced Yahoo's mobile-first era.

The app actually links back to a Flickr pool of weather and metropolitan-related photos, which Shamma compared to any standard group on Flickr -- many of which have been around since the dawn of the photo platform itself.

"People are submitting their weather photos. The editors knew this, and they would reach out," Shamma explained. "If we understand the social model here, people are searching for weather photos online."

Yahoo app developers and engineers did analysis to surface where these photos are and  how they were being judged.

"When we did that, it was purely a social computing effort," Shamma recalled. The result was six million high-quality weather photos, which were scrubbed down considerably based on quality, resolution, and geo-tagging information, amid other qualifications.

With a relatively smaller batch of 1.5 million photos, engineers can then classify photos by attributes (i.e. location, weather, time depicted in the photos) in order to improve image recognition and search algorithms.

Garrigues was quick to point out that many of these photos are not "viral" by any means, suggesting it's much easier for a photographer to have his or her work noticed from a remote area in Norway than downtown San Francisco.

"It's not just to surface people with $3,000 digital SLRs," Shamma chimed in.

Yet all of this feels a bit analog -- if not even overly complex with extra work involved -- for modern search technologies.

Li reiterated Flickr's reliance on "social computing," which she defined as a combination of artificial intelligence and human effort.

"We are at the forefront of having the right data with our users to introduce it in the right way," Garrigues remarked. "The people working at Flickr are nerds, but we're using the technology too."

Garrigues also acknowledged that image recognition and search also has a bit of an awareness problem -- or lack thereof -- which could be stalling the advancement of this technology too. 

"We think there is a missing link there in bringing the technology in and helping people in their lives," Garrigues reflected. 

Flickr currently offers one terabyte of free storage to any and all users, which Garrigues posited could hold a "lifetime of photos." While many of the photos on the cloud-based platform have restricted sharing and downloading privleges defined by their owners, Garrigues stressed the availability of public search too.

Some of the results now populating public search have been equipped with automatic and suggested tags for faster and better-indexed results. 

Also dropping hints of Yahoo's favored mobile-first approach these days, Li noted the increased use of smartphone cameras has helped provide automatic judgment and detection (not to mention metadata and location info) for digital photos too.

Nevertheless, Garrigues acknowledged there are bound to be mistakes given that this is very new, cutting-edge technology. 

Shamma added that the goal -- which might not be achieved for a considerable time -- is to eventually have more of a hands-off, automatic approach than the aforementioned curation tactics.

"We are at the forefront of having the right data with our users to introduce it in the right way," Garrigues remarked. "The people working at Flickr are nerds, but we're using the technology too."

Screenshot via Yahoo Weather

Editorial standards