PC Forum: Searching the dark matter and users

Esther started off the PC Forum panel with search stars, by saying the search is getting boring and has peaked to get some reaction. They all properly disagreed and the discussion focused on the amount of information that isn't indexed or searchable.
Written by Dan Farber, Inactive

Esther started off the PC Forum panel with search stars, by saying the search is getting boring and has peaked to get some reaction. They all properly disagreed and the discussion focused on the amount of information that isn't indexed or searchable.

Rich Barton, CEO of Zillow (the hot real estate search service) and former CEO of Expedia, described the universe of information in cosmological terms. "The energy in the visible universe constitutes 4 percent. Search so far has found the visible universe, but 96 percent of the interesting knowledge is yet to be discovered and found," Barton said. The data sits in peoples' heads, databases, government offices, closets, etc. "Lighting up the dark matter is going to be one of biggest drivers in next step of search," Barton added.


Omid Kordestani, Google; Rich Barton, Zillow; Ellen Siminoff, Efficient Frontier; Jeff Weiner, Yahoo; Esther Dyson

Jeff Weiner, senior vice president of search and services at Yahoo, did some back-of-the-envelope calculations on the size of the dark data matter. With a global population of about 7 billion, and each person having tens of thousands of documents, Weiner came up with 350 trillion objects (50,000 per person and not all are of equal value), which equates to .0058 percent of world's knowledge indexed, he said, based on the comparison to the current indexed Web. Weiner also pitched Yahoo's focus on tapping into its user population to improve the search experience, with both implicit and explicit data. In terms of explicit data, "what the user puts into something has to be exceeded by what comes out."

Yahoo is intensely focused on figuring out how to make its more than 400 million users contributors to the search quality cause. According to Weiner, the notion of search as the 'great democratizer' is a misnomer. "Search is the tyranny of the Web master, Weiner said. The only people getting into search indices are those sophisticated enough to get into a search index--they only can generate relevancy by incoming links, and there are a number of people for whom that doesn't apply. Yahoo is using social bookmarking, tagging, Yahoo Answers and other sources of user inputs to push its search forward.

Omid Kordestani, senior vice president of global sales and business development at Google, talked about the search engine's approach to indexing the world's information and how search is more than PageRank, which was implemented n 1998. "Personalization technology is just another signal in the way we analyze information." 


Seachers: Omid Kordestani, Google, and Jeff Weiner, Yahoo

WhenU CEO Bill Day asked about the search players, Microsoft in this case, providing financial incentives to gain users. Kordestani said purely financial incentives aren't successful. "Making music free [as an incentive] could disrupt the [search] industry, but innovation needs to be meaningful. Many try to get you to use a service, giving you more bells and whistles, but at the end of the day it's a more complicated experience. It's not the purity of having a product work well for you. We haven't seen it impacting our activities. 

"One would think that in profit driven culture, it would lead to right behavior, but it doesn't work. It changes the way people interact with one another," Weiner said.  He claimed that a radical change is taking place on Web with people sharing and participating, and in return becoming authorities and gaining respect. "It's not as simple as providing economic incentive. It will be interesting to see how it plays out. It's a multivariable equation." Yahoo is more focused on personal and social incentives in a community context.

Ellen Siminoff, CEO of search engine marketer Efficient Frontier, said that some incentive models work, such as frequent flier programs. However, incentive programs that pay people everytime they come to a site won't work. "Free music for life gets into an economic model that's not going to work over time."

The panelists were asked about breaking out of the box that has dominated search interface. "UI [user interface], relevancy, personalization, customization tools--we have this concept of one box--put in an equation or flight number and get answers you wanted. The UIs are going to transform. We are thinking about the experience of users and getting to the right answer as efficiently as possible," Kordestani said. 

Barton described how the established companies can get paralyzed over making even minor interface changes, which he said leaves the door open for other players to innovate. Zillow conveniently fits into that mold.

Vertical search, more contextuality, surfacing more metadata and meme tracking were also discussed as areas in which Web search is heading. No mention of the Semantic Web or efforts to provide more structure to the content on pages that would make their crawlers far more effective and search results more relevant.

Esther suggested that Google and Yahoo and others doing search business in China get together and go to the Chinese government. "Institutions should be willing to give you answers," Esther said.

Finally, Seth Goldstein of Root Markets, which allows users to store and share their Web trails (data streams), asked Kordestani and Weiner whether users own the rights to the data trails they leave on Google and Yahoo. Weiner agreed that the users have rights to their data. Kordestani was non-commital. "If it makes sense, we should probably allow it," he said.

Editorial standards