Will rebels shake the Web in 2001?

Napster is now a veritable grandfather by Internet standards. And the rug rats it spawned may be just as rebellious as the old man. The file-swapping application developed by Shawn Fanning changed the way music is distributed, and the second generation of peer-to-peer programs may similarly change the way people search the Web.

Indeed, the same P2P software that allows users to swap MP3 files is the foundation of several new search engines that have already premiered or will make their debut this year. That has some enthusiasts gushing about how the new technology may help expand the Web's searchable territory 30-fold.

The next generation engines are expected to search the hard drives of members of a Napster-like network of computers for more than just MP3s. The objective: search for everything.

Netscape co-founder Marc Andreessen described one such engine, Pointera, as "chang(ing) the Internet in a way that it hasn't changed since the browser."

This is the first of the so-called "distributed search engines." The Pointera sharing engine, which debuted in mid-2000, is the first to let portals and content site users share files through a standard Web browser. Pointera, a division of Spinfrenzybeing, plans to market the search engine as a way to augment existing searches predominantly in a business environment.

Two other peer-to-peer engines are still in development. One is coming from Gonesilent, the new name of the company founded by former Gnutella developer Gene Kan, who is working on a search engine based on the same file-swapping software at Gnutella.

A version of the search engine is already running. It does things most traditional search engines can't: list stock quotes if a company is searched, show photos, and even solve math problems.

Another is from the Gnutella family itself. The team working on a new and improved Gnutella, right now called gPulp, which developer Sebastien Lambla says can be used for searching, among other applications.

The difference between the traditional search engines and the new generation, is in what, and how much, is being accessed.

Search engines now online, like Alta Vista or Google, use mechanisms to find information then index it for users. Some say they miss about half of the information on the Web, and take their sweet time doing so--anywhere from 24 hours to a month in some cases.

Peer-to-peer search doesn't need to find, retrieve, categorise, and package. It finds what's on another hard drive at the moment the search goes out.

While some in the industry are convinced distributed search is the sliced bread of the year, others think that it's key--a totally open system--may ultimately be its downfall.

"There is a lot of excitement for it because of the success of Napster," said Danny Sullivan, editor of Search Engine Watch. "But I don't see peer-to-peer environment replacing traditional search."

He said that any search engine would tell you it gets spammed as well. Most get their information from individual users who submit it.

"There's a huge industry of people trying to manipulate search engines on the Web," Sullivan noted. "Some will try to direct you to porn sites."

"When you have a system where anyone can contribute, there will be some people who try to manipulate the system," he said.

Alta Vista, for example, discovered that about a million of its pages were bogus listings. It was around the time there were relatively few pages, 20 million, available to Alta Vista users.

"Someone occupied a 20th of their inventory with junk," Sullivan said. "If you don't have guards in place, you are going to open your system up to big trouble."

What do you think? Tell the Mailroom. And read what others have said.

Take me to the MP3 Special

Take me to the Napster Special

Is music file sharing wrong? To have you say online click on the TalkBack button and go to the ZDNet News forum.