The recent debate about News Corp. threatening to leave the Google index, and Google's problems in attempting to index out-of-print books are all related to its commercial status. What if the search index were held by a non-profit? A lot of those probelms would go away. It's interesting to note that Google's founders once believed that search should be non-profit. Take a look at page 39 "Inside Larry and Sergey's Brain" by Richard Brandt (referral link).
Andrei Broder, who led the team that created the AltaVista search engine, the best of its time, talks about meeting Larry and Sergey. "When the discussion turned to the topic of making money from the technology, Broder found that Page had a profound difference of philosophy on the subject. "It was a very funny thing about Larry," Broder recalls. "He was very adamant about search engines not being owned by commercial entities. He said it should all be done by a nonprofit. I guess Larry has changed his mind about that."
Brian Lent, now CEO at Medio Systems, came across the same thing when he met with the Google duo.
"The problem with the Google search engine at the time, Lent recalls, is that Larry and Sergey didn't want to commercialize it, and Lent was anxious to become an entrepreneur. Their mantra at the time was more socialistic than entrepreneurial. "Originally, 'Don't be evil' was 'Don't go commercial,'" says Lent.
When he was at Stanford University, Sergey Brin wrote a paper: "The Anatomy of a Large-Scale Hypertextual Web Search Engine."
In that paper, he argued against an ad-supported service as a corrupting influence. "Advertising-funded search engines will be inherently biased towards the advertisers and away from the needs of the consumers," he wrote.
Could this be a possible future outcome? Could a non-profit search engine "out Google Google?" I think it could.
If site owners blocked all commercial search with a robots.txt file but allowed a non-profit search engine, that would build a vendor-neutral index very quickly.
A non-profit search engine could support itself by licensing the index to various companies -- even Google, so that they can then apply their algorithms to rank the results according to their specialty.
After all, the value isn't in the index it's in how you present results from the index.
Maybe Google could gain everything if it were to spin-off its index into a non-profit? It would fulfill its founders' original ideals and Google could still be Google.
If the index were a commonly-owned resource, every site owner would be able to grant permission to different groups, to access its part of the index.
That's what's missing with the current system. You can either allow a search engine to index your site or not. You have no control over who can access your part of the Google or Bing index.
But if the index were held in common, newspaper publishers and others, would have more options and we could avoid search engine wars and problems with incomplete indexes.