A single search index would speed up the entire Internet

I was looking at my server logs the other day and saw that 16 spiderbots visit every day. And it's astonishing how much bandwidth and processing they consume.

They are responsible for more than one third of the hits and consume 45% of total bandwidth!

Multiply this out across the scale of the Internet and it is easy to see that the spiderbots are a huge drain on the resources of the global Internet. All because they are making their own search indexes.

But what if the search index were in the public domain, administered by a non-profit and everyone had easy access to the same data? It would result in an immediate zero-carbon speed boost for very little cost. You'd have to install billions of dollars in new equipment and communications lines to get the same result.

A single search index available to all, would also create a platform for innovation. Startups could develop new types of algorithms and applications without having to go out and spider their own index.

Google's founders originally believed that the search index should be operated by a non-profit. They could spin off their index and still be Google because the value lies in the analysis and the ranking of the results.

A search index operated by a non-profit would alleviate a lot of problems Google is facing in regard to indexing out-of-print books, and also delisting by News Corp and others.

Google's founders would be able to fulfill their ideals, and put their mission to index all the world's information back on track. Plus, the entire Internet gets a huge zero-carbon speed boost. And they get to keep all their money.

How often does anyone get a chance do do something as massive as that?