Linux and Open Source

Dana Blankenhorn & Paula Rooney

Can the Google index go open source without killing it?

By Dana Blankenhorn | July 19, 2010, 5:08am PDT

Summary

Google, Yahoo, Microsoft and others should start negotiating on a statement of principles, explained in code, under which commercial search engines will operate. Governments would be assured that all engines were playing by rules, and they weren’t gaming the system for their own ends.

Topics

Google Inc., Engineering, Open Source, Dana Blankenhorn

Blogger Info

Paula Rooney

Bio
Contact

Vendor HotSpot

Here to help you with your Document Management Needs

Read the DocuMentor blog now

Learn More »

CNET’s Tom Krazit has offered a great run-down of recent moves aimed at forcing Google to give up its search algorithm, and Google’s pushback.

Google open sources more code than anyone else, but it has never revealed how its search engine works.

This is done, it says, to protect the Net from spammers and scammers and others who might game the results. Even with this secrecy, the scams persist. A vast underground industry exists to perpetrate them.

But those algorithms have become so important, critics insist, that Google has gained an editorial power over the whole Internet with them, and so they should be opened to scrutiny.

The controversy picked up steam when a Google Brazil executive told Tom Foremski open sourcing the index might be a good idea. Foremski ran with it, saying that a single index would save an incredible amount of bandwidth.

Giving away Google’s secret source is a bridge too far for me. I don’t think it would eliminate search engine competition in any case, as Foremski seems to believe. He notes some “unknown robot” is currently dragging more bandwidth from him than Google and Yahoo combined.

Still, there is a public interest in all this, and I am in a compromising mood this morning, so let me offer a modest proposal.

An open standard. Google, Yahoo, Microsoft and others should start negotiating on a statement of principles, explained in code, under which commercial search engines will operate.

There would be plenty of room for differentiation under that standard. It would be in every engine’s interests to have such differentiation, for precisely the reasons Google states. But governments would be assured that all engines were playing by rules, and they weren’t gaming the system for their own ends.

I wouldn’t even insist on a transparent process for the development of this standard. It could be done through a series of secret meetings on a high mountaintop somewhere, and government would not need to participate.

Just code some assurances you’re not mucking with the results and go about your business. You’re not being evil anyway. Now make sure your competitors aren’t being evil either.

Kick off your day with ZDNet's daily e-mail newsletter. It's the freshest tech news and opinion, served hot. Get it.

Dana Blankenhorn has been a business journalist for 30 years, a tech freelancer since 1983.

More from “Linux and Open Source”

Open source pulls a Gates in Jobs

NASA gives OpenStack instant credibility

Related Discussions on TechRepublic

Did you know you can take part in these discussions with your ZDNet membership?

Ask a Question Start a Discussion

Talkback Most Recent of 14 Talkback(s)

Follow via:: RSS; Email Alert

Google will never reveal how its search engine works

Google already proved that they are an evil company. They are also proven liars. In addition, their employees can manipulate searches. Their creepy search engine is equivalent to a despicable villain's ray gun. Why would they hand that over?
iPad-awan

(Edited: 07/19/2010 08:07 AM)
- Reply to
- Flag
ZDNet Blogger

I disagree

@iPad-awan I have seen many companies with immense, albeit temporary, power operate in my years in this business. Google has been among the gentlest in its use of power. Compared with Microsoft or Apple or IBM, they're meekness personified.
DanaBlankenhorn

07/19/2010 11:04 AM
- Reply to
- Flag
Well put

@DanaBlankenhorn

Well put
I challenge anyone to find behavior from Google that in any way remotely compares to the behavior from Microsoft which is documented in the following link from the U.S. justice department.

http://www.albion.com/microsoft/findings.html
dfolk2

07/19/2010 11:34 AM
- Flag
RE: Can the Google index go open source without killing it?

@DanaBlankenhorn Yeah, the way they introduced Buzz was - well, it wasn't so gentle/meek. They just flipped a switch and suddenly everybody had to deal with it.

I'm sure that the people themselves inside the company can be gentle/meek, but that rarely translates to the corporate image.

In addition, you totally missed the point - "iPad-awan" wasn't talking about gentleness or meekness at all, you just pulled that out of nowhere to knock down a straw man. "Evil" people and "liars" can certainly work under the guise of meekness. That makes them cunning - it does not make them right.

The most dangerous people are not those who have clearly evil personalities - they will be caught soon enough. The most dangerous people are those who do wrong under the cover of a good personality.
CobraA1

07/20/2010 09:57 AM
- Flag
RE: Can the Google index go open source without killing it?

@DanaBlankenhorn

Wrong Google.

This is the Google that monitors your emails in Gmail to see what you're interested in? That keeps search requests for the lifetime of their servers? That monitors everything that happens in Chrome?

All in the name of targetted ads, Google is a legal spam company. Nothing friendly about them.
Cyberjester

07/27/2010 12:39 AM
- Flag
RE: Can the Google index go open source without killing it?

"Google open sources more code than anyone else": That's not true. They took a lot of Java code from Apache foundation, they took the Linux kernel code, and they did what pleased them with it. Their rate of return to the community is very small in comparison on what they took. At the end, their OS shares less an less with Linux, and their java is not Java anymore. Who is the winner ?
atari_z

(Edited: 07/19/2010 04:00 PM)
- Reply to
- Flag
RE: Can the Google index go open source without killing it?

@atari_z

What exactly does Google owe you? You ask: Who is the winner? How is that Google's problem? They have a business to run. The winner is the person getting search results. No one using Google cares one bit about code that came from the Apache foundation.
croberts

07/20/2010 08:45 AM
- Reply to
- Flag
ZDNet Blogger

No secret sauce...

Dana, thanks for your input. But I'm not asking Google to share its secret sauce, which is its analysis/algorithm. All the robots are collecting the same open information - what's on a web site. Why duplicate that effort 20 times (on my site 20 robots visit daily using 45% of my bandwidth.)

A central open index could be collected by one "openbot" and accessible to any company. That should be cheaper than collecting all that data individually. The analysis of that data is where the secret sauce is.
foremski

07/19/2010 08:48 PM
- Reply to
- Flag
ZDNet Blogger

RE: Can the Google index go open source without killing it?

@foremski Thanks for writing. I was confused a little by the point where you noted some "unknown" robot more bandwidth than the Google and Yahoo robots combined.

Ever figure out who the bad robot was?
DanaBlankenhorn

07/20/2010 12:00 PM
- Reply to
- Flag
RE: Can the Google index go open source without killing it?

@foremski

Interesting idea.

I doubt it would happen, and here's why.

Just a little personal anecdote. I was living in shared housing whilst in uni. I rent a bedroom, share facilities. We each buy our own food. There was, no kidding, 8 bottles of exactly the same milk in the fridge, 8 cartons of margerine, 8 loaves of bread. I moved into a house with some friends.

There's now 2 people in this house, with 2 bottles of the same milk, 2 cartons of the same margerine, 2 loaves of the same bread.

No-one trusts each other enough to co-operate. If a company has an idea to change it's robot, it won't share it to benefit everyone, it'll slap 50 patents on it. You won't get to Google's level by sharing. As much as it would benefit everyone, it just wouldn't happen.
Cyberjester

07/27/2010 12:44 AM
- Reply to
- Flag
Why would you hand over your money maker

The search engine is Google's primary source of income, why would they open source it? For one thing, they do have the largest bandwidth on the planet, the other is that advertising through them is how they make money, which is why everything they offer is "free" for use.
Maarek

07/20/2010 09:50 AM
- Reply to
- Flag
What is the business value in doing this?

Can the author add some addendum to the article to tell us what value the business will get by doing this? That is the part I don't really understand. How is it a win-win-win for everyone?
gudushi.kutushi

07/20/2010 11:25 AM
- Reply to
- Flag
ZDNet Blogger

Credibility

@gudushi.kutushi Having the basic rules be transparent gives everyone the assurance no one is gaming the system. This benefits all players. The biggest player benefits most.
DanaBlankenhorn

07/20/2010 12:01 PM
- Reply to
- Flag
RE: Can the Google index go open source without killing it?

@DanaBlankenhorn

You're assuming the players are all playing fair to start with. :P

There's a game, can't remember which. It's like the police offer. Two criminals are caught, each is given an offer. Turn in the other one and your sentence will be lessened. If they both confess and burn the other, their sentence increases. If one burns the other, their sentence is decreased. If neither confess, their sentence stays the same.

Same = 1
Decrease = 0.5
Increase = 3

It's much more beneficial for both of them to remain silent. But the majority of people in this game will burn the other. We're much too selfish to play fair, and because of which we're too paranoid to trust the other, thinking they'll do us in before we can do them in.
Cyberjester

07/27/2010 12:47 AM
- Flag