The magic that makes Google tick

The magic that makes Google tick

Summary: Google's vice-president of engineering was in London this week to talk to potential recruits about just what lies behind that search page. ZDNet UK snuck in to listen

SHARE:
TOPICS: Servers
16
The numbers alone are enough to make your eyes water.

  • Over four billion Web pages, each an average of 10KB, all fully indexed
  • Up to 2,000 PCs in a cluster
  • Over 30 clusters
  • 104 interface languages including Klingon and Tagalog
  • One petabyte of data in a cluster -- so much that hard disk error rates of 10-15 begin to be a real issue
  • Sustained transfer rates of 2Gbps in a cluster
  • An expectation that two machines will fail every day in each of the larger clusters
  • No complete system failure since February 2000

It is one of the largest computing projects on the planet, arguably employing more computers than any other single, fully managed system (we're not counting distributed computing projects here), some 200 computer science PhDs, and 600 other computer scientists.

And it is all hidden behind a deceptively simple, white, Web page that contains a single one-line text box and a button that says Google Search.

When Arthur C. Clarke said that any sufficiently advanced technology is indistinguishable from magic, he was alluding to the trick of hiding the complexity of the job from the audience, or the user. Nobody hides the complexity of the job better than Google does; so long as we have a connection to the Internet, the Google search page is there day and night, every day of the year, and it is not just there, but it returns results. Google recognises that the returns are not always perfect, and there are still issues there -- more on those later -- but when you understand the complexity of the system behind that Web page you may be able to forgive the imperfections. You may even agree that what Google achieves is nothing short of sorcery.

On Thursday evening, Google's vice-president of engineering, Urs Hölzle, who has been with the company since 1999 and who is now a Google fellow, gave an insight to would-be Google employees into just what it takes to run an operation on such a scale, with such reliability. ZDNet UK snuck in the back to glean some of the secrets of Google's magic.

Google's vision is broader than most people imagine, said Hölzle: "Most people say Google is a search engine but our mission is to organise information to make it accessible."

Behind that, he said, comes a vast scale of computing power based on cheap, no-name hardware that is prone to failure. There are hardware malfunctions not just once, but time and time again, many times a day.

Yes, that's right, Google is built on imperfect hardware. The magic is writing software that accepts that hardware will fail, and expeditiously deals with that reality, says Hölzle.

The problem
Google indexes over four billion Web pages, using an average of 10KB per page, which comes to about 40TB. Google is asked to search this data over 1,000 times every second of every day, and typically comes back with sub-second response rates. If anything goes wrong, said Hölzle, "you can't just switch the system off and switch it back on again."

The job is not helped by the nature of the Web. "In academia," said Hölzle, "the information retrieval field has been around for years, but that is for books in libraries. On the Web, content is not nicely written -- there are many different grades of quality."

Some, he noted, may not even have text. "You may think we don't need to know about those but that’s not true -- it may be the home page of a very large company where the Webmaster decided to have everything graphical. The company name may not even appear on the page."

Google deals with such pages by regarding the Web not as a collection of text documents, but a collection of linked text documents, with each link containing valuable information.

"Take a link pointing to the Stanford university home page," said Hölzle. "This tells us several things: First, that someone must think pointing to Stanford is important. The text in the link also gives us some idea of what is on the page being pointed to. And if we know something about the page that contains the link we can tell something about the quality of the page being linked to."

This knowledge is encapsulated in Google's famous PageRank algorithm, which looks not just at the number of links to a page but at the quality or weight of those links, to help determine which page is most likely to be of use, and so which is presented at the top of the list when the search results are returned to the user. Hölzle believes the PageRank algorithm is 'relatively' spam resistant, and those interested in exactly how it works can find more information here.

Topic: Servers

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

16 comments
Log in or register to join the discussion
  • Hello
    One thing that ticks me off about google is that it desplays multiple results on a search query from the same web site.
    How hard can it be to limit the exposer to one url per query?
    Is it logical to have a website show up hundreds of times on a search query?
    Try searching for the word "algarve" (region in Portugal).
    I think it is googles mission to deliver me to a website where I can find the content I am looking for, and not to worry about holding me up to reach the cookie jar and sink it down my mouth.
    Show me where the supermarket is, I will find the shelf!
    Thank you.
    Keep clicking...

    Hilton Santos
    www.polorentacar.com
    hilton@polorentacar.com
    anonymous
  • Dude... don't know what you're talking about. Tried your search and got 10 results for 10 different websites listed. Google DOES eliminate results that are alike.
    anonymous
  • Hi Carlos
    Thank you for your feedback.
    Try google.co.uk
    Search for "algarve car hire"
    You will notice as I have (saved the results in the case you would like me to send them to you)
    that the same website:
    www.portugal-linha.pt
    comes in at 2nd and 3rd position, if you continue down the list you will find it time and time again.
    Are the results relevant?
    Well... I guess one of the results is, all others are repeats, that point to the same web page...
    hilton@polorentacar.com
    anonymous
  • Open letter to Google.

    Hello my dear Googlers

    My comment has nothing to do with google suggest, I find it very useful.
    Now what really squeezes the sh.. out of me is ONE THING and this is it:

    Why the hell do you allow a website to show up multiple times on one search query?

    Why GOOGLE? Why?

    It seams so simple to deal with, and trust me when I say, that I AM NOT A GENIUS!

    Is there not a way of eliminating, the multiple exposer of a given website by means of it
    anonymous
  • QUOTE FROM PREVIOUS POST

    "Now please excuse me if I sound dramatic, but, HOW HARD can it be to eliminate the following results after the first one has been shown?"

    typical question from someone who has no idea what they are asking for.

    HOW HARD can it be?
    have you tried doing this? Have you tried to put up thousands of computers and make a "slight" adjustment on a program running on the whole network?
    If you have not tried doing it, dont pretend to understand how it works. You have no idea what you are asking for. You can probably send some money to Google so they can have the resources to process your request... Its not like you actually paid Google to do the search for you right? You want a personalized search engine? Go make one yourself. It's relatively easy to do... well, that is, if you have the brain for it.
    anonymous
  • Very dear Frustrated Research and Development specialist

    I am sorry you feel this way...
    Why don
    anonymous
  • Oh Wilton, Alem de parvo es muito burro!!!!
    Epa why Google?? mesmo de puto.

    j
    anonymous
  • And he calls himself development specialist ...
    :-)
    Some guys are so stupid !!!
    anonymous
  • Sou puto mas tenho colh... para assinar o que escrevo...
    anonymous
  • Dear Hilton,

    First of all, please forgive me for the bad english.
    Good afternoon. I don't think very ethical from your part to criticize other's websites listed on google. If you want to criticize google then e-mail them with general matters. Never with particular cases. I think that might not be the best way to gain customers or positions on google. I know several websites, experts in many matters that only show up on page 14 on google and on yahoo thay're on page 1. Can you imagine. I hope you try to compete with other companies in defferent ways then criticizing their marketing on the internet. Criticize Google if you want, but never pointing your finger to a particular website.
    anonymous
  • I do not have time for people who do not assume there identity....
    You are either born with balls or they grow on you...
    anonymous
  • anonymous
  • anonymous
  • October 27 Today's Hand-picked Gallery (Must See!)
    October 27 : cute 19 years old aileen showing her natural boobs (Amateurs)
    October 27 : watch as she gets her breasts examined (Amateurs)
    October 27 : the doctor then gives her a gyno and rectal exam (Amateurs)
    October 27 : sativa rose screwing two well hung men (Blowjob)
    October 27 : dirty housewife desirae stripping from her jeans (Amateurs)
    October 27 : little freaky chick plays with her delicious body parts solo (Hairy)
    October 27 : very hairy blonde chick exposes her fury one in the tub (Hairy)
    October 27 : milf's in a lil'cheating (Housewifes)
    October 27 : flower panties amateur stripping (Blonde)
    October 27 : sexy black slut banged by two big white cocks (Black and Ebony)
    October 27 : petite brunette invites her friend over for a little didldo fun (Dildos)
    October 27 : lucky guy fucking two busty blondes (Gangbangs)
    October 27 : blonde and redhead lesbians fucking (Babes)
    October 27 : lays on the table and touches herself (Pussy)
    October 27 : horny busty granny getting her pussy pounded in a threesome (Gangbangs)
    October 27 : gorgeous big tit latina babe loves strip teasing in bikini (Babes)
    October 27 : big tits blonde babe in blue underwear stripping (Babes)
    October 27 : lucky guy fucking a girls ass with a dildo (Anal)
    October 27 : kinky contortionist using an anal sex toy (Anal)
    October 27 : two sexy lesbians playing with toy (Dildos)
    October 27 : two teens getting pierced and tortured (Amateurs)
    October 27 : sexy brunette teen shows her pussy (Brunettes)
    October 27 : naughty blonde gets fucked hard and squirts her cum everywhere (Amateurs)
    October 27 : sexy goth babe with a black umbrella (Amateurs)
    October 27 : crissy moran taking off sexy dress (Beautiful)
    October 27 : insatiable housewife takes on three guys (Gangbangs)
    October 27 : teen redhead slut gives a deep throat blowjob (Blowjob)
    October 27 : amateur teen sucking n gets hard fucking in studio (Amateurs)
    October 27 : gorgeous big breasted milf stripping naked on the couch (Amateurs)
    October 27 : anita dark dressed like a sexy pirate (Beautiful)
    October 27 : all natural busty babe roxy carter spreads her pussy (Babes)
    October 27 : busty natulal teen showing virgin pussy in forest (Beautiful)
    October 27 : teen babe in hard sex act (Blowjob)
    October 27 : emily18 looking mighty hot (Beautiful)
    October 27 : purrfect sadie in panties (Beautiful)
    October 27 : busty teen dildoing her pussy hard with a big toy (Dildos)
    October 27 : blonde tanlined mature housewife doing your laundry (Blonde)
    October 27 : carmel moore gets cum on her tits (Hardcore)
    October 27 : blue eyed mature vixen stuffs dildos into both her mature holes (Dildos)
    October 27 : two seductress want to whack some balls (Babes)
    October 27 : blonde with glass toy (Babes)
    October 27 : helpless gagged girl bondaged and fucked with device in maledom (BDSM)
    October 27 : two sexy lady playing with waiter slave game (Babes)
    October 27 : cute sexy babe does handjob sex hard and blowjob (Amateurs)
    October 27 : busty blonde marketa showing her pussy and tits (Pussy)
    October 27 : hottest bare chick hard bondaged and fucked by master (BDSM)
    October 27 : sexy naked phone call (Babes)
    October 27 : penny flame exposes her tits and ass (Babes)
    October 27 : anime church orgies with priests and nums sex (Anime)
    October 27 : brunette teen takes off panties and spreads hairy pussy (Hairy)
    October 27 : cute brunette babe sucks huge cock and gets fucked on bed (Babes)
    October 27 : attractive brunette with jumbo juggs strips out of lingerie (Beautiful)
    October 27 : wild hardcore groupsex party (Anal)
    October 27 : toying cute lesbian couple (Babes)
    October 27 : slim and smalltittied indian teen (Brunettes)
    October 27 : lesbian anal sex with strapon dildo shoved in asshole (Anal)
    October 27 : sexy skillful feet making a guy cum all over them (Babes)
    October 27 : busty asian babe in pink panties shawing off ripe ass (
    anonymous
  • Welcome to www.polorentacar.com

    You will find everything you need RIGHT HERE, and probably on this page.!
    Don't go bookmarking 20 different sites. Bookmark this one!

    Why? Simple! because we have it all... TGP Galleries, Free Sites, AVS Sites, Movie Galleries, Sex Contacts, E-cards and every bloody thing else.

    You will find all the new stuff on this page and the rest in the categories.



    Dave's Bullshit of the Day/Week/Month/Year
    Sep 24th - News - Server Fixed - Everything is now running fast and smoothly
    Sep 22/23 - News - Slow Server - Network Down - Part of the network that serves DangerDave.com.au and our host is on the fritz
    Sep 9th - Cards - New DangerCards added - Kissing Girls - Send one to a friend today
    Sep 8th - News - New Categories added - Lesbian Strapon, Girls Kissing - Amy Reid
    Aug 9th - Blog/News - Threats to your internet - Burglars! - Porn at "work" and Would you believe
    Jul 25th - Blog/News - New Categories - New categories include - College Coed Sex, Nude College Coeds. We have a new daily fix too.. our Babe of the Day. For those that like porn trailers, we have a new category just for trailers - Reality Porn Trailers
    July 18th - Blog/News - Dell Porn - Dell's new blog has a clone!
    Apr 22nd - Cards - New DangerCards added - Nude Cheerleaders - Send one to a friend today
    Apr 12th - News - Yep we turned the bullshit upside down.. to match the rest of the site :)
    Mar 30th - Blog/News - New Categories - New categories include:- Pussy Closeups, Thai Girls, and Doggy Style.

    Note - Gay links tagged as - Gay, Shemale links tagged as She, For Women links as fW, and Bisexual links as Bi
    anonymous
  • Hello One thing that ticks me off about google is that it desplays...

    agreed can be a bit of a pain
    Trust me i can help