Google's secret home-brew 10 gigE switch

Google's secret home-brew 10 gigE switch

Summary: Has Google has built its own 10 Gigabit Ethernet switch - one far cheaper and power-efficient than existing switches - as it seeks to widen its infrastructure cost advantage over Yahoo, MSN and AOL?Andrew Schmitt, an analyst at Nyquist Capital makes the case.

SHARE:

Has Google has built its own 10 Gigabit Ethernet switch - one far cheaper and power-efficient than existing switches - as it seeks to widen its infrastructure cost advantage over Yahoo, MSN and AOL?

Andrew Schmitt, an analyst at Nyquist Capital makes the case.

Through conversations with multiple carrier, equipment, and component industry sources we have confirmed that Google has designed, built, and deployed homebrewed 10GbE switches for providing server interconnect within their data centers.

Lean and mean Why does Google care? They've already made major power and cost cuts in their server infrastructure: low-end PC designs; direct-connect disks velcro'd to caseless mobo's; and cheap, unmanaged fast Ethernet switches tying local nodes together.

After cutting the big numbers it is time to cut the smaller ones. They avoid several power and money hungry features of costly enterprise switches. Twinax cabling for short runs. Ancient 850 nm optical interfaces and cabling for longer runs up to about 100 m. No equalization. Inexpensive switch chips from Broadcom.

Why 10 gigE? The Google File System and Bigtable are designed to use cheap, slow networks. That's why there aren't any NAS or RAID arrays in the main Google infrastructure: too much network overhead. Not to mention the cost.

What Andrew doesn't explore is why Google needs 10 gigE. Because it's there? No. They need the scale out.

Scale-out heaven Let's say you had a warehouse with 300,000 Core Duo 2 servers and a rapidly growing workload. Each query slogs through 70-100 MB of data to find your Phoebe Price photos. That traffic is mostly local, but more is on the way.

Machine translation. Cell phone ads. Hybrid maps. Collaboration apps. Bigger social networks. And more.

The 10 gigE switches mean that dozens of gigE uplinks from cheap unmanaged switches can be aggregated on 10 gigE links, increasing the span of 2nd level switches and eliminating the need for a 3rd switch tier. This reduces power, cost and latency, at not much more cost than standard 2nd tier switches today.

The Storage Bits take Google's focus on power is paying dividends for the entire industry today. Their focus on cheap, high-speed networking will also drive 10 gigE down the price curve faster than Cisco and Juniper would like.

At the same time their infrastructure cost advantage will continue to grow. Bad news if you are trying to compete with Google.

Comments welcome, as always.

Topics: Google, Hardware, Networking, Servers, Storage

About

Robin Harris has been a computer buff for over 35 years and selling and marketing data storage for over 30 years in companies large and small.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

18 comments
Log in or register to join the discussion
  • Why would google's custom switch. built for internal use...

    affect the pricing of switches ???

    "he Storage Bits take
    Google?s focus on power is paying dividends for the entire industry today. Their focus on cheap, high-speed networking will also drive 10 gigE down the price curve faster than Cisco and Juniper would like."
    mrOSX
    • Well, if you believe in the law of supply and demand, less demand means,

      lower prices. But, in general, suppliers seeing customers build their own equipment because they can do it cheaper and better should be a pretty big wakeup call that they need to improve and/or simplify the product and lower prices.
      DonnieBoy
      • Google is an outlier

        Within every company the world over people build what they need if what is currently offered on the market is either too expensive, or has too much overhead, or just doesn't do what needs to be done.

        These "solutions" are just a matter of course for companies. For example do you think that every manufacturing company has all the tools they need for a new product line or even an existing one. Check out the Bosch Erector sets, these are available so that people can build their own solution.

        It just so happens that Google's infrastructure is commoditized in that virtually everything they have within their datacenters has been cut down to the bare essentials. They have the cash to do it so why not. I bet if Google could they would build their own power plants; it's just a matter of time.
        THEE WOLF
        • Could not agree more!! Google is also helped by the volumes that they

          require. Yahoo and MS dealing with much smaller volumes, and that "I need a throat to choke" mentality, will have significantly higher costs for building out data centers.
          DonnieBoy
          • Wow. You dragged both MS and Yahoo into the story

            that had absolutely nothing to do with them, and once again you "already know who can build the best and cheapest, ect, ect, ect."

            Keep filling up on the Kool-Aid, little one. :)
            GuidingLight
          • This was all about how Google has lower costs, and guess who the other two

            top search providers are?? Can you say Yahoo and MS??

            But, just shut up and argue your point, if you have one. You are only making a fool of yourself.

            Back on topic, can you argue that MS and Yahoo do NOT have higher costs to build data centers?
            DonnieBoy
      • While I can see why Google does it...

        I dont see a lot of companies out there building their own. I know that last 3 big corps. I worked for did not like home grown stuff, they wanted to have somebody outside the company to hold accountable.
        mrOSX
        • Right, though sounds like Google is focused like a laser beam on reducing

          energy use and costs. The other big thing is that risk-takers are NOT punished within Google if it does not always work. They understand that innovation REQUIRES a lot of risk-taking. So, while others keep it safe and have throats to choke if it does not work, Google quietly reduces costs and increases the rate at which they can build out data centers.

          The other companies can be glad that they played it safe!!
          DonnieBoy
          • My goodnes, you are quite the delusional one anymore

            You have absolutely zero idea what you are talking about, yet you continue to ramble on like you have a clue?

            [i]The other big thing is that risk-takers are NOT punished within Google if it does not always work[/i]

            *Yawn* Typical DonnieBoy Make beleieve

            [i]They understand that innovation REQUIRES a lot of risk-taking.[/i]

            *Yawn* Typical DonnieBoy Make beleieve

            [i] while others keep it safe and have throats to choke if it does not work,[/i]

            *Yawn* Typical DonnieBoy Make beleieve.

            [i]Google quietly reduces costs and increases the rate at which they can build out data centers[/i]

            *Yawn* Typical DonnieBoy Make beleieve

            [i]The other companies can be glad that they played it safe!![/i]

            *Yawn* Typical DonnieBoy Make beleieve

            You have no way of knowing anything that goes on in Google or anywhere else, so where do you get your "insight" from?

            You do not have to answer as I would not want to force you to think up any more fantasies for today.
            GuidingLight
          • It is very well known that to innovate, you need to take risks, it is also

            very well known that Google encourages risk taking and creativity. Google also lets EACH employee spend 20% of their time doing what they want.

            Just argue you point and drop the bull sheeet.
            DonnieBoy
  • Not such a big deal

    You are offering a fairly simplified view that isn't a very accurate comparison.

    The Broadcom chipsets have been out for a while now and they make it easy to create a simple switch that is practically plug and play on any simple single board computer with a PCI bus running vXworks or Linux. Cisco and Juniper units are not simple switches; they offer a lot of functionality that you don???t get with a dumb switch. The functionality is why you purchase a Cisco or Juniper, not the bandwidth.
    balsover
    • Still, this give Google a cost advantage. The switches on the market are

      more complicated than what Google needs and do NOT give them any performance benefit. So, they figure out how to use cheap parts and get the same performance, that is very significant.

      How many switches does Google buy for each data center???
      DonnieBoy
      • Custom hardware costs more than consumer hardware

        The break even point is pretty far down the road, perhaps farther than you think that it is.

        The author feels that Google's custom hardware is going to have some impact on Cisco or Juniper's market which is totally wrong. Like I said, there is much more to it than just a high bandwidth 10 gigE connection. Perhaps that is all that Google needs it for today, but anyone else in the market are going to need those additional features that the Google wirewrap project probably does not support. It's specific to Google's internal needs and has nothing to do with anyone elses needs or market for that matter.
        balsover
        • It does not sound all that custom, just made from standard ships. Broadcom

          did all of the heavy lifting. Kind of like buying motherboards and cases and putting it together. Ok, well more complex than that, but certainly very doable considering the massive volumes that Google needs to aggressively build out data centers. They probably even pay a contract manufacturer to turn them out.
          DonnieBoy
        • Useless features abound ...

          "but anyone else in the market are going to need those additional features that the Google wirewrap project probably does not support."

          Like everyone needs to do mail merging and VBA programming inside Microsoft Word?

          Face it, there are lots of features in commercial switches that a lot of people don't need, especially if the switch is not on an "edge" boundary. Bloat and complexity are becoming big issues, even as customers need more raw bandwidth than they do packet filtering, and so forth.

          I commend Google for being innovative, and I firmly believe that some networking company will take these ideas and run with them.
          terry flores
  • Google's smart

    They do daring things no other company would do in the name of saving money, and it works. Using consumer drives, classic switches, old cabling, basic servers, all things other businesses would never consider but Google does it and has never looked back. Perhaps it's about time for other businesses to consider making these same moves themselves...

    - John Musbach
    John Musbach
  • The Google impact works two ways

    First, the chip makers get extra volume which drives down their costs.

    Second, Google's low cost of operations puts pressure on competitors to do
    likewise. So they put pressure on Cisco, Juniper, et. al.

    And yes, this does affect Yahoo and MSN. Someone suggested that Microsoft give
    content publishers 100% of MSN's AdWords equivalent revenue. But that wouldn't
    work because Google could match them and still make money. They are that much
    more efficient.

    Robin
    R Harris
    • I think there is still some confusion

      Google makes their own switch and does not sell it to other vendors.

      If Google decided to resell the switch, I could see Juniper and Cisco in a world of hurt.
      nucrash