Between the Lines

Larry Dignan, Sam Diaz, Andrew Nusca
Home / News & Blogs / Between the Lines

Cloudy day: Google falters; Packets lost in key cities

By Larry Dignan | May 14, 2009, 8:25am PDT

Summary

Updated: Here’s how Google’s outage unfolded on Thursday… 
We’re getting various reports via that Google services are down or at least sucking some serious wind. The service appears to be back as of noon-ish EDT in New York City. Google cites a “traffic jam” in Asia for the outage. 
The fail whale Google style:

Initial pings on Google.com [...]

Topics

Blogger Info

Larry Dignan

Biography

Larry Dignan

Larry Dignan

Larry Dignan is Editor in Chief of ZDNet and SmartPlanet as well as Editorial Director of ZDNet's sister site TechRepublic. He was most recently Executive Editor of News and Blogs at ZDNet. Prior to that he was executive news editor at eWeek and news editor at Baseline. He also served as the East Coast news editor and finance editor at CNET News.com. Larry has covered the technology and financial services industry since 1995, publishing articles in WallStreetWeek.com, Inter@ctive Week, The New York Times, and Financial Planning magazine. He's a graduate of the Columbia School of Journalism and the University of Delaware.

For daily updates, follow Larry on Twitter.

Sam Diaz

Biography

Sam Diaz

Sam Diaz

Sam Diaz is a senior editor at ZDNet. He has been a technology and business blogger, reporter and editor at the Washington Post, San Jose Mercury News and Fresno Bee for more than 18 years. He's a member of the National Association of Hispanic Journalists and a graduate of California State University, Fresno.

Andrew Nusca

Biography

Andrew Nusca

Andrew Nusca

Associate Editor

Andrew J. Nusca is an associate editor for ZDNet and SmartPlanet. As a journalist based in New York City, he has written for Popular Mechanics and Men's Vogue and his byline has appeared in New York magazine, The Huffington Post, New York Daily News, Editor & Publisher, New York Press and many others. He also writes The Editorialiste, a media criticism blog.

He is a New York University graduate and former news editor and columnist of the Washington Square News. He is a graduate of the Columbia University Graduate School of Journalism. He has been named "Howard Kurtz, Jr." by film critic John Lichman despite having no relation to him. A native of Philadelphia, he lives in New York with his fiancee and his cat, Spats.

Follow him on Twitter.

Vendor HotSpot
Here to help you with your Document Management Needs

Read the DocuMentor blog now

Learn More »

Updated: Here’s how Google’s outage unfolded on Thursday… 

We’re getting various reports via that Google services are down or at least sucking some serious wind. The service appears to be back as of noon-ish EDT in New York City. Google cites a “traffic jam” in Asia for the outage. 

The fail whale Google style:

Initial pings on Google.com show packet losses in New York, San Francisco, Chicago, Australia, France, China and other locales. The outages ran across all of Google’s properties such as YouTube, Gmail, Google Analytics, Google Maps and Google Docs. The Twitter Gmail-o-meter for Google fail and Gmail is running amok. Also see the Internet Storm Center and Stephen Shankland and Techmeme. 

Also see: Google’s App Status dashboard.

Here’s the report from via Just Ping:

A few minutes later it appears some places are recovering, but the packet losses are rotating to new areas. We’re nearing the half hour mark of Google’s troubles. 

Update: In the stray email department, I was informed that it’s an AT&T routing issue. Anything that touches Google via AT&T is down. Trying to confirm that now so take it for what it’s worth.

Here’s the traceroute pointing to AT&T:

Update 2: Got some our resident IT guru to explain this in English. It does appear Google is stopped at the AT&T border. Note: This may just be New York City specific. Here’s the diagram:

Update 3: A reader reports:

Enterprise gmail, applications and regular gmail is down in Los gatos CA…almost all morning, what a disaster, we run our business on it. 

Update 4: Google appears to be back. Outage lasted about an hour from New York City. However, the packet losses are still rotating:

Update 5: Not convinced this was an AT&T specific issue judging from the talkbacks here and here. In any case, other sites worked fine for me through the entire Google outage. I assume we’ll get explanation at some point soon. 

Update 6: CNet News’ Shankland has two interesting nuggets. First the Google statement: 

“We’re aware some users are having trouble accessing some Google services. We’re looking into it, and we’ll update everyone soon.”

Gmail is reportedly all clear. Meanwhile, Keynote is showing packet losses at NTT and Qwest. That fact means it’s more than just AT&T at work behind the Google issues. 

Update 7: Google’s App Status dashboard is handy. Here’s the Gmail information as of 1:13 p.m. EDT:

Update 8: Google gets to Twitter a bit late, let’s call it 1:20 p.m. EDT. 

Update 9: AT&T says via Twitter that it’s not responsible for the Google outage.

The telecom giant also issues the following statement:

 

After receiving speculative reports in the media that Google experienced  an outage related to the AT&T network, we looked into the matter.  We have not identified any specific problems in our network that could have caused the reported outage.

Update 10: McAfee argues that the Google outage was due to a IPv6 upgrade, reports CNet News’ Tom Krazit. Google isn’t elaborating in its blog post. Google explains:

Imagine if you were trying to fly from New York to San Francisco, but your plane was routed through an airport in Asia. And a bunch of other planes were sent that way too, so your flight was backed up and your journey took much longer than expected. That’s basically what happened to some of our users today for about an hour, starting at 7:48 am Pacific time.

An error in one of our systems caused us to direct some of our web traffic through Asia, which created a traffic jam. As a result, about 14% of our users experienced slow services or even interruptions. We’ve been working hard to make our services ultrafast and “always on,” so it’s especially embarrassing when a glitch like this one happens. We’re very sorry that it happened, and you can be sure that we’ll be working even harder to make sure that a similar problem won’t happen again. All planes are back on schedule now. 

Update 11: Fairly or not, Google’s outage will be a reminder that cloud providers can go down. Will companies be prepared for outages? Sam Diaz argues Google’s outage isn’t enough to convince folks to avoid the cloud. But it certainly doesn’t help. 

Update 12: Arbor Networks has a great chart on the GoogleLapse.

 

Larry Dignan is Editor in Chief of ZDNet and SmartPlanet as well as Editorial Director of ZDNet's sister site TechRepublic.

Disclosure

Larry Dignan

Larry Dignan has nothing to disclose. He doesn’t hold investments in the technology companies he covers.

Biography

Larry Dignan

Larry Dignan is Editor in Chief of ZDNet and SmartPlanet as well as Editorial Director of ZDNet's sister site TechRepublic. He was most recently Executive Editor of News and Blogs at ZDNet. Prior to that he was executive news editor at eWeek and news editor at Baseline. He also served as the East Coast news editor and finance editor at CNET News.com. Larry has covered the technology and financial services industry since 1995, publishing articles in WallStreetWeek.com, Inter@ctive Week, The New York Times, and Financial Planning magazine. He's a graduate of the Columbia School of Journalism and the University of Delaware.

For daily updates, follow Larry on Twitter.

More from “Between the Lines”

Related Discussions on TechRepublic

Did you know you can take part in these discussions with your ZDNet membership?
Ask a Question Start a Discussion

Talkback Most Recent of 88 Talkback(s)

  • So lets see.....
    No downtime in 3 years for exchange including a migration during that time from 2003 to 2007 which can be done with no downtime. Or Gmail which has an outage every month or so. I think my company would can me if I implemented google mail for our business.
    ZDNet Gravatar
    OhTheHumanity
    05/14/2009 08:47 AM
  • It's the network
    Great you are able to manage your Exchange servers, but you are unable to pinpoint an ISP failure ?
    In three years of operation you haven't seen your emails delayed because of on of your router ISP gone bad.
    No downtime at all, well I don't buy it.
    ZDNet Gravatar
    Mighty_BOFH
    05/14/2009 10:22 AM
  • Yeah......
    Network redundancy is bliss. This however is not an ISP issue since it is occuring across multiple ISP's. I wouldn't want to depend my business on gmail.
    ZDNet Gravatar
    OhTheHumanity
    05/14/2009 10:39 AM
  • Go To Trackback #23
    Todays related news in broadband.........>
    ZDNet Gravatar
    Gillman_Zorgam
    05/14/2009 10:51 AM
  • Exchange??
    I can show you plenty of Exchange downtime in one large organization. In fact ironically it had almost become the norm to use Gmail for non important stuff.
    ZDNet Gravatar
    storm14k
    05/14/2009 10:58 AM
  • So you blame....
    Exchange for this? Did it ever cross your mind that whoever is running the show may be the one to blame. I worked in a very large organization with thousands of servers, desktops, and users. I was not the exchange admin there, but it was up each and every day. Wonder how such a crappy product can perform excellent at one organization and then fail miserably at yours? Take what you will from that, but humans are still in charge.
    ZDNet Gravatar
    OhTheHumanity
    05/14/2009 11:16 AM
  • If I can't blame....
    Exchange then you can't give exchange credit because its the people that keep it up crappy or not. I've seen sendmail handle an entire mid sized university before and I don't call it an advanced product but the people running it kept it up. Just as in this case with Google it may have been the people at AT&T causing the problem. They could have caused the same problem with me trying to reach exchange remotely from home.

    I didn't make any claim of Exchange being crappy or not. You just assumed. I simply pointed out that your post is pointless and that Exchange is not the answer whether you meant because of the product itself or the fact that its run in house. Any email solution can go down.
    ZDNet Gravatar
    storm14k
    05/14/2009 11:28 AM
  • Exactly....
    Sorry I misunderstood. My point was to explain exactly what you posted above.
    ZDNet Gravatar
    OhTheHumanity
    05/14/2009 11:57 AM
  • ??
    Why are you complaining about Exchange when the subject is "Google down"? And if Exchange is not working for you, maybe you should find a different Exchange admin - we are running everything on Exchange (4000+ mailboxes, calenders, public folders, tasks, you name it) since 3 years now and never had any failure of that thing.
    ZDNet Gravatar
    Paradise Lost
    05/14/2009 12:38 PM
  • !!
    Maybe you should read the post I was responding to as Exchange was being discussed there.
    ZDNet Gravatar
    storm14k
    05/14/2009 02:08 PM
  • Well, if you can then show away.....
    with all due respect, I find it a bit unusual you seem to have an anecdote about MS and downtime or problems or whatever the case may be for most any occasion.
    I'm not saying this with any intended sarcasm or in a mean spirited fashion, nor am I in anyway suggesting it's not true, I'd just like to take you up on that offer to show us the plenty of Exchange downtime in one large organization which you claimed you can.

    I know you are not the kind to blow smoke, so go for it.

    ZDNet Gravatar
    xuniL_z
    (Edited: 05/14/2009 05:56 PM)
  • RE: Cloudy day: Google, Gmail sucking wind or down; Packets lost in key cities
    I can't function like this; I need Google! Thanks for reporting this news, it's nice to know it's not me, I guess.
    ZDNet Gravatar
    brimedd
    05/14/2009 08:53 AM
  • Out in Denver
    Hope they're back soon. Glad my business does not depend on Google.
    ZDNet Gravatar
    ThePrairiePrankster
    05/14/2009 09:00 AM
  • RE: Cloudy day: Google, Gmail sucking wind or down; Packets lost in key cities
    Ah Hem. Google does not control every router from point A to point B.

    Cr*p happens. Maybe they are dealing with a Denial Of Service attack or something. Chill...
    ZDNet Gravatar
    cybr2th@...
    05/14/2009 09:04 AM
  • Out in DC
    It has been for about 30 minutes.
    ZDNet Gravatar
    ebuzzmaster
    05/14/2009 09:06 AM
View More Comments

Talkback - Tell Us What You Think

advertisement

Get it the way you want it

ZDNet Newsletters

Get the best of ZDNet delivered straight to your inbox

Blog Roll

Blog Archive

White Papers, Webcasts, & Resources
advertisement