Cloudy day: Google falters; Packets lost in key cities

Updated: Here's how Google's outage unfolded on Thursday... 

We're getting various reports via that Google services are down or at least sucking some serious wind. The service appears to be back as of noon-ish EDT in New York City. Google cites a "traffic jam" in Asia for the outage. 

The fail whale Google style:

Initial pings on Google.com show packet losses in New York, San Francisco, Chicago, Australia, France, China and other locales. The outages ran across all of Google's properties such as YouTube, Gmail, Google Analytics, Google Maps and Google Docs. The Twitter Gmail-o-meter for Google fail and Gmail is running amok. Also see the Internet Storm Center and Stephen Shankland and Techmeme. 

Also see: Google's App Status dashboard.

Here's the report from via Just Ping:

A few minutes later it appears some places are recovering, but the packet losses are rotating to new areas. We're nearing the half hour mark of Google's troubles. 

Update: In the stray email department, I was informed that it's an AT&T routing issue. Anything that touches Google via AT&T is down. Trying to confirm that now so take it for what it's worth.

Here's the traceroute pointing to AT&T:

Update 2: Got some our resident IT guru to explain this in English. It does appear Google is stopped at the AT&T border. Note: This may just be New York City specific. Here's the diagram:

Update 3: A reader reports:

Enterprise gmail, applications and regular gmail is down in Los gatos CA...almost all morning, what a disaster, we run our business on it. 

Update 4: Google appears to be back. Outage lasted about an hour from New York City. However, the packet losses are still rotating:

Update 5: Not convinced this was an AT&T specific issue judging from the talkbacks here and here. In any case, other sites worked fine for me through the entire Google outage. I assume we'll get explanation at some point soon. 

Update 6: CNet News' Shankland has two interesting nuggets. First the Google statement: 

"We're aware some users are having trouble accessing some Google services. We're looking into it, and we'll update everyone soon."

Gmail is reportedly all clear. Meanwhile, Keynote is showing packet losses at NTT and Qwest. That fact means it's more than just AT&T at work behind the Google issues. 

Update 7: Google's App Status dashboard is handy. Here's the Gmail information as of 1:13 p.m. EDT:

Update 8: Google gets to Twitter a bit late, let's call it 1:20 p.m. EDT. 

Update 9: AT&T says via Twitter that it's not responsible for the Google outage.

The telecom giant also issues the following statement:

 

After receiving speculative reports in the media that Google experienced  an outage related to the AT&T network, we looked into the matter.  We have not identified any specific problems in our network that could have caused the reported outage.

Update 10: McAfee argues that the Google outage was due to a IPv6 upgrade, reports CNet News' Tom Krazit. Google isn't elaborating in its blog post. Google explains:

Imagine if you were trying to fly from New York to San Francisco, but your plane was routed through an airport in Asia. And a bunch of other planes were sent that way too, so your flight was backed up and your journey took much longer than expected. That's basically what happened to some of our users today for about an hour, starting at 7:48 am Pacific time.

An error in one of our systems caused us to direct some of our web traffic through Asia, which created a traffic jam. As a result, about 14% of our users experienced slow services or even interruptions. We've been working hard to make our services ultrafast and "always on," so it's especially embarrassing when a glitch like this one happens. We're very sorry that it happened, and you can be sure that we'll be working even harder to make sure that a similar problem won't happen again. All planes are back on schedule now. 

Update 11: Fairly or not, Google's outage will be a reminder that cloud providers can go down. Will companies be prepared for outages? Sam Diaz argues Google's outage isn't enough to convince folks to avoid the cloud. But it certainly doesn't help. 

Update 12: Arbor Networks has a great chart on the GoogleLapse.