Microsoft Office 365 email outage affecting some US customers
Some users are reporting on Twitter that Office 365 email is currently experiencing substantial delays in various spots across the US.
I'm seeing some say they had problems sending and receiving email for more than six hours today, June 30.
(I'm not having email issues, and I'm in New York City -- one of the areas where some people are reporting problems. My Office 365 status dashboard is not operable, however.)
The DownDetector.com site is showing a spike in Office 365 reported issues with Exchange, Outlook, and server connections starting around 10:30 am ET Thursday.
I asked Microsoft at 5 pm ET for an update on what's happening. No word back.
One of my Twitter contacts, Ryan Hoffman, said his company has had issues since 11:30 am ET. He shared a screenshot of queued messages trying to be sent to Office 365 (see above).
A Microsoft spokesperson told The Register early Thursday that some customers were experiencing delays in sending and receiving mail, but service was being restored.
One affected user, Michael LaMontagne, managed to see his Exchange Online status dashboard, which showed service degradation starting around 10:30 am.
Microsoft's status message noted that "a portion of the infrastructure responsible for processing Exchange Online Protection (EOP) message filtering became degraded, resulting in message transport delays." The message added that a subset of users were affected and that Microsoft was "restarting services to bring the affected infrastructure back to a healthy state".
If and when I hear more, I'll update this post.
Update No. 1: Here's what Microsoft is telling Office 365 dedicated customers at this point (8 pm UTC/4 pm ET)
Incident Status:6/30/2016 8:09:20 PM (UTC) - We've identified that the configuration change to increase the inbound connection threshold has resumed inbound mail delivery for some customers, though, impact has not been fully mitigated as email delivery queues still exist. We're investigating alternate mitigation steps. In addition, we're reverting a recent update on the affected infrastructure that could be contributing to the problem. The next update will be provided by Thursday, June 30, 2016, at 10:00 PM UTC.
An hour before this, Microsoft was rerouting inbound and outbound connections to try to mitigate the issue. Officials said they were reviewing logs to try to isolate the cause of the underlying problem.
Update No. 2: Another update from Microsoft as to what's going on:
6/30/2016 9:50:11 PM (UTC) - We identified that a recent update to the environment caused an EOP process that analyzes email to perform below acceptable thresholds, causing email messages to queue from both inbound and outbound sources. We've implemented optimizations to allow existing email queues to be processed and are monitoring the queues to ensure that all messages are delivered and to validate full service restoration. The next update will be provided by Thursday, June 30, 2016, at 11:30 PM UTC
I'm starting to hear complaints from Exchange Online users outside the U.S. who are affected. If users' email accounts are hosted in Microsoft's North American datacenters, those users also may be affected.
Update No. 3 (on July 1): Robert Lavin shared an update he got last evening that showed an end time of 11:30 p.m. UTC for the Exchange Online outage.
Preliminary root cause listed as the infrastructure responsible for processing Exchange Online Protection message-filtering "became degraded."
"Next steps" included Microsoft analyzing data and trends on affected systems to prevent a recurrence of the issue. The company also is reviewing its code for optimizations and automated recovery options, according to the update shared by Lavin. Microsoft is set to publish a post-incident report within five business days, according to the status update.
I've been hearing from various users in various countries that they've been getting large amounts of queued email suddenly flooding their inboxes as of this morning, July 1.