DreamHost customers hit with nightmare

Hosting company DreamHost is becoming a nightmare for customers. The company has had trouble keeping its customer sites up and running as it migrates to a new data center.

The problems began to appear on Sunday and are stretching almost into Thanksgiving. Customers have reported that their sites have been down for 24 hours at a clip and when there is a recovery it isn't a reliable one.

Among the problems:

• DreamHost has been upgrading their shared hosting hardware;
• The upgrade went way wrong;
• Customer support didn't know what was going on.

The gory details can be found here as DreamHost writes:

DreamHost is currently experiencing a fairly large network failure. The extent is unknown at this time however it seems that most of our central services (dreamhost.com, panel.dreamhost.com, webmail, etc…) as well as our customer sites and email are affected.

We have our network experts up and looking into the situation right now, hopefully a solution or at least more information will be forthcoming.

In an email to a customer flabbergasted at the outage, DreamHost's customer support team wrote:

From: DreamHost Customer Support Team <support@dreamhost.com> Date: Tue, Nov 24, 2009 at 9:27 PM Subject: Re: Site is down

------------------------------------------------------------------------ - After reading this response, please consider visiting - the URL below to comment on its quality. Thanks! - - http://www.dreamhost.com/survey.cgi?n=30693556&m=5589763 ------------------------------------------------------------------------

Howdy,

We're quite sorry about the problems you've been running into today! Our admin team recently upgraded the network in the data center where your machine is located.  Unfortunately, we had a major network outage last night that caused one of those upgrades to no longer work correctly.

Now here's where things get tecnical.  Sorry if your eyes glaze over while I geek out over the details...

Basically, after the network outage started happening, we had to reseat the Gigabit Interface Converter (GBIC) that was no longer being used.  Once the GBIC was reseated the network started working correctly.  In fact, all of the problems and packet loss went away again.

As it stands, our admin team is closely monitoring our network.  They have been since this weekend's data center move - but we are now on even higher alert since last night's network outage.

Even tho I said this before, it bears repeating:

Sorry about all the problems!

I'll be straight with you - our team is in a bit of a rough spot (from a networking standpoint) as we integrate all the new machines to this data center.  If you are noticing any lingering problems do not hesitate to reply to this message.

---

Addendum: We've noticed that WordPress is returning 500 errors for a large number of customers.  This is possibly due to problems with accessing the database that were caused by the network issues.

If you happen to notice this happening to your WordPress install, you have two plans of attack:

1. You can change the "template" and "stylesheet" settings in your wp_options table to the value "default".  After you've done this, visit your site again.  It should hopefully come up.

If it does, go ahead and set the value back to what you had previously and go about your day.

2. If mucking with the database isn't your thing, go ahead and change the name of the folder your current theme is in.  You should be able to locate the theme folder via FTP or SSH in the "wp-content/themes" folders wherever your copy of WordPress is installed.

Doing this will make WordPress freak out (since it can't find the files it needs) and flip your site back onto the Default theme.  Then you can visit your site - which should be coming up in stripped down form - log into the admin interface and change your theme back to your preferred one.

As I said previously, please contact support if you are still noticing any issues.  Even moreso if our bulk issue mover managed to put you into the high volume support queue when your intial ticket wasn't even related to outages or WordPress 500 errors.  We earnestly want to assist you with your problem and get your site running ASAP - so anything we can do to help, we will.

Thanks, Jason

P.S. I also need to apologize for using a canned response.  The amount of support that was generated by this problem called for it tho.  If I could have responded to each question individually, I would.  I hope you understand that.

Lesson: Plan your migrations better.