New Jersey suffers yet another state datacenter failure

It came as quite a surprise to the hundreds of people who showed up at NJ department of Motor Vehicles offices Thursday to find that there was no waiting in line. But that surprise rapidly turned to annoyance and frustration as they were told that the computers were down and no services were available.

For the third time in the last seven weeks a problem with the state’s datacenter brought down services offered to state residents. This time it was a power failure at 5 AM that was unresolved for four hours. But even after power was restored, the state was unable to completely restore all of the affected services until well into the afternoon, with everything back up between 2 and 3 PM.

This time around, the state placed the blame squarely on their power provider, with Bill Quinn, Treasury spokesman stating "We are reviewing the cause of today's outage with PSE&G, our power supplier, and the steps that should be taken to prevent a recurrence of this incident." The state has not released any information about what the actual problem was.

But since they are claiming it was a power problem, and it took six hours after the power was fully restored to get all of the applications back up and running, it seems to beg the question of what kind of backup power does the state use for their datacenters. Personally, I can’t recall the last time I visited a major datacenter where a part of the tour wasn’t the backup generators and a discussion of how long the facility could run without external power. In the case of this apparently important NJ state datacenter, the answer seems to be that it can’t run without external power.

The fact that application went down and required considerable time to restart indicates that there was not an orderly shutdown of applications and servers but more of a “somebody pulled the plug” response. It’s hard to imagine that no one in the state government has noticed how important it was to the state to keep these facilities up and running given the weather related disasters of the past few years.