New Jersey suffers yet another state datacenter failure

New Jersey suffers yet another state datacenter failure

Summary: Exactly how big a stick do you need to get them to pay attention?

SHARE:

It came as quite a surprise to the hundreds of people who showed up at NJ department of Motor Vehicles offices Thursday to find that there was no waiting in line. But that surprise rapidly turned to annoyance and frustration as they were told that the computers were down and no services were available.

For the third time in the last seven weeks a problem with the state’s datacenter brought down services offered to state residents. This time it was a power failure at 5 AM that was unresolved for four hours. But even after power was restored, the state was unable to completely restore all of the affected services until well into the afternoon, with everything back up between 2 and 3 PM.

This time around, the state placed the blame squarely on their power provider, with Bill Quinn, Treasury spokesman stating "We are reviewing the cause of today's outage with PSE&G, our power supplier, and the steps that should be taken to prevent a recurrence of this incident." The state has not released any information about what the actual problem was.

But since they are claiming it was a power problem, and it took six hours after the power was fully restored to get all of the applications back up and running, it seems to beg the question of what kind of backup power does the state use for their datacenters. Personally, I can’t recall the last time I visited a major datacenter where a part of the tour wasn’t the backup generators and a discussion of how long the facility could run without external power. In the case of this apparently important NJ state datacenter, the answer seems to be that it can’t run without external power.

The fact that application went down and required considerable time to restart indicates that there was not an orderly shutdown of applications and servers but more of a “somebody pulled the plug” response. It’s hard to imagine that no one in the state government has noticed how important it was to the state to keep these facilities up and running given the weather related disasters of the past few years.

Topics: Data Centers, Government

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

3 comments
Log in or register to join the discussion
  • Sometimes the poo just hits the fan.

    One center I worked at had multiple backup generators (3). A commercial power failure would drop the UPS to running on batteries, and start the generator(s). All worked fine during testing and for several years afterward.

    Slow growth of the center, and the load gradually grew. Still within the specs for all the generators. Backup testing worked fine.

    Until the power failure hit, and the fan spread poo everywhere.

    1. the failover occurred normally.
    2. two of the generators started normally.
    3. the third one however, blew a cylinder.

    The UPS batteries ran out, and the power was insufficient for the center... causing an overload on the generators... damaging the batteries and potentially the generators except that the central power monitor powered the entire building off.

    Recovery took time - first the lights had to be recovered. Then an emergency generator trucked in while the remaining generators began recharging batteries (and the replacement of the damaged ones). It wasn't until 8 hours later that the center began coming back on line - even then, each primary system had things damaged (mainframes don't like power surges during failures, and don't like loosing cooling at the same time). Secondary systems had occasional file corruption to be repaired, but no (detected) hardware damage. Tertiary systems were no problem (no major heat generation) and these systems used small UPC backups for the disks and were easy to power off.

    Other power failures were no problem. It was just a combination of failures that can cause that many problems even for centers properly prepared.
    jessepollard
  • raise taxes

    The state's governor is on record supporting job offshoring. When you lose jobs you lose revenue to pay for things so you have to cut costs and hope things work. Since offshoring led to a larger market and depressed wages to match, that too leads to less revenue (lower tax bracket)...

    Public or private, there is much hoping going on right now.....
    HypnoToad72
    • Don't even start with the governor

      Chris Christie has spent his time in office fixing the utter ruin caused by our last (one-term) governor, Jon Corzine. He is not perfect -- nobody is -- but I don't think there are many people on this earth who could have brought the state back to the point it is right now, this quickly. So don't go crapping on the NJ governor, especially if you don't live here.
      Speednet