Once again, a datacenter “upgrade” takes down the datacenter

Summary:If you're putting lots of eggs in one basket it's important to have a plan if the basket gets dropped

With data communication centralized for more than 90 different Oregon state agencies, it’s not surprising that when a planned storage upgrade Monday somehow screwed up network communications with the datacenter, that the state referred to it as a catastrophic failure. Oregon State Administrative Department spokesman Matt Shelby placed the blame firmly on their storage vendor, Hitatchi, but gave no details as to the root cause of the failure. The crash solely affected communications; the state claims that there was no loss of data.

The crash did affect the normal operations of some of those 90 departments, however. A number of state services went down due to the problem, including high-profile services such as the ODOT TripCheck road cameras. Some of the reported high-level failures might leave you scratching your head, such as the problem faced by the department of Forestry when the responded to a fire.  Apparently, the database crash impacted their job by not permitting the department to access some of their database forms.

The most significant impact was on the State Employment department. The datacenter availability problem meant that 70,000 checks for unemployment compensation were delayed, with the longest delay suffered by new filers, who get their first check via the mail (later payments are made via direct deposit or to a benefits ATM card).

As the state repeatedly confirmed that the failure was the result of communication issues and that no data was lost or compromised it’s difficult to extrapolate what the storage vendor might have done in the normal course of a storage upgrade to cause such widespread impact on what should be unrelated systems.

 

Topics: Data Centers, Storage

About

With more than 20 years of published writings about technology, as well as industry stints as everything from a database developer to CTO, David Chernicoff has earned the term "veteran" in the technology world. Currently the principal of an independent consulting business and an active freelance writer, David has most recently been a Seni... Full Bio

zdnet_core.socialButton.googleLabel Contact Disclosure

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Related Stories

The best of ZDNet, delivered

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.