Once again, a datacenter “upgrade” takes down the datacenter

With data communication centralized for more than 90 different Oregon state agencies, it’s not surprising that when a planned storage upgrade Monday somehow screwed up network communications with the datacenter, that the state referred to it as a catastrophic failure. Oregon State Administrative Department spokesman Matt Shelby placed the blame firmly on their storage vendor, Hitatchi, but gave no details as to the root cause of the failure. The crash solely affected communications; the state claims that there was no loss of data.

The crash did affect the normal operations of some of those 90 departments, however. A number of state services went down due to the problem, including high-profile services such as the ODOT TripCheck road cameras. Some of the reported high-level failures might leave you scratching your head, such as the problem faced by the department of Forestry when the responded to a fire.  Apparently, the database crash impacted their job by not permitting the department to access some of their database forms.

The most significant impact was on the State Employment department. The datacenter availability problem meant that 70,000 checks for unemployment compensation were delayed, with the longest delay suffered by new filers, who get their first check via the mail (later payments are made via direct deposit or to a benefits ATM card).

As the state repeatedly confirmed that the failure was the result of communication issues and that no data was lost or compromised it’s difficult to extrapolate what the storage vendor might have done in the normal course of a storage upgrade to cause such widespread impact on what should be unrelated systems.