Follow up: Northrop Grumman on Tuesday said that it is committed to restoring the information systems behind key state of Virginia services and supported a thorough post mortem.
Nevertheless, some Virginia agencies are still suffering from an outage that has lasted almost a week.
If you recall, an outage---notably at Virginia's Department of Motor Vehicles, Elections and Taxation and Revenue agencies---has dragged on for nearly a week. An EMC storage area network failure was the culprit, according to the state's Information Technologies Agency (VITA).
That networked storage unit, an EMC DMX 3, has been repaired. Now teams are verifying and restoring data. The state said:
This outage has not crippled state government. It has created some challenges and the DMV outage has impacted citizens seeking drivers’ licenses but the vast majority of state government computing functions are fully operational. Approximately two-thirds of state agencies were not impacted by this outage. We ask for the continued understanding and patience of state employees and citizens as the recovery effort continues.
In a statement, attributed to Sam Abbate, vice president of the VITA Program for Northrop Grumman, said something similar. To sum it up: Recovery operations continue and thanks for the patience.
Also: Virginia's IT outage doesn't pass management sniff test
Here's Abbate's statement in full:
Statement by Sam Abbate, Vice President, VITA Program for Northrop Grumman
Regarding Information Technology Outage in the Commonwealth of Virginia
Over the last six days, Northrop Grumman and its teammates have been working around the clock with the Virginia Information Technologies Agency (VITA) staff and staffs within the agencies, to carefully and deliberately verify and restore access to data after a failure in a critical component in the infrastructure.
This failure set off a chain of events that impacted 485 out of a total of approximately 4,800 servers and has seriously impacted several agencies and hampered their ability to conduct business. As the servers have been brought back online each agency’s technical staff has been working to test the integrity of the data.
Within 72 hours of the incident 21 out of 27 agencies affected were operational. However, the data recovery timeline for the Department of Motor Vehicles, State Board of Elections and Department of Taxation and Revenue has taken longer than anticipated. We recognize the Commonwealth cannot afford to be without the services of its critical agencies and we are grateful for the patience of the agencies and their customers throughout this event.
We cannot afford to let any vulnerability in the infrastructure go unresolved. We have an obligation to VITA and the agencies and citizens they serve, to learn everything we can from this occurrence. We will conduct a root cause analysis, carefully analyze and review the findings, develop lessons learned and make necessary changes. Northrop Grumman supports the Governor of Virginia, Bob McDonnell’s call for an independent analysis of the situation, from the cause of the initial failure, through the process of restoration that has taken place.
Our commitment to this partnership is absolute and we are dedicated to making the Commonwealth’s information technology infrastructure the very best of its kind.