How a paranoid US dept took a $2.7m wrecking ball to its own IT systems

A report (PDF) released by the US Department of Commerce has revealed how one US agency, fuelled by the paranoia of a nation-state attack, spent US$2.7 million trying to destroy US$3 million worth of its own IT equipment, even though evidence of such an attack was never found.

At the end of 2011, the United States Computer Emergency Readiness Team (US-CERT) advised the Department of Commerce's Computer Incident Response Team (DOC CIRT) that its systems may contain a malware infection. DOC CIRT shortly after narrowed this infection down to a network shared by the National Oceanic and Atmospheric Administration (NOAA), the US Economic Development Administration (EDA), and other US departments and agencies.

While NOAA's response team had cleaned the infection by January 12, 2012, the warning instead placed EDA on high alert.

To determine the extent of the perceived infection, EDA asked DOC CIRT to provide a listing of what IT components may have been potentially infected. This began a line of miscommunication into the severity of the infection, with DOC CIRT providing a list of 146 IT components that were simply within the network boundary.

In fact, only two components were found to be infected. Although EDA was not equipped to handle the issue alone, DOC CIRT asked EDA to resolve the issue. EDA, believing that DOC CIRT had identified 146 cases of infection, fired back that it was unable to do so. From EDA's response, DOC CIRT believed that EDA had done the analysis to identify that all 146 components were infected, and thus both parties had then convinced each other of a widespread infection that actually did not exist.

Its reaction to the perceived threat grew more extreme. By January 24, 2012, it had enlisted the help of US-CERT and the Department of Energy (DOE), and cut its email, website services, and access to its database applications off from the network. It instead requested the US Census Bureau to provide internet access and email services.

Further paranoia about the possibility of the attack being conducted by nation-state actors resulted in EDA bringing on an external information security contractor to examine its systems in addition to the existing resources examining the issue.

This contractor initially reported to EDA that it had found "indications of extremely persistent malware and suspicious activity", giving weight to the belief that a sophisticated attack was underway. However, US-CERT's report indicated at the time that although common malware was present, there was no evidence of any nation-state activity or the extremely persistent malware as first thought.

Shortly after these reports were filed, EDA requested the help of the US National Security Agency (NSA), and a day later, DOE also filed its report, noting the same results as US-CERT — that there was no nation-state attack.

A little less than two weeks after its initial report, the contractor reversed its position, admitting that its initial analysis had been wrong, and there was no evidence of a highly sophisticated attack.

The situation was further confused by the Department of Homeland Security (DHS), however, which issued a report based on the inaccurate information provided by DOC CIRT that began the entire chain of miscommunication. The NSA later used DHS' report as fact, and did not attempt to verify whether the information was sound, despite finding no nation-state activity or persistent malware in its own analysis.

Ultimately, only six components within EDA's IT infrastructure were found to be infected, and only by common malware. Given that its systems had been looked at by several government agencies and an external contractor, EDA believed that little was to be gained from further forensic analysis, and on May 15, 2012, decided to turn its focus onto cleaning its data.

Still paranoid about the possibility of a nation-state attack despite the findings of several reports, EDA's CIO ordered the destruction of all of EDA's IT components.

These components included desktops, printers, TVs, cameras, computer mice, and keyboards. Even laptops that had been purchased prior to the incident and had not ever been put into operation were included.

By August 1, 2012, EDA had destroyed over US$170,000 worth of its infrastructure. It had only been prevented from destroying the remaining US$3 million worth as it had run out of funding for the operation, and the Commerce IT Review Board refused to approve the US$26 million it would need to continue its recovery operations.

By that stage, however, EDA had spent US$823,000 on its external security contractor, US$1.061 million on temporary infrastructure, US$175,000 to destroy its equipment, and US$688,000 on external assistance for its recovery operations.

In all, EDA spent US$2.7 million to combat an infection that had never existed.