How a paranoid US dept took a $2.7m wrecking ball to its own IT systems

How a paranoid US dept took a $2.7m wrecking ball to its own IT systems

Summary: Convinced that its systems had been compromised by nation-state hackers, a US government agency spent $2.7 million trying to destroy its IT equipment — even its mice — and only stopped because it ran out of money.

SHARE:

A report (PDF) released by the US Department of Commerce has revealed how one US agency, fuelled by the paranoia of a nation-state attack, spent US$2.7 million trying to destroy US$3 million worth of its own IT equipment, even though evidence of such an attack was never found.

At the end of 2011, the United States Computer Emergency Readiness Team (US-CERT) advised the Department of Commerce's Computer Incident Response Team (DOC CIRT) that its systems may contain a malware infection. DOC CIRT shortly after narrowed this infection down to a network shared by the National Oceanic and Atmospheric Administration (NOAA), the US Economic Development Administration (EDA), and other US departments and agencies.

While NOAA's response team had cleaned the infection by January 12, 2012, the warning instead placed EDA on high alert.

To determine the extent of the perceived infection, EDA asked DOC CIRT to provide a listing of what IT components may have been potentially infected. This began a line of miscommunication into the severity of the infection, with DOC CIRT providing a list of 146 IT components that were simply within the network boundary.

In fact, only two components were found to be infected. Although EDA was not equipped to handle the issue alone, DOC CIRT asked EDA to resolve the issue. EDA, believing that DOC CIRT had identified 146 cases of infection, fired back that it was unable to do so. From EDA's response, DOC CIRT believed that EDA had done the analysis to identify that all 146 components were infected, and thus both parties had then convinced each other of a widespread infection that actually did not exist.

Its reaction to the perceived threat grew more extreme. By January 24, 2012, it had enlisted the help of US-CERT and the Department of Energy (DOE), and cut its email, website services, and access to its database applications off from the network. It instead requested the US Census Bureau to provide internet access and email services.

Further paranoia about the possibility of the attack being conducted by nation-state actors resulted in EDA bringing on an external information security contractor to examine its systems in addition to the existing resources examining the issue.

This contractor initially reported to EDA that it had found "indications of extremely persistent malware and suspicious activity", giving weight to the belief that a sophisticated attack was underway. However, US-CERT's report indicated at the time that although common malware was present, there was no evidence of any nation-state activity or the extremely persistent malware as first thought.

Shortly after these reports were filed, EDA requested the help of the US National Security Agency (NSA), and a day later, DOE also filed its report, noting the same results as US-CERT — that there was no nation-state attack.

A little less than two weeks after its initial report, the contractor reversed its position, admitting that its initial analysis had been wrong, and there was no evidence of a highly sophisticated attack.

The situation was further confused by the Department of Homeland Security (DHS), however, which issued a report based on the inaccurate information provided by DOC CIRT that began the entire chain of miscommunication. The NSA later used DHS' report as fact, and did not attempt to verify whether the information was sound, despite finding no nation-state activity or persistent malware in its own analysis.

Ultimately, only six components within EDA's IT infrastructure were found to be infected, and only by common malware. Given that its systems had been looked at by several government agencies and an external contractor, EDA believed that little was to be gained from further forensic analysis, and on May 15, 2012, decided to turn its focus onto cleaning its data.

Still paranoid about the possibility of a nation-state attack despite the findings of several reports, EDA's CIO ordered the destruction of all of EDA's IT components.

These components included desktops, printers, TVs, cameras, computer mice, and keyboards. Even laptops that had been purchased prior to the incident and had not ever been put into operation were included.

By August 1, 2012, EDA had destroyed over US$170,000 worth of its infrastructure. It had only been prevented from destroying the remaining US$3 million worth as it had run out of funding for the operation, and the Commerce IT Review Board refused to approve the US$26 million it would need to continue its recovery operations.

By that stage, however, EDA had spent US$823,000 on its external security contractor, US$1.061 million on temporary infrastructure, US$175,000 to destroy its equipment, and US$688,000 on external assistance for its recovery operations.

In all, EDA spent US$2.7 million to combat an infection that had never existed.

Topics: Security, Government, Government US, Malware

Michael Lee

About Michael Lee

A Sydney, Australia-based journalist, Michael Lee covers a gamut of news in the technology space including information security, state Government initiatives, and local startups.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

6 comments
Log in or register to join the discussion
  • Sounds like someone needed a reason

    to get a new contract handed out to a "preferred" supplier, since they'd have to replace that hardware from somewhere.
    William Farrel
  • Here's The Fun Part...

    If this follows true federal employee history, whoever was in charge of this fiasco and wasting all of those tax $$ is probably being shifted into another department with no reprimand, loss of pay and nothing on their permanent record to indicate anything is wrong.
    TechnoCritter
  • thanks, Michael

    For pointing out to this "comic accident".
    Nothing surprising really. Another manifestation of complete incompetence and professional idiocy. For once just fire yourself and your kin, get rid of the f%$king Windows and get some competent IT managers and staff! How many times do you need to get a infected or to get paranoid over a false infection? If your system can "magically" catch a virus (and even spread further). When requires some obscure AV software that scans and removes that malware through some magic, just get rid of this system, get another one requiring no magic, but logic and your own brain power, re-educate yourself.
    eulampius
    • a conclusion

      Only one question remains here, the NSA that allegedly snoops on most people around the world, dumps the data onto its servers and ... yes, what are the odds that someone else is wiretapping these overly competent wiretappers?

      #Talking about competence, or lack thereof. I didn't get this piece pass the great zdnet antispam filter.
      eulampius
  • Dumb

    These people actually passed an interview and got hired? You gotta be one of the dumbest stumps to think a simple mouse or keyboard can carry a computer virus. With people like this in gov't, we're doomed I say!
    John Mann
  • Made my Day!

    That the most hilarious item I have read this year.

    What a bunch of bungling incompetent bozos.

    The whole fiasco sound like a series of Dilbert cartoons.
    I doubt even Scott Adams could have dreamt this up.
    ITenquirer