NSA datacenter a ticking time bomb

NSA datacenter a ticking time bomb

Summary: Design flaws are causing a billion dollar boondoggle.

SHARE:

According to a story in the Wall Street Journal, Americans have little to worry about from the new NSA datacenter, for one simple reason; it doesn’t work.

NSA offices
NSA Headquarters

In an exercise in how not to rush the completion of a major datacenter project, the new facility is plagued with as yet unsolved electrical problems, which have resulted in the destruction of hundreds of thousands of dollars’ worth of equipment.

According to an article in Forbes, an insider reported that the facility was laid out with equipment density that was beyond the capabilities of the wiring plant, resulting in the creation of "kill zones" that resulted in electrical arc fires that would destroy the equipment in a specific area.

Documents reviewed by the Wall street Journal showed that this had happened at least 10 times since the facility was commissioned, most recently the last week in September.

It is being reported that the contractors responsible for the design and construction of the datacenter were unable to pinpoint the cause of the problem so the government sent in a US Army Corps of Engineers Tiger Team to save the day. Unfortunately, the problem apparently remains unsolved.

The Journal article also covers a report that was issued on the problem and states that it blames the decision to fast track the construction of the datacenter and bypass regular quality control in design and construction. Full operation of the facility is expected to be delayed at least a year.

I find it hard to believe that issues of equipment density and power weren’t on the top of someone’s list of concerns. I don’t recall ever talking to a datacenter infrastructure person or people building new facilities who couldn’t tell me, down to the watt, the amount of power that they could deliver to each rack in the facility and how, exactly, that power was provided.

Delivering power to the IT load equipment is one of the fundamental design and construction issues when you build a datacenter, and to have electrical faults on this scale is unconscionable, not only in a government project, where they suffer from the "other people's money" issue, but in any datacenter design.

Related stories:

Topics: Data Centers, Government US

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

28 comments
Log in or register to join the discussion
  • I wonder......

    Did somebody or Government Stuxnet the NSA ?
    Alan Smithie
  • Short Circuit

    Electricians probably took short cuts building it so the physical wiring doesn't match the specifications and drawings. To fix in place you would have to verify the placement, connection of every wire, grounding of every circuit, panel, transformer, etc. nearly impossible I'd think.

    Isolate the old, replace with new, rip out the old.
    greywolf7
    • Cheaper to close the criminal agency.

      Since the entire NSA agency is a rogue agency, just close them down and deport the employees. Huge boost to the USA's national IQ.
      Reality Bites
      • Really, Deport All The Employees for Doing There Jobs?

        Maybe you mean to deport those in charge at NSA? I doubt they are all evil. And I am guessing they are trying to do good even if the means doesn't justify the . . .
        rmark@...
  • Is there no end to American incompetence?

    Or as Gewirtz suggests "it's all part of the plan".

    I volunteer to press the self-destruct button. No charge.

    (No charge - get it?)
    jacksonjohn
    • Saved by a boondoggle...

      Might even be thought of as sabotage...

      Protecting the citizens of the US... And getting paid for not working as well.

      I have run across many "shortcuts" that were technically valid - like running ground wires to the building support beams.

      Works fine... Until somebody puts a lightning rod on the roof. After that, every lightning stroke causes a huge power event that can fry nearly anything, and everything.
      jessepollard
  • Kinda like the Affordable Care Enrollment snafu's!

    The best don't get the contract, the ones with the connections do (even if they are incompetent)!
    kd5auq
  • The parking lot is well-lit

    .
    Rabid Howler Monkey
  • God throwing a wrench in your plans?

    As the Universe would have it, your plans shall be delayed. The river of blood is not yet ripe for the harvest. Caesar is not ready to claim his throne and abuse the power you are about to be given pretending its steward.
    Vapur9
  • ELECTRICAL ARC fires? TEN ?!?

    REALLY?

    Having worked as an electronic tech for a decade years ago, I find that impossible to believe!

    Maybe they have overheating problems from poor ventilation. They may have improper grounding problems that are leaving equipment at a potential seriously above ground (zero) voltage. But electrical arcs? C'mon!
    Rick_R
    • Don't forget.... lowest bidder & the dumbest employer in the world.

      The federal hacks wouldn't recognize a switchbox if you shoved their fingers in it and turned it on.

      Only the dumbest of the dumb or psychopaths work for the most evil agency in the world, neither of which are known for common sense or intelligence.
      Reality Bites
    • Saw an article that seemed to bear this out...

      The description was "a bolt of lightning in a 2-foot box." That sounds like an electrical arc to me. Of course, more detail would have been nice, like what voltage these circuits were at, loads, etc., but I think the image is pretty definitive.

      Shocking...
      Den2010
    • Remember the size of the facility...

      1.5 million square feet. if even a quarter of that is the building size (.3 million) that creates one hell of a lot of basic static build up.

      You can get 10s of thousands of volts from a two story building, in even GOOD weather. If the building frame isn't properly grounded (and everything ELSE gets a separate ground..) fires are the least of your worries.

      EMP discharge during storms will destroy computers - especially when funneled into the buildings due to improper grounds. I saw one Cray have its memory fried every time a thunderstorm blew by... and each time caused downtime of a day.
      jessepollard
      • data center neutral size

        The most common miscalculation in sizing a neutral conductor for a data center is ignorance of harmonic currents.The NEC recommends doubling the normal calculated neutral size in these situations.Insofar as grounds, all points must connect to a common single ground to reduce difference in potentials.Separate grounds will cause problems and can create hazardous voltage.Loose connections due to improper tools or procedures seems to be the problem in most explosive arc cases;loose crimps,split bolts,or improper tensioning of wrapping insulating tape can cause catastrophic failures of junctions or joints. The contractor probably hired local manual labor with no experience, or training to do these critical jobs.The engineer is responsible for design of the system, and the electrical contractor is responsible for supervising and directing the work of his crew.If he has good job foremen, his job is a lot easier.Problem is, when you get someone trained to that level, he gets his license and starts his own company.So key jobs always have a high turnover rate.
        The inspectors for these government jobs should be independent, and not government employees,that way there is accountability.
        I have seen far too many jobs that were sub-par when inspected by gov.inspectors that failed several years after the installation.If this is the first wave before they have even got their feet wet, the future does not bode well.This is only the tip of the iceberg.
        More and worse will follow.
        tsimmone
  • Lowest bidder and the dumbest employer in the universe

    It will cost the taxpayer 100 times the lied about total, and cost 1000 times the lied about budget per year.

    If its federal, its a parasite and a liar....100% of the time.
    Reality Bites
  • Would you expect anything less

    From a government project? Nothing new here, all government projects are like this.
    RAK5
  • They have a Snowden Electrician

    Smells like a saboteur!
    ammohunt
  • Misleading headline

    A whole lot of nothing in this article. Nothing but an inflammatory title without any need for it - there is nothing about to blow up except the author's reputation.
    GDCM
  • Wrong Picture

    The caption is incorrect as far as identifying the image, it is otherwise accurate that NSA is building a datacenter in Utah.

    The image in question is not Utah but the NSA headquarters in Maryland at night, with some lens fanciness to blur headlights and taillights, a standard stock image but not what it purports to be.
    DataRepoMan
    • Corrected

      Now it matches. I retract my original statement.
      DataRepoMan