Inside the NSA's perspective on the release of certain previously classified documents

Earlier today, the Director of National Intelligence declassified a series of previously (and correctly) classified documents regarding an error in the capture and intelligence analysis of telephony metadata that took place back in 2009.

Also today, I had the opportunity to be briefed by and speak with certain senior intelligence officials in order to explore the circumstances of the original error and this document release. The briefing was authorized, but the officials requested not to be identified publicly by name so that they might "feel free to discuss information more freely."

I've done these "on background" meetings before and the reason for official anonymity has less to do with anything nefarious and more to do with the fact that officially attributed statements need to be approved by many levels of legal review, where these "on background" things are much more like real conversations (and, frankly, much more useful).

The documents being declassified were described as "properly classified," meaning that when they were classified, they met the criteria for documents that could not be disclosed because of national security. The gentlemen I talked with today said that, from a national security perspective, these documents are still sensitive, but the President determined that the need for transparency with the American people was of a greater importance in this instance.

On the other hand, none of the officials I spoke with mentioned another little detail: that the EFF had filed a Freedom of Information Act Lawsuit and a court ordered the documents to be released.

While the documents are being released, there are still some elements that remain redacted, but the belief is that the ability for the public and the press to see the details of the 2009 incident were of particular import in light of the recent scrutiny of the NSA.

The most interesting piece of meat in our discussion was the core of what went wrong back in 2009. The intelligence community has a classification standard called RAS (reasonable articulable suspicion) which is a base-level standard any intelligence suspect must satisfy before investigation is permitted. It's a relatively high bar, because any RAS suspect must be able to be justified by the RAS criteria and that criteria must stand up before a judge.

In 2009, the NSA was scanning telephony metadata and was supposed to select for analysis only those pings that met RAS criteria. This would have allowed a much larger percentage of metadata to pass through the system untouched.

However, that did not happen. The compliance violation was that roughly 17,000+ items were on something called an Alert List (a list of possible terrorism-related items), but only about 1,800 met the RAS criteria. Even so, the NSA subjected incoming traffic analysis against all 17,000 to the intelligence process, not just the ten percent or so that were RAS-worthy.

To make matters worse, once an item that didn't meet RAS criteria was identified, it was sent to a supervisor (which is not something that's supposed to happen) and, on top of that, the non-RAS item was, and I'm quoting here, "mis-described," implying all the selectors had been RAS-approved.

The details of all that are in the documents released by DNI James Clapper.

The officials I spoke to today did their best to convince me that there was no intentional effort to evade the law in these instances, that the NSA discovered the problem, and reported it to the FISA court. I was also told that once this problem had been identified, the NSA subjected itself to (and this is another quote), "a thorough, painstaking, and self-critical scrub." During that process they found other issues (the specific details of which haven't been shared with me), and those issues were reported to the FISA court as well.

The key contention was that the NSA system is both wildly complex and operated by humans, and that combination is bound to result in errors. I tend to side with this view of the NSA data collection operation rather than the Big Brother view, simply because I know how hard many of America's civil servants work, I understand the scale of both the protection challenge and the data management challenge, and -- in my experience -- if something complex can either go wrong because someone is acting in an "evil" way or something can go wrong because something broke, the very high odds favor the something breaking.

Even so, the senior intelligence officials I spoke with today wanted to reinforce that this "transparency exercise" should help reassure the public that policy decisions are being made by our leaders in ways to provide robust protections for privacy, vigorous oversight, and an attention to national security.

That said, they were also sure to reinforce that anything as technologically complex as the NSA's systems was "not going to have perfection," but they do make it a high priority to find and fix problems.

Stay tuned. There's more to come.

By the way, I'm doing more updates on Twitter and Facebook than ever before. Be sure to follow me on Twitter at @DavidGewirtz and on Facebook at Facebook.com/DavidGewirtz.