NSA, the FISA Court, and Risks of Tech Summaries

Yesterday the U.S. government released a previously-secret 2011 opinion of the Foreign Intelligence Surveillance Court (FISC), finding certain NSA surveillance and analysis activities to be illegal. The opinion, despite some redactions, gives us a window into the interactions between the NSA and the court that oversees its activities—including why oversight and compliance of surveillance are challenging.

The opinion has enough (unredacted) technical detail to get a basic picture of what the NSA was doing. The court was considering the NSA’s capture of traffic from Internet backbone links (“upstream capture”), which accounted for about 9% of the Internet traffic captured by the NSA. (The other 91% came directly from service providers, for example when email providers turned over the contents of targeted accounts.) The eavesdropping apparatus would do some kind of pattern matching on the traffic it saw, and would record traffic that “hit” on an approved pattern. Captured data would go into a database where human analysts could search and examine it.

The NSA was supposed to be capturing one discrete “communication” (e.g. an email message) at a time, but due to limitations in the capture technology they could only capture data in a unit called a “transaction” (e.g. an interaction between a user’s computer and an email server while the user was reading email). About 90% of the transactions that were captured contained only a single communication. The remaining 10% of transactions were Multi-Communication Transactions (“MCTs”), meaning that they contained at least one communication that hit on a pattern, along with one or more other communications that might not be a hit.

Based on the opinion plus remarks by officials in a press call yesterday, it appears that the NSA had a list of “targeted” email addresses that it was allowed to monitor, and the capture technology would grab a chunk of data such as an Internet packet, whenever one of the targeted email addresses was contained in that chunk. The NSA said it had no reasonable way to capture more precisely—although the detailed limitations of the NSA’s capture technology were redacted. The Court took these technical limitations at face value but said that these limitations did not make the resulting over-capture legal.

The Court scolded the NSA:

The Court is troubled that the government’s revelations regarding NSA’s acquisition of Internet transactions mark the third instance in less than three years in which the government has disclosed a substantial misrepresentation regarding the scope of a major collection program.

In March, 2009, the Court concluded that its authorization of NSA’s bulk acquisition of telephone call detail records from [redacted] … “ha[d] been premised on a flawed depiction of how the NSA uses [the acquired] metadata,” and that “[t]his misperception by the FISC existed from the inception of its authorized collection in May 2006, buttressed by repeated inaccurate statements made in the government’s submissions, and despite a government-devised and Court-mandated oversight regime.”

(footnote 14 on p. 16)

This footnote has been much-quoted as evidence of the Court’s disappointment. What struck me in this passage, though, is that the Court was unhappy because the NSA withheld important technical details.

Whenever technologists have to interact with less-technical courts or policymakers, there are communication challenges. It’s not helpful to tell the non-expert everything about how the system works—supplying too much detail is can be just as detrimental to decision-making as supplying too little. You have to supply a summary.

The trick is to supply a summary containing just the right details, that is, the details that matter to the non-expert in light of what they are trying to accomplish. If a non-expert judge needs to decide whether an activity is lawful, they need to know the details that relate to the legal questions they are considering.

Choosing which details to reveal can be very challenging—you have to understand something of the law, along with the technology—even if you’re doing your level best to help the decisionmaker. And if your motivations are less pure, it’s all too feasible to cook up a misleading summary of the facts. The Court seemed to suspect that the NSA had been doing exactly that.

Our legal system normally addresses this unreliable-summary problem by relying on the adversarial process. The opposing party gets access to the underlying facts and can cross-examine the expert who provides the summary. This process isn’t perfect but it does tend to deter and correct unreliable summaries.

This safeguard is not in place when the FISC is considering NSA activities. The FISC has no choice but to rely on the NSA’s own summary of the technical details. Even if NSA employees are trying in good faith to tell the Court what it needs to know, it’s not too hard to see how human error plus institutional pressures could lead the agency to not reveal “unhelpful” facts.

Congress could address this problem by changing the law to create an opposing party in FISC procedures—a change that the President recently endorsed. But a really effective opposing party will need to have access to technical evidence and expertise as well.