Technical debt in the NSA's phone call data program?

According to stories last Friday in the Washington Post and Wall Street Journal, the NSA’s phone call data program seems to be less comprehensive than previously thought—the agency is apparently collecting data on only about 20-30% of calls, and mostly from landlines.

I’m quoted in the Post story saying that this “calls into question whether the rationale offered for the program is consistent with the way the program has been operating.” Advocates of the program tend to claim that it is necessary to “get the whole haystack” in order for the kind of chaining analysis done with this data to be effective. If the NSA has only about 25% of the full dataset—and if that is mostly from domestic landlines, which one would expect to be the type least used by the terrorists who are the targets of this program—one wonders how effective the program can be.

So what is going on? A few theories come to mind.

Theory A: Not under this program: One theory is that the NSA is actually getting a lot of domestic phone call data from another source, so this is another one of the “not under this program” evasions. This would mean the NSA is getting domestic phone call data via some method other than a Section 215 court order. For example, Marcy Wheeler argues that the data is coming from a foreign partner agency.

The argument against this theory is that it assumes the NSA is still willing to deceive the public and policymakers with the “not under this program” maneuver. The price to the agency’s credibility of getting caught in such a trick at this late date would seem to be fairly high.

Theory B: Can’t get court orders: A second theory is that the agency is somehow unable to get orders from the FISA Court to enable collection from the largest wireless and VoIP carriers. This seems inconsistent with what we know about the FISA Court’s rulings on the phone call data program.

Theory C: Low priority: A third theory is that the program is not very important to the NSA because the cost of operating it has long outweighed the intelligence benefits. If this is correct, the real question before us is why the agency has fought so hard to defend a program that has little value. And the likely answer is that this is reflects political tactics, either to defend the flanks of other programs, or to make an eventual shutdown of the program seem like a bigger concession than it really is.

Theory D: Technical debt: Another theory is that the lack of collection is driven by limits in the program’s underlying technology. The Post article advances this theory, with sources saying that the agency was struggling to prepare the system to receive the vast quantities of data that would come with an expansion to near-100% coverage. What is odd about this is that the quantities of data involved would not be challenging for today’s technology to handle; nor are the format-compatibility issues mentioned in the Post article very challenging.

Why might straightforward technical issues be holding up the program? One reason is that the program might be mired in technical debt.

For those not familiar with the concept, technical debt is a concept from software engineering. If your project has an engineering problem to address, the “right” response is to understand the underlying cause and address it in a careful (yet cost-aware) fashion. Alternatively, you can slap on a quick and dirty “band-aid” solution that makes the problem go away in the short run but leaves the system more fragile and bug-prone. If you opt for the band-aid approach, you are taking on technical debt. Until you pay back the principal by addressing the underlying engineering problem, you will have to keep paying interest on the debt by devoting engineering effort to coping with extra crashes and bugs.

Although prudent managers take on technical debt at times, there is also a trap—as with financial debt—in which the burden of interest payments makes it more difficult to dig yourself out of debt, and your engineering staff spends all their time “putting out fires” rather than improving the product. Worst case, you can’t keep up with interest payments and can only pay the bills (i.e. keep the system alive) by taking on further debt. Then you slide into technical insolvency, where the system never really works right.

Government systems seem to be at higher risk of technical debt or insolvency, for reasons that would require another post to unpack.

So Theory D is that the technology that receives domestic phone call data at the NSA has technical debt problems, and therefore is very difficult to upgrade. This is consistent with what we know about the system’s problems with enforcing compliance rules, and with the picture painted in the recent stories about a system that once handled nearly 100% of domestic calls sliding into a state of handling only about 25%.

If this theory is correct, one has to wonder how common technical debt is in NSA systems, and what this means for policy.