The Myth of Total Information Awareness

2006-03-01T15:21:00-08:00

Lost in the last few weeks in the incessant coverage over Dick Cheney’s decision to hunt the ultimate prey were some interesting revelations about the NSA’s new mechanisms for spying on electronic communications. The Christian Science Monitor broke this story first, report on a system named ADVISE that would spider blogs, wishlists and other relics of online presence to build up dossiers on people. Some people thought that ADVISE was simply the rejected Total Information Awareness (TIA) program in new clothing, but Newsweek in a truly excellent work of reporting broke the information that the core of TIA was actually renamed to a project called Topsail and ADVISE was something else. TIA’s core motivation was to simply scan communications for “suspicious activities” and then notify human analysts of potential problems. ADVISE seems to have a much grander scope of building up dossiers of people’s interests and intents to identify “suspicious people” instead. Yikes.

In some sense, this is nothing new. Way back in 1993, the NSA made waves by trying to pass through a mandatory encryption standard called the Clipper chip that would enable the government to decrypt any encrypted communications and the Echelon project has been steadily accumulating intercepted electronic communications under the NSA’s purview. But the NSA has always had issues analyzing the volume of messages they grab and very, very little of the data they retrieve ever makes its way in front of an analyst. Making light of this, some Emacs developer even added a feature M-x spook that would spit out a series of suspicious words suitable for activists to add to their email and overwhelm the already overstrained capabilities of the government like so:

top secret SAFE terrorist ANZUS New World Order enforcers radar TELINT Serbian advisors FIPS140 INSCOM government CNCIS secure

But here we are again, with the government claiming a need to spy on us and the media leading a fight against it. If it continues like it has though, we fighting this are destined to lose. The problem is that most of the ensuing discussion of the government’s data mining operations have been like those for the wire tapping scandal; criticism is focused on the political and ethical problems of the systems and lies are exposed, but the underlying technical problems are glossed over by tech-averse journalists. Indeed, most discussions on the legality of wire tapping implicitly assume that the technology is completely effective but should be avoided strictly out of moral concerns. To see what I mean, recall how the debate over torture that started last year played out in newspapers and TV shows. The “anti-torture” side would start the argument by positing the moral horror of torture. The “pro-torture” side would almost always then counter with a hypothetical situation of capturing a mastermind who we are unquestionably certain knows about a master plot to destroy an American city and from whom torture is the only way to get such information. The torture advocate thus sidesteps the moral horrors of the situation by claiming there is no viable alternative. Of course, such situations never happen and torture rarely ever yields true information, but that’s besides the point. The argument is thus glibly reduced to “idealists vs. pragmatists”, and in these times the pragmatists always win the debate for public opinion.

This process will likely happen again with the debate over data mining. That Newsweek article does an exemplary job of exploring the technical issues, but they’re an exception to the rule. The pragmatists will advance the argument that nobody really like spying on Americans, but it’s the only way to catch the bad guys. And this is what makes me really upset. It’s bad enough that data mining is likely illegal and invasive, but even more galling that the system most likely will never work in the first place.

eavesdropping emc ARPA HAMASMOIS Aldergrove AGT. AMME Freeh White House jihad csystems MIT-LL 22nd SAS NWO pink noise mania

So, what are the technical problems that make such a system unfeasible? For starters, this isn’t actually data mining. I used to work at a data mining software developer (Dimensional Insight) and the goal of those products was to organize complicated data into easily traversable ways for analysts to drill down for connections. In a typical case, you might want to look over your sales data for the last year to see how products sold in particular parts of the country, which sales divisions did best, or similar queries. The process is human-driven and its sole purpose is to represent complex multi-dimensional data (ie, price, product, sales person, city, region, state, time, correlation to other product sales, etc.) in an easily viewable and usable manner to drill down through the data for connections. In addition, data mining involves looking backwards from the present to gain insight into past purchasing patterns to drive future sales (the classic success story is a supermarket finding a correlation between diaper and beer sales).

Instead, the NSA largely seems to be interested in predicting novel future behavior and retrieving warnings when suspicious activities occur. This is actually more in line with Artificial Intelligence research on classifiers. Essentially what the NSA seems to be striving for is some sort of theoretical Threat Box which can be fed a steady stream of events and spit out a warning for human analysts to follow up in certain cases. Whether the backend classifier is a neural net, support vector machine, or other sort of technology, the process of training classifiers usually includes the same parts. First, a training set needs to be assembled out of a mixture of scored positive and negative events. So, if you were creating a “terrorist email classifier”, the positive events might be emails from Osama Bin Ladin, the negative would be emails from your Aunt Sue (there are usually many more negative than positive selections). When the testing is done, a similarly pre-classified testing set is used to evaluate how good the classifier actually is. The goal of a well-trained set is to make the correct correlations between certain inputs (so that the presence of “bomb” and “white” and “house” triggers an alarm), but a constant risk with such systems is the danger of erroneously assuming certain correlations are meaningful (eg, all of Mohammed Atta’s emails included the word “the”, so the system concludes that “the” indicates the email is dangerous). To minimize such problems, both the training and test sets usually go through a process of feature selection, where meaningless information is filtered out so it doesn’t affect the classifier. If this sounds like more of an art than a science, it is and there are several ways in which errors can manifest:

Human Bias – people are necessary to select and classify the training and test sets as well as feature selection. This can create biases in the system that reflect the assumptions of the creators. As Malcolm Gladwell’s excellent article on profiling explored, such biases create systems that solve the wrong problems and vulnerabilities exploitable by intelligent attackers.
Too Little Human Bias – on the opposite end of the spectrum, it’s possible to have too much faith in the effectiveness of a classifier. The difficulty here is that the judgement of the computer will be accepted as absolute. One problem is that it generally is impossible to extricate any clear explanation of the classification’s reasoning (any explanation is like teasing out thought at the neuron level, making it too low-level to be sensible). Furthermore, even if such explanations were available, the experience with the TSA’s No Fly Lists suggests they would not be made available to agents acting on the classifier output. At best, this would only mean that the NSA is inundated with erroneous data. At worst, it could lead to extensive spying, internment, and misguided strategic directions. Best not to contemplate this further.
False Positives and Negatives – any classification system usually contains some mix of false positives and negatives. In this system, the political pressures seem to mandate that the false negatives (ie, missing a threat) should be 0 if at all possible. Unfortunately, minimizing the false negatives invariably increases the false positives, meaning that more events will be erroneously triggered. For the dangers this presents, look at item #2.
The Wrong Tool For the Job – Even if the classifier were able to achieve a remarkable level of correctness and accuracy, it’s still possible it would be the wrong tool for the job. As one blog has observed, Osama Bin Ladin probably isn’t clicking around on Amazon, meaning that these tools for signal intelligence won’t be very useful if the enemy is not creating any signals for them to detect. How likely are terrorist cells going to be using email when you can fedex or hand-deliver documents anywhere around the globe in a few days? How hard is it to exploit steganography or misdirection to thwart any tracking systems? In this case, this elaborate NSA system just becomes another example of America’s heavy reliance on technology (smart bombs, sigint, spy satellites) over the crude and dirty human-based methods of gathering information and waging war. How many gaps in our knowledge will this system ever fill? Or are there other ways to more effectively gather information for the cost of building this surveillance infrstructure?

freedom Kennedy chameleon man mindwar BROMURE Echelon TELINT Armani Marxist Bletchley Park FIPS140 nuclear supercomputer mania USDOJ

Of course, the NSA will claim their systems are effective and already performing vital tasks in the war on terror. Indeed, the article about ADVISE reports “the system – parts of which are operational, parts of which are still under development – is already credited with helping to foil some plots.” But what are the nature of these plots? Even if I give the NSA credit and assume they aren’t being duplicitous about current problems in the hopes they can fix them later (a condition known as hope creep), I wonder if the same circular logic presented in the case of Guantanamo Bay detainees will also be applied here: Guantanamo Bay is only for Al Qaeda terrorists, therefore everybody interred there must be a dangerous terrorist. Again, what are these thwarted attacks? Are they real, orchestrated and viable or just some bored teens talking smack on MySpace? Such information will never be made publicbecause the NSA needs foiled plots to justify the system and thus any vagule plausible detected messages will become foiled terrorist plots. Meanwhile we lurch closer to national insolvency, confident in our abilities to detect the next 9/11 if only those darn terrorists would play by our expected rules. And we lay the groundwork of a national surveillance state, just ready to be exploited by some avaricious future leaders.

So, what is to be done? I wish there was something. If the stakes were not so high, I’d suggest that it might be worthwhile to update M-x spook with a whole new lexicon to sabotage the surveillance mechanisms now before more money is thrown into the whole. But these are no times for pranksters, and the only real solution is for Congress to exert the oversight they have. They killed TIA one already and the could do it again. The oversight is theirs; if only they had the courage and the wisdom to use it.

Nimble Code: Tag advise

The Myth of Total Information Awareness