April 25, 2014

avatar

Low Hit Rate Isn't the Problem with TSA Screening

The TSA, which oversees U.S. airport security, comes in for a lot of criticism — much of it deserved. But sometimes commentators let their dislike for the TSA get the better of them, and they offer critiques that don’t stand up logically.

A good example is yesterday’s USA Today article on TSA’s behavioral screening program, and the commentary that followed it. The TSA program trained screeners to look for nervous and suspicious behavior, and to subject travellers exhibiting such behavior to more stringent security measures such as pat-down searches or short interviews.

Commentators condemned the TSA program because fewer than 1% of the selected travellers were ultimately arrested. Is this a sensible objection? I think not, for reasons I’ll explain below.

Before I explain why, let’s take a minute to set aside our general opinions about the TSA. Forget the mandatory shoe removal and toiletry-container nitpicking. Forget that time the screener was rude to you. Forget the slippery answers to inconvenient Constitutional questions. Forget the hours you have spent waiting in line. Put on your blinders please, just for now. We’ll take them off later.

Now suppose that TSA head Kip Hawley came to you and asked you to submit voluntarily to a pat-down search the next time you travel. And suppose you knew, with complete certainty, that if you agreed to the search, this would magically give the TSA a 0.1% chance of stopping a deadly crime. You’d agree to the search, wouldn’t you? Any reasonable person would accept the search to save (by assumption) at least 0.001 lives. This hypothetical TSA program is reasonable, even though it only has a 0.1% arrest rate. (I’m assuming here that an attack would cost only one life. Attacks that killed more people would justify searches with an even smaller arrest rate.)

So the commentators’ critique is weak — but of course this doesn’t mean the TSA program should be seen as a success. The article says that the arrests the system generates are mostly for drug charges or carrying a false ID. Should a false-ID arrest be considered a success for the system? Certainly we don’t want to condone the use of false ID, but I’d bet most of these people are just trying to save money by flying on a ticket in another person’s name — which hardly makes them Public Enemy Number One. Is it really worth doing hundreds of searches to catch one such person? Are those searches really the best use of TSA screeners’ time? Probably not.

On the whole, I’m not sure I can say whether the behavioral screening program is a good idea. It apparently hasn’t caught any big fish yet, but it might have positive effects by deterring some serious crimes. We haven’t seen the data to support it, and we’ve learned to be skeptical of TSA claims that some security measure is necessary.

Now it’s time for the professor to call on one of the diehard civil libertarians in the class, who by this point are bouncing in their seats with both hands waving in the air. They’re dying to point out that our system, for good reason, doesn’t automatically accept claims by the authorities that searches or seizures are justified, and that our institutions are properly skeptical about expanding the scope of searches. They’re unhappy that the debate about this TSA program is happening after it was in place, rather than before it started. These are all good points.

The TSA’s behavioral screening is a rich topic for debate — but not because of its arrest rate.

Comments

  1. Strilanc says:

    A: 1 criminal was found out of every 1000 searched.
    B: Searching caused 1 in 1000 criminals to be caught.
    These two statements are not equivalent!

  2. Khürt says:

    I think the flaw in the TSA approach to security is that the TSA is looking to prevent terrorism versus just doing risk management. If we go by the WikiPedia definition:

    The objective of risk management is to reduce different risks related to a preselected domain to the level accepted by society.

    Risk management assumes that not all risks are known and for the ones that are known the cost of “prevention” is too high. Some risks must be accepted.

  3. Crosbie Fitch says:

    The TSA is not designed to appease statisticians, intellectuals, libertarians, or even to catch terrorists.

    It is designed to reduce terror (the perception of risk, not the actual risk), and in that respect it probably has been fairly successful.

    That doesn’t sanction TSA from a civil rights perspective, but no doubt those who did consider it sanctionable had other perspectives.

    • Anonymous says:

      It is designed to reduce terror (the perception of risk, not the actual risk), and in that respect it probably has been fairly successful.

      Speaking of which, it has also been fairly successful against attacks of pink elephants, martians taking over the congress, and the resurrection of Elvis Presley. I mean, not a single incident of those has been occured either since the TSA theatre has started.

    • Tel says:

      The primary purpose of the TSA (just like every government department) is to ensure that their pay comes in at the end of the month, and their budget is not cut. In order to do this, they must convince people that we need them. Thus, their primary interest is to increase the perception of terror by constantly reminding travelers about these “deadly crimes” that float somewhere in the shadows.

      If anyone wants to check the stats for the 20th century, governments killing their own people resulted in several times as many deaths as all the soldiers killed by enemy military action. Statistically, terrorism does not exist in comparison to murder perpetrated by police forces.

      I recommend that everyone reading this article take the time to review R. J. Rummel’s law. Loss of freedom costs lives.

  4. Brent says:

    I hate the word commentator. It bugs me.

    I call this kind of argument a ‘haggling over the price’ fallacy. You’re right, the ineffectiveness of semi-random searching at airports is irrelevant to the rightness or wrongness of the search. When people point to the fact that people DO get caught, they are agreeing that in principle being searched semi-randomly is a bad thing, but they’re just haggling over the price before they accept the truth that it’s JUST a bad thing not a justifiable bad thing.

  5. Vincent Clement says:

    Crosbie: It may reduce perception of risk, but it increases the general grumpiness of the travelling public. Flying in the US has become a joke.

  6. Anonymous says:

    “You’d agree to the search, wouldn’t you?”
    No, because I respect my rights. Any reasonable person would not trade in their rights to save a deadly crime. Sure we might be able to catch more criminals with torture, but does that make it right? No, because we might torture an innocent person. Just like we might search an innocent person. An extreme example, sure, but its the principle. No illegal searches.

  7. Mark Huss says:

    // Forget the mandatory shoe removal and toiletry-container nitpicking. Forget that
    // time the screener was rude to you. Forget the slippery answers to inconvenient
    // Constitutional questions. Forget the hours you have spent waiting in line.

    Hmm, that’s a lot to forget!

    I really think Bruce Schneier has it right on this subject: movie-plot security theater, which greatly inconveniences *all* travelers, (supposedly) makes us feel better, but does nothing statistically significant to decrease our risk.

    • felten says:

      As I understand it, Schneier actually likes the idea of behavioral profiling, at least in the abstract. That is, he advocates fewer procedural hoops (taking off shoes, etc.) and more reliance on well-trained, vigilant security personnel.

  8. Gary McGath says:

    An 0.1% rate of “stopping a deadly crime” simply isn’t equivalent to an 0.1% arrest rate. You seem to acknowledge that in the next paragraph, so I can’t understand what point you’re making.

    If the TSA simply stopped people randomly for searches, it could probably find grounds for arresting 1 in 1000. If that’s true, then behavioral profiling is giving the TSA no information whatsoever.

  9. TimH says:

    Ed – I’m have been reading your’s and Bruce’s stuff for a while, and based on that training, your arguments really don’t make sense:
    1. Firstly, there’s no quantative data on the magic 1%. “fewer than 1% of the selected travellers were ultimately arrested.” For what? Unpaid fines? Remember that the TSA’s function is largely illusion, so faking success will be part of their job. I am very sure that if any terror plots were foiled, that would be announced in full detail. We have been given no details, so my disengenuousness detector is blinking wildly.
    2. Agreeing will the comment above on this: “…give the TSA a 0.1% chance of stopping a deadly crime. You’d agree to the search, wouldn’t you?”. Then let’s search every house, every car, every computer then. There may be deadly crimes being planned. It’s the same argument! That Constitution specifically balanced the privacy/harrassment vs find-the-criminals arguments, and put law in place respecting the former.
    3. Lastly, don’t forget Bruce’s oft-repeated point regarding focus on the method. So, it becomes impossible for a passenger to walk on a plane with a weapon by searching everyone. How about multiple passengers bringing parts across? Parts being pre-concealed on the plane? A lot more people have plane access than just the passengers.

    • felten says:

      My point is just that the “less than 1%” number, by itself, does not establish that the program is unjustified. If, as you suggest, the rate at which the system apprehends truly dangerous ticking-bomb criminals is zero, *that* is a good argument against. If, as you also suggest, this tactic causes attackers to shift to another, equally effective, mode of attack, that is also a good argument against.

      I’m not saying that the program stands up to all of these criticisms. I’m only saying that critiquing it because of its less than 1% arrest rate doesn’t make sense.

      • TimH says:

        You are effectively supporting TSA’s product of an unquantified* number (1%) by inventing scenarios where their actions are good. You have a lot of credibility in the risk/security field, so have leverage through your blog to insist to Hawley to substantiate the 1% statement. By de facto accepting the 1% statement as an acceptable talking point, you are moving the discussion beyond the probable Big Lie that it represents. Instead, your piece need be taken only a little out of context to mean “Dr Felten supports passenger screening because it’s a success”. With respect, your attempt to play fair is politically naive.

        *What offences (and the relevance to air protection) the 1% committed is undisclosed

        • Anonymous says:

          With respect, your assumption that Dr. Felten’s goal is to play politics seems to be suspect, particularly in light of his actions.

          • TimH says:

            I wasn’t suggesting that the good Prof. was being political. My point is that others could misuse his piece to imply his support of the searches, when that wasn’t his discussion.

          • Anonymous says:

            Well, then, it must instead be that your assumption is that Prof. Felten should play politics, and that his choice not to do so is somehow incorrect.

            Of course, it’s actually your assumption that’s suspect.

  10. rp says:

    Well, yes, the low arrest rate — especially when the arrests are for crimes unconnected with terrorism — is a problem. That’s because the TSA’s (and the federal government’s) resources are finite. So when Kip Hawley is asking you to submit to the pat-down, he’s also telling you that his people won’t be doing a bunch of other things (cargo screening, securing airport perimeters) that quite possibly have a greater chance of stopping a (terrorist) crime. What if he said, “If you agree to this pat-down, there’s a 0.1% chance of stopping a crime, but the person doing the patdown would otherwise be preventing ground personnel from bringing in contraband, with a 0.2% chance of stopping a crime.”

    On the federal level, that patdown is taking money away from countless other initiatives that could stop crimes or save lives in other ways, but Hawley isn’t disclosing that either.

    And, of course, on the personal level, there’s the time you and your fellow travelers spend waiting for patdown, which is time you might be spending saving lives in other ways. Also lost. Add to that the 0.00001% chance that outrage at having your civil liberties violated to no good purpose sends you over the edge and turns you into a jihadi collaborator…

    • Anonymous says:

      The TSA probably meets political resistance to doing anything more effective, such as screening all access to the plane, by the DEA and other forces.

      Consider the DEA, in particular: an effective TSA might reduce drug smuggling success by air. That reduces the scale of the War on Drugs on the home front. That means the DEA gets to eat a pay cut.

      So, departmental turf war time!

  11. Another Kevin says:

    Selected travelers who were arrested were charged mostly with drug violations and false ID. I wonder what fraction of those were convicted of anything at all. My suspicion is that the lion’s share of the “false ID” cases consisted of people who unwittingly presented expired drivers’ licenses. And I wonder what fraction of the “drug violation” cases consisted of people who had prescription medications in, say, a 7-day “pill minder” rather than appropriately labeled prescription containers. (Associates of mine have been threatened with arrest for a “pill minder”, and even for an original prescription bottle, because the latter was “not sealed”.)

  12. PsiCop says:

    Concerning the proposition:

    “Now suppose that TSA head Kip Hawley came to you and asked you to submit voluntarily to a pat-down search the next time you travel. And suppose you knew, with complete certainty, that if you agreed to the search, this would magically give the TSA a 0.1% chance of stopping a deadly crime. You’d agree to the search, wouldn’t you?

    I’d have to say that I would not agree to this search, and for a very good reason: I already KNOW I’m not involved in a crime, and therefore KNOW that searching me will be a complete waste of my time and Hawley’s. I would MUCH rather that OTHER people — who may or may not be involved in a crime, so far as I know — get searched instead.

    Ed is correct that the arrest-rate should not be the sole determinant of success. A more useful determinant would be a tally of actual crimes thwarted, not merely arrests. Even this, however, would probably still be a very low number, so we risk being deceived by it anyway.

    TSA’s real problem is credibility. They make travelers jump through hurdles which are asinine and ineffective. Removing shoes? Forbidding liquids? Please! These are annoyances that everyone must face, but which produce nothing significant. All they do is create an appearance they’re doing something to secure planes … but these things do little or nothing in that regard. One of the biggest, and most real, holes in air travel security is baggage, which is still not universally searched for explosives (only some bags are sniffed by dogs or detectors). TSA searches everyone’s shoes but not everyone’s checked baggage. How much sense does that make?

    Until TSA does more about dangers which are real, and stop doing things purposely to annoy travelers in the name of making them feel more secure, any assertions they make about the effectiveness of anything they do MUST be treated with suspicion. It’s really up to TSA to convince travelers that they’re genuinely interested in providing true security. Once they do, they will stand on much better ground when they claim their programs are successful.

  13. Wayne says:

    The point of screening is to deter criminals by their being a reasonable chance of being caught before you commit a criminal act. If the potential criminal believes there is a chance of being caught, he will look for another place to commit the crime. Terrorism is just one crime; others are just as worthy.

    There’s a lot wrong with the current screening process. Silly measures, such as banning all liquids but not having any consequences for attempting to bring them through security, don’t deter. (See Bruce Schneier’s web site for a much better argument that I can make.)

    However, selecting people based on their behaviour is better than just random selection. Nervous people are nervous for a reason. It may be, and in most cases is, completely unrelated to any criminal activity. However, somone attempting to commit a crime is more likely to be nervous, than someone who is just trying to catch a flight. Since this increases the chances of being caught, it serves as a greater deterent.

    We shouldn’t be criticising this measure. We should be looking to replace the many other ineffective measures with ones that actually deter criminal acts.

  14. Village Idiot says:

    Regular people committing petty crimes, in this case meaning a crime with no intent to harm others, will be nervous and possibly trigger secondary screening after behavioral profiling. A true psychopath on the other hand is the most likely candidate for committing a violent act intended to harm others and will pass any behavioral screening with flying colors. He or she will not be nervous or agitated at all due to a complete lack of empathy for others (serial killers are likewise often found to be very charming by their eventual victims because they are so calm and confident).

    Likewise, a religious fanatic who kills for their god will probably not be nervous either due to relentless indoctrination of the righteousness of their mission and a belief they are securing a place in Paradise for themselves, which I would imagine could make their behavior appear identical to someone going on vacation.

    I call shenanigans on the TSA, but not while I’m quietly and compliantly standing in one of their lines lest I meet the agent in the back room who wears latex gloves. The first time is always the worst, but after awhile you’ll get used to it and it’ll seem like business-as-usual.

    • Bitmonger says:

      Psychopaths are not going to be responsible for most violent acts. You are right they could slip right passed this kind of screening, but the idea that many terrorists or murderers are psychopaths is unlikely. It may seem be comforting to think special amoral people to commit most truely evil acts, but the world is not so simple.

      History shows us: Normal people can do horrible things without being a psychopath and most horrible acts are carried out my “normal” people.

    • RH says:

      I’m actually rather interested in two studies:
      1. has a study been done profiling what a terrorist will actually behave like?
      2. could such a study be made public without defeating its own goals by telling the terrorists what not to do?

  15. Jay Clip says:

    Dear Mr. Felton – You trivialize the number of fake IDs TSA intercepts by implying that it’s just a matter of ticket fraud. It isn’t. It’s often a matter of a criminal or an illegal immigrant (another type of criminal, IMHO), who has a forged instrument of identification intentionally acquired with the intent to commit deception. If these people can’t at least be honest about their identity, do you really want them in the seat next to you on your plane, eyeballing your MP3 player, iPod, iPhone, Blackberry, or the innumerable valuable items people leave in airplane seatback pockets while they go to the lavatory, or when they store their carry-ons, purses, etc three rows away from where they’re sitting. Try to track a criminal down on an airline’s manifest? Fake ID – no joy, remember?

  16. Avi8tor says:

    Terrorism: The Syrian Connection :: Daniel PipesTerrorism: The Syrian Connection :: Daniel Pipes. … probably under orders from Syrian intelligence, with Ann-Marie Doreen Murphy. Murphy, a 32-year-old, …
    http://www.danielpipes.org/article/1064

    Note: Whether it be the use of fraudulent travel documents, artfully concealed drugs on the passenger, or as noted below, a real attempt at terrorism, the TSA BDO program is worth it. Always remember: all terrorists are criminals and engage in criminal behavior (see also: The Seattle Times Series: The Terrorist Within)

    ….It was no accident that Murphy was caught, and in large part because her actions roused suspicions. To begin with, her ticket had been rebooked, an action which automatically triggers special scrutiny by El Al. Then the security interview at Gate B23 was a calamity. Murphy passed through X-ray inspection without problems and reached the gate with the bag still on an airport cart. There she waited calmly until about 9:10 a.m., when an El Al agent ran through the company’s standard questions with her. First, when asked whether she had packed her bags by herself, she answered in the negative, setting off bells. As reconstructed by Neil C. Livingstone and David Halevy, the following exchange went approximately like this:

    “What is the purpose of your trip to Israel?”

    Remembering what she had been told to answer, she answered, “For a vacation.”

    “Are you married, Miss Murphy?”

    “No.”

    As unmarried pregnant Irish women do not often go for vacation in Israel, the agent probed further. “Traveling alone?”

    “Yes.”

    “Is this your first trip abroad?”

    “Yes.”

    “Do you have relatives in Israel?”

    Hesitating, Murphy relied, “no.”

    Every reply increased the passenger’s oddity. “Are you going to meet someone in Israel?”

    “No.

    “Has your vacation been planned for a long time?”

    “No.”

    Quizzing her now with some intensity, he asked: “Where will you stay while you’re in Israel?”

    “The Tel Aviv Hilton.”

    “How much money do you have with you?”

    “Fifty pounds [about $70].”

    “Do you know how much a room costs at the Hilton?” Then, not waiting for an answer (it costs at least $100 a night), he asked, “Do you have a credit card?”

    “Oh, yes,” she replied, and proceeded to produce from her purse an identification card for check-cashing purposes.

    This was too much. Convinced that something was amiss, the agent emptied the bag and found it “quite heavy,” with “a sort of double bottom.” He sent Murphy to be body-searched and took her bag to a staff room. Although she had nothing on her person, inspection of the bag turned up a plastic bag at the bottom full of a yellowish, oily substance – Semtex. A closer look then turned up the detonator in the Commodore calculator.

  17. Anonymous says:

    I know it’s been brought up repeatedly but how could we ever measure how much HASN’T happened as a result of the additional security measures? If I think like a terrorist for a second, had the 911 response appeared more tepid and unfocused, I’d be mighty tempted to try it again and again until the authorities actually did something that appeared to have the potential to catch me.