According to government documents studied by The New York Times, the FBI asked several phone companies to analyze phone-call patterns of Americans using a technology called “communities of interest”. Verizon refused, saying that it didn’t have any such technology. AT&T, famously, did not refuse.
What is the “communities of interest” technology? It’s spelled out very clearly in a 2001 research paper from AT&T itself, entitled “Communities of Interest” (by C. Cortes, D. Pregibon, and C. Volinsky). They use high-tech data-mining algorithms to scan through the huge daily logs of every call made on the AT&T network; then they use sophisticated algorithms to analyze the connections between phone numbers: who is talking to whom? The paper literally uses the term “Guilt by Association” to describe what they’re looking for: what phone numbers are in contact with other numbers that are in contact with the bad guys?
When this research was done, back in the last century, the bad guys where people who wanted to rip off AT&T by making fraudulent credit-card calls. (Remember, back in the last century, intercontinental long-distance voice communication actually cost money!) But it’s easy to see how the FBI could use this to chase down anyone who talked to anyone who talked to a terrorist. Or even to a “terrorist.”
Here are a couple of representative diagrams from the paper:
Fig. 4. Guilt by association – what is the shortest path to a fraudulent node?
Fig. 5. A guilt by association plot. Circular nodes correspond to wireless service accounts while rectangular nodes are conventional land line accounts. Shaded nodes have been previously labeled as fraudulent by network security associates.
i believe that this patriot act should be abolished because the FBI already has the authority to track terrorists if they have reasonable suspicion therefore making this act pointless.
Does any know the ATT, Verizon and Comcast share their FTTx network Infrastructure as a service to ISP or private content?
Architectural difference between the three vendors
How do they monitor the customer traffic? monitoring tool that is used and customer traffic attributes
solo cessione
Drop AT&T! Not a good deal.
Personal responsibility = good. Religious moralization and losing separation of church and state however = bad.
Funny you’d mention condoms, which are an important part of satisfying urges *responsibly*, taking measures to prevent undesired consequences. Condoms and sex education are a sign of a civilization trying to be mature and responsible. Dire religious fearmongering and preaching indicate Peter Pan syndrome: a tendency to want not to have to grow up and become responsible, to cling to fairy tales and a security blanket and try to preserve the childhood state of only having to do as father tells you and good old Dad will take care of everything for you.
Since security is as fake as the Wizard of OZ and the freedoms we enjoy are being supplanted by “freedom from” freedoms why do typically liberal folks keep quoting highly religious people like Benjamin Franklin?
The rights a freedoms he envisioned were not predominately military in nature but more of the securities people seek through wealth and having a nanny state wipe, powder and diaper the collective behinds of those who have returned to nursing on the mother government!
Using the means at our hands to catch those whose purpose it is to bring us into servitude to some socialist ideology is not freedom.
Just because a law prermits or prohibits something does not mean it is also correct, right or the way it really is or should be. Just because abortion is legal does not mean the environment, God or history teaches us it is a correct course of action. Nature has miscarriages, preditors etc… these laws govern the nature of life. When man creates a law that counteracts that law we are forced to inherit the results of those laws.
It is legal to clear cut forests in places. It’s legal for elementary schools to hand out condoms, but prohibited for opposite minded people to teach those same kids the mental, emotional and physical effects of succumbing to every urge that crosses the mind or body. Teaching people correct principles is a first step to people governing themselves. Teaching people that the government or some corporation will cover you for your stupidity is no way to ensure people even comprehend the law of action and reaction which is an imutable law. Though, there are many attempting to legislate consequences to be illegal.
Trying to legislate personal responsibility away from the individual does them no service. Liberals like to say we are a profit and corporate driven society yet expect those same “RICH” corporations to pony up millions to idiots who spill hot coffee on their crotches.
Until we expect humans to re-accept responsibility for their stupid actions we will keep looking at life as though someone else (i.e. the government) is responsible for my happiness, financial well being, education, and protection. Then those same people expecting this take care of me mentality say the government is taking their freedoms away though they desire to have a homogenized education and earning system.
Separation of School and State.
@Mike C: “Those of us who would prefer to put our heads in the sand, will likely wake up one day to a very sad reality and an extremely weak economy.”
Nearly got coffee all over my keyboard. Today, the Canadian dollar moved to an all-time high relative to the U.S. dollar.
Some people think they have already woken up to a sad reality and an extremely weak economy, and it isn’t because they haven’t given up enough privacy.
>> Those of us who would prefer to put our heads in the sand, will likely wake up one day to a very sad reality and an extremely weak economy.
We see the threat of terrorist attack with a clear eye and a calm heart.
While acknowledging that evil people can do horrible things, we choose not to live in fear and terror and allow little people with delusions of grandeur to control our hearts and minds.
We must first have liberty, for the state of the economy to have any meaning. The freedom to think is much more important than the freedom to buy.
Consider this: the term “traitor” was used as an epithet to describe people during the Revolutionary War. Treason remains the only crime defined in the Constitution as requiring the testimony of two witnesses to the same overt act, or confession in open court.
In the 21st century, we have allowed “terrorist” to take the place of the 18th century “traitor.” Labeling people as terrorists based on network analysis is like labeling people traitors based on who their friends and family are.
We also define corruption of blood and ex post facto laws as unconstitutional. This form of guilt by association smacks of both.
Tracking actual honest-to-goodness terrorists? With all that taxpayer money, you’d better.
Denying people the opportunity to work and to travel on the basis of secret government lists? Not in my Constitutional republic you don’t.
@mike c
I hope you didn’t miss this quote mike c, it rings quite true, and I love slapping idiots in the face every-so-often with how far removed they are from the founding ideals of the country they claim to be so fond of (when more and more it seems they are fond of little more than simply pissing all over it):
“Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety.†– Benjamin Franklin, in “An Historical Review of the Constitution and Government of Pennsylvania.†(1759)
Brilliant quote, contemporary psuedo-poly-sci-majors would do well to take note.
Please quit voicing your ignorant support of the destruction of our freedom in the name of security, because neither you or the idiot congressman who represents you have any competence or insight into how to properly secure anything, let alone secure it in a manner that preserves our essential freedoms. The chances of being killed in a terrorist attack in this spoiled trust-fund-baby of a country is less than that of being struck by lightning. Neither scenario scares me in the least, and it shouldn’t scare you either. What does scare me is the constant legislation of religious morality, the ever-increasing power shift from government to the mega-corporations, the ignorant masses who speak their recycled, spoon-fed ideas so loudly that their idiocy is all anyone can hear anymore, and the exponential downward spiral of our loss of essential civil liberties.
To all of you fools out there who suck at the nipple of this for-profit-fear-mongering nonsense: you need to have your heads examined, or be stripped of your right to vote since it seems you cannot think for yourselves.
The only thing this country truly needs is a fucking education, and maybe a diet. Wake up.
Unless your bad actors are extraordinarily socially isolated (which they can take care not to be), the fact that many of their links to other bad actors are within one or two hops only does you good if you have enormous monitoring capabilities. Perhaps if the bad-actor network converged in some fashion, traffic analysis would get you somewhere, but even decent tradecraft is going to prevent that.
“6 degrees of separation” means nothing in this technology. The links are not random. They are to a specific number: a suspected “bad actor”.
1st order linkages are useful. 2nd and 3rd order links to “bad guys” are probably very good to know (and may not be covered by a legal warrant).
I think the technology is useful and one day might catch a potential attack. I’ll trade off some privacy for a better economy and safety for my kids.
Those of us who would prefer to put our heads in the sand, will likely wake up one day to a very sad reality and an extremely weak economy.
@carrie:
If by “recently” you mean “in the last several decades”.
This is one where the infosec people are late to the party, not the social scientists.
Ok, here’s the problem. The algorithm by itself is worthless. Once you have identified a bad actor, then it becomes useful. Once you have identified a bad actor, you can also get a legal warrant and all their call information. You can also get a warrant for anyone they called on the grounds of probable cause.
Which simply means there is no reason or excuse for breaking FISA, since the only useful information is also easily obtainable by following the law. The only thing that would be denied a warrant would be calls to or from people not connected to a bad actor/terrorist. Which also means the only reason to get that data on people not connected to bad actors/terrorists is because you want to do something illegal with it.
Very intelligent post Andrew. This is what keeps America great-the freedom to examine and challenge the efforts and policies of the agencies empowered to protect us. We must utilize technology to track bad actors. If that means our signal output is tracked, so be it. Just because we have the freedom to do nothing, doesn’t mean we will remain free forever. There is a price to pay for freedom. The pursuits to keep us free, at times, carry a heavy burden. Though I respect the right of those that choose not to acknowledge the threat of terrorism, I trust in the American community at large to recognize this grave danger and step up and join the fight in Keeping America free. This wonderful technology to track known terrorists and their connections. We all should endeavor to make sure the power to do so, isn’t abused and our liberties aren’t trampled upon. Keep up the good work Andrew. BTW, do you have any alternative solutions?
The ‘terrorist threat’ is the bogeyman being sold to lock-down the entire planet into a totalitarian information grid.
“Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety.” – Benjamin Franklin, in “An Historical Review of the Constitution and Government of Pennsylvania.” (1759)
“The purpose of government is to protect the secrecy and the privacy of all individuals, not the secrecy of government.” – Ron Paul, from the California Republican Debate May 3, 2007,
Write AT&T a letter and drop your contract on privacy violation grounds. Inform yourself of the writings and positions of the Hon. Rep Dr. Ron Paul. Please donate generously to his campaign and get your fellow citizens to vote for him in the upcoming Republican primaries.
And I agree that terrorism is just the catch phrase used to rob us of our liberties. There was more than enough information before 9/11 to catch the terrorists — our “intelligence” services and worse, our executive branch simply did nothing. Before we give up our rights and roll back the Constitution, why not try using competence first?
“Any two people on the planet are seperated by no more than six degrees.”
That simply isn’t true. It’s true for most people — but there are isolated pockets of people who are very far removed from civilization. There’s at least one person who’s known to be the last of his group and has never communicated with other humans — so he’s infinitely separated from every other human. There are also known to be tribes that have no contact with the outside world, particularly in South America but also in Indonesia and probably in Africa. There, everyone is one degree away from everyone else in their group, but infinitely separated from the rest of the world.
Terrorists? We still falling for that elaborate ruse to empty the Treasury, lock down remainding natural resources and re-write the Constitution into toilet paper?
@richard – If you are mining from scratch, the analysis is true and it is still a needle in a slightly smaller hay stack. However, if you have a known bad actor, this analysis can be used to check out friends and friends of friends. As you start getting additional hits (connections to other known bad actors), patterns begin to emerge.
The analysis presented here is what happens when people with a lot of knowledge and little experience try to understand a problem…
Drop AT&T! Not a good deal.
Umm…that should be “…the pool of innocents is far greater than the pool of bad actors…”
@Mark: The problem is precisely what you state – the pool of innocents is far greater than the pool of innocents.
Try this with some simplified numbers. Take a population of 1,000,000. Imagine that 1 in 1,000 is a bad actor (leaving 999,000 innocents).
Bad actor within 2 hops: 60% = 1,000 * 0.6 = 600
Innocents within 2 hops: 15% = 999,000 * 0.15 = 149,850
So you have identified over 150K people (15% of your population), and you know that as many as 0.4% of them are bad actors. Not particularly helpful in the grand scheme of things.
Now try that with more realistic numbers. US Population: 300 million (with 80%+ phone ownership). Bad actor rate: 1 in 10,000? 1 in 100,000? 1 in a million?
Taking the “best” values (for finding bad actors) that gives something like[1] 36,000,000 innocents and 18,000 bad actors. Does that sounds like a good, precise, targeted technology to you?
[1] Taking 300 million population at 80% phone ownership versus 1 in 10,000 bad actors with 100% phone ownership
See also: http://www.americanscientist.org/AssetDetail/assetid/53062
@Paul: That’s one of the things the excerpted graphs address. For the “credit card fraud” problem, 60% of the bad actors are within 1 or 2 hops of another bad actor. But, more than 60% of innocents are 4 or more hops away from a bad actor. (Unfortunately the pool of innocents is far larger than that of bad actors…) It’s not necessary to go to 6 hops to connect the dots between bad guys.
However, it is not clear that the “terrorism” problem is amenable to the same analysis.
There is a distinct problem with this. The network expands with extraordinary speed. A famous rule of thumb is that of six degrees. Any two people on the planet are seperated by no more than six degrees.
This has recently been becoming a “popular” methodology in the social sciences, but is called Social Network Analysis. If anyone is interested in how it can be interfaced with psychology through the introduction of behavioral correlations and predictions.
This is an easy one to play with. Any terrorist/drug dealer who suspects that their phone number or that of an associate whom they call may be compromised can just use a regular modem & free-to-download program to dial up hundreds (if not thousands) of random phone numbers. To make it harder to screen out the “noise”/false positives this would generate, they can have a pre-recorded message of “you have won a prize! continue to listen to hear how to collect” to have the receiver stay on the line longer. Personally, I think EVERYONE (innocent and guilty alike) should place calls to local & federal political officials & gov’t offices to ensure they stay in the “monitoring” loop too.
I’m fascinated to see how truncated the communities of interest apparently are, according to the graph you reproduce. Friends and friends of friends, but no friends of friends of friends at all. If that holds for terrorists (and there’s no a priori reason it shouldn’t) then all of the broad-scale data-mining work is so much garbage.
(hope this doesn’t wind up a double comment, since my original is not visible)
To quote the messiah Brian: Worse? How could it get any worse?
update customer c set terroristlevel = (select sum(terroristlevel) from customer rc where rc.lastcallerid = c.customerid)
Surely one SQL statement can’t be as unerringly accurate as a complex data mining rig that even comes with its own academic-sounding whitepaper… but using either one as an input to create a list of people who are now considered terrorists doesn’t seem quite sane. I guess those numbers have to come from somewhere.