According to government documents studied by The New York Times, the FBI asked several phone companies to analyze phone-call patterns of Americans using a technology called "communities of interest". Verizon refused, saying that it didn't have any such technology. AT&T, famously, did not refuse.
What is the "communities of interest" technology? It's spelled out very clearly in a 2001 research paper from AT&T itself, entitled "Communities of Interest" (by C. Cortes, D. Pregibon, and C. Volinsky). They use high-tech data-mining algorithms to scan through the huge daily logs of every call made on the AT&T network; then they use sophisticated algorithms to analyze the connections between phone numbers: who is talking to whom? The paper literally uses the term "Guilt by Association" to describe what they're looking for: what phone numbers are in contact with other numbers that are in contact with the bad guys?
When this research was done, back in the last century, the bad guys where people who wanted to rip off AT&T by making fraudulent credit-card calls. (Remember, back in the last century, intercontinental long-distance voice communication actually cost money!) But it's easy to see how the FBI could use this to chase down anyone who talked to anyone who talked to a terrorist. Or even to a "terrorist."
Here are a couple of representative diagrams from the paper:
Fig. 4. Guilt by association - what is the shortest path to a fraudulent node?
Fig. 5. A guilt by association plot. Circular nodes correspond to wireless service accounts while rectangular nodes are conventional land line accounts. Shaded nodes have been previously labeled as fraudulent by network security associates.
Front page posts
(hope this doesn't wind up a double comment, since my original is not visible)
To quote the messiah Brian: Worse? How could it get any worse?
update customer c set terroristlevel = (select sum(terroristlevel) from customer rc where rc.lastcallerid = c.customerid)
Surely one SQL statement can't be as unerringly accurate as a complex data mining rig that even comes with its own academic-sounding whitepaper... but using either one as an input to create a list of people who are now considered terrorists doesn't seem quite sane. I guess those numbers have to come from somewhere.
I'm fascinated to see how truncated the communities of interest apparently are, according to the graph you reproduce. Friends and friends of friends, but no friends of friends of friends at all. If that holds for terrorists (and there's no a priori reason it shouldn't) then all of the broad-scale data-mining work is so much garbage.
[...] Link [...]
[...] Freedom to Tinker] Link to This [...]
This is an easy one to play with. Any terrorist/drug dealer who suspects that their phone number or that of an associate whom they call may be compromised can just use a regular modem & free-to-download program to dial up hundreds (if not thousands) of random phone numbers. To make it harder to screen out the "noise"/false positives this would generate, they can have a pre-recorded message of "you have won a prize! continue to listen to hear how to collect" to have the receiver stay on the line longer. Personally, I think EVERYONE (innocent and guilty alike) should place calls to local & federal political officials & gov't offices to ensure they stay in the "monitoring" loop too.
This has recently been becoming a "popular" methodology in the social sciences, but is called Social Network Analysis. If anyone is interested in how it can be interfaced with psychology through the introduction of behavioral correlations and predictions.
There is a distinct problem with this. The network expands with extraordinary speed. A famous rule of thumb is that of six degrees. Any two people on the planet are seperated by no more than six degrees.
@Paul: That's one of the things the excerpted graphs address. For the "credit card fraud" problem, 60% of the bad actors are within 1 or 2 hops of another bad actor. But, more than 60% of innocents are 4 or more hops away from a bad actor. (Unfortunately the pool of innocents is far larger than that of bad actors...) It's not necessary to go to 6 hops to connect the dots between bad guys.
However, it is not clear that the "terrorism" problem is amenable to the same analysis.
[...] Andrew Appel created an interesting post today on AT&T Explains Guilt by Association.Here’s a short outline:When this research was done, back in the last century, the bad guys where people who wanted to rip off AT&T by making fraudulent credit-card calls. (Remember, back in the last century, intercontinental long-distance voice communication … [...]
[...] clipped from www.freedom-to-tinker.com [...]
[...] 30th, 2007 Ed Felten’s blog relays a report from the New York Timesexplaining the target selection strategies used to detect [...]
See also: www.americanscientist.org/AssetDetail/assetid/53062
@Mark: The problem is precisely what you state - the pool of innocents is far greater than the pool of innocents.
Try this with some simplified numbers. Take a population of 1,000,000. Imagine that 1 in 1,000 is a bad actor (leaving 999,000 innocents).
Bad actor within 2 hops: 60% = 1,000 * 0.6 = 600
Innocents within 2 hops: 15% = 999,000 * 0.15 = 149,850
So you have identified over 150K people (15% of your population), and you know that as many as 0.4% of them are bad actors. Not particularly helpful in the grand scheme of things.
Now try that with more realistic numbers. US Population: 300 million (with 80%+ phone ownership). Bad actor rate: 1 in 10,000? 1 in 100,000? 1 in a million?
Taking the "best" values (for finding bad actors) that gives something like[1] 36,000,000 innocents and 18,000 bad actors. Does that sounds like a good, precise, targeted technology to you?
[1] Taking 300 million population at 80% phone ownership versus 1 in 10,000 bad actors with 100% phone ownership
Umm...that should be "...the pool of innocents is far greater than the pool of bad actors..."
[...] Another link on the subject. [...]
Drop AT&T! Not a good deal.
@richard - If you are mining from scratch, the analysis is true and it is still a needle in a slightly smaller hay stack. However, if you have a known bad actor, this analysis can be used to check out friends and friends of friends. As you start getting additional hits (connections to other known bad actors), patterns begin to emerge.
The analysis presented here is what happens when people with a lot of knowledge and little experience try to understand a problem...
[...] AT&T research paper published in 2001 and unearthed today by Andrew Appel at Freedom to Tinker shows how the phone company uses Hancock-coded software to crunch through tens of millions of long [...]
Terrorists? We still falling for that elaborate ruse to empty the Treasury, lock down remainding natural resources and re-write the Constitution into toilet paper?
"Any two people on the planet are seperated by no more than six degrees."
That simply isn't true. It's true for most people -- but there are isolated pockets of people who are very far removed from civilization. There's at least one person who's known to be the last of his group and has never communicated with other humans -- so he's infinitely separated from every other human. There are also known to be tribes that have no contact with the outside world, particularly in South America but also in Indonesia and probably in Africa. There, everyone is one degree away from everyone else in their group, but infinitely separated from the rest of the world.
And I agree that terrorism is just the catch phrase used to rob us of our liberties. There was more than enough information before 9/11 to catch the terrorists -- our "intelligence" services and worse, our executive branch simply did nothing. Before we give up our rights and roll back the Constitution, why not try using competence first?
The 'terrorist threat' is the bogeyman being sold to lock-down the entire planet into a totalitarian information grid.
"Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety." - Benjamin Franklin, in "An Historical Review of the Constitution and Government of Pennsylvania." (1759)
"The purpose of government is to protect the secrecy and the privacy of all individuals, not the secrecy of government." - Ron Paul, from the California Republican Debate May 3, 2007,
Write AT&T a letter and drop your contract on privacy violation grounds. Inform yourself of the writings and positions of the Hon. Rep Dr. Ron Paul. Please donate generously to his campaign and get your fellow citizens to vote for him in the upcoming Republican primaries.
Very intelligent post Andrew. This is what keeps America great-the freedom to examine and challenge the efforts and policies of the agencies empowered to protect us. We must utilize technology to track bad actors. If that means our signal output is tracked, so be it. Just because we have the freedom to do nothing, doesn't mean we will remain free forever. There is a price to pay for freedom. The pursuits to keep us free, at times, carry a heavy burden. Though I respect the right of those that choose not to acknowledge the threat of terrorism, I trust in the American community at large to recognize this grave danger and step up and join the fight in Keeping America free. This wonderful technology to track known terrorists and their connections. We all should endeavor to make sure the power to do so, isn't abused and our liberties aren't trampled upon. Keep up the good work Andrew. BTW, do you have any alternative solutions?
Ok, here's the problem. The algorithm by itself is worthless. Once you have identified a bad actor, then it becomes useful. Once you have identified a bad actor, you can also get a legal warrant and all their call information. You can also get a warrant for anyone they called on the grounds of probable cause.
Which simply means there is no reason or excuse for breaking FISA, since the only useful information is also easily obtainable by following the law. The only thing that would be denied a warrant would be calls to or from people not connected to a bad actor/terrorist. Which also means the only reason to get that data on people not connected to bad actors/terrorists is because you want to do something illegal with it.
@carrie:
If by "recently" you mean "in the last several decades".
This is one where the infosec people are late to the party, not the social scientists.
Hancock: el lenguaje para vigilancia masiva...
De los laborotorios AT&T, llega esta maravilla: un lenguaje destinado al espionaje masivo de 'comunidades de interés'. Usado para relacionar llamadas a larga distancia, no es difÃcil imaginarse cómo lo pueden estar empleando los servicios de i...
[...] their boots over the next terrorist attack, the technology has found itself useful in other areas. Freedom to Tinker blog chimes in on the subject:Â According to government documents studied by The New York Times, [...]
"6 degrees of separation" means nothing in this technology. The links are not random. They are to a specific number: a suspected "bad actor".
1st order linkages are useful. 2nd and 3rd order links to "bad guys" are probably very good to know (and may not be covered by a legal warrant).
I think the technology is useful and one day might catch a potential attack. I'll trade off some privacy for a better economy and safety for my kids.
Those of us who would prefer to put our heads in the sand, will likely wake up one day to a very sad reality and an extremely weak economy.
Unless your bad actors are extraordinarily socially isolated (which they can take care not to be), the fact that many of their links to other bad actors are within one or two hops only does you good if you have enormous monitoring capabilities. Perhaps if the bad-actor network converged in some fashion, traffic analysis would get you somewhere, but even decent tradecraft is going to prevent that.
@mike c
I hope you didn't miss this quote mike c, it rings quite true, and I love slapping idiots in the face every-so-often with how far removed they are from the founding ideals of the country they claim to be so fond of (when more and more it seems they are fond of little more than simply pissing all over it):
“Those who would give up Essential Liberty to purchase a little Temporary Safety, deserve neither Liberty nor Safety.†- Benjamin Franklin, in “An Historical Review of the Constitution and Government of Pennsylvania.†(1759)
Brilliant quote, contemporary psuedo-poly-sci-majors would do well to take note.
Please quit voicing your ignorant support of the destruction of our freedom in the name of security, because neither you or the idiot congressman who represents you have any competence or insight into how to properly secure anything, let alone secure it in a manner that preserves our essential freedoms. The chances of being killed in a terrorist attack in this spoiled trust-fund-baby of a country is less than that of being struck by lightning. Neither scenario scares me in the least, and it shouldn't scare you either. What does scare me is the constant legislation of religious morality, the ever-increasing power shift from government to the mega-corporations, the ignorant masses who speak their recycled, spoon-fed ideas so loudly that their idiocy is all anyone can hear anymore, and the exponential downward spiral of our loss of essential civil liberties.
To all of you fools out there who suck at the nipple of this for-profit-fear-mongering nonsense: you need to have your heads examined, or be stripped of your right to vote since it seems you cannot think for yourselves.
The only thing this country truly needs is a fucking education, and maybe a diet. Wake up.
>> Those of us who would prefer to put our heads in the sand, will likely wake up one day to a very sad reality and an extremely weak economy.
We see the threat of terrorist attack with a clear eye and a calm heart.
While acknowledging that evil people can do horrible things, we choose not to live in fear and terror and allow little people with delusions of grandeur to control our hearts and minds.
We must first have liberty, for the state of the economy to have any meaning. The freedom to think is much more important than the freedom to buy.
Consider this: the term "traitor" was used as an epithet to describe people during the Revolutionary War. Treason remains the only crime defined in the Constitution as requiring the testimony of two witnesses to the same overt act, or confession in open court.
In the 21st century, we have allowed "terrorist" to take the place of the 18th century "traitor." Labeling people as terrorists based on network analysis is like labeling people traitors based on who their friends and family are.
We also define corruption of blood and ex post facto laws as unconstitutional. This form of guilt by association smacks of both.
Tracking actual honest-to-goodness terrorists? With all that taxpayer money, you'd better.
Denying people the opportunity to work and to travel on the basis of secret government lists? Not in my Constitutional republic you don't.
@Mike C: "Those of us who would prefer to put our heads in the sand, will likely wake up one day to a very sad reality and an extremely weak economy."
Nearly got coffee all over my keyboard. Today, the Canadian dollar moved to an all-time high relative to the U.S. dollar.
Some people think they have already woken up to a sad reality and an extremely weak economy, and it isn't because they haven't given up enough privacy.
[...] Hancock. Apparently it is able to handle the massive amount of date more easily and efficiently. Freedom to Tinker » Blog Archive » AT&T Explains Guilt by Association [...]
[...] technology has proven useful for spying on American citizen: Freedom to Tinker blog chimes in on the subject: According to government documents studied by The New York Times, the [...]
Since security is as fake as the Wizard of OZ and the freedoms we enjoy are being supplanted by "freedom from" freedoms why do typically liberal folks keep quoting highly religious people like Benjamin Franklin?
The rights a freedoms he envisioned were not predominately military in nature but more of the securities people seek through wealth and having a nanny state wipe, powder and diaper the collective behinds of those who have returned to nursing on the mother government!
Using the means at our hands to catch those whose purpose it is to bring us into servitude to some socialist ideology is not freedom.
Just because a law prermits or prohibits something does not mean it is also correct, right or the way it really is or should be. Just because abortion is legal does not mean the environment, God or history teaches us it is a correct course of action. Nature has miscarriages, preditors etc... these laws govern the nature of life. When man creates a law that counteracts that law we are forced to inherit the results of those laws.
It is legal to clear cut forests in places. It's legal for elementary schools to hand out condoms, but prohibited for opposite minded people to teach those same kids the mental, emotional and physical effects of succumbing to every urge that crosses the mind or body. Teaching people correct principles is a first step to people governing themselves. Teaching people that the government or some corporation will cover you for your stupidity is no way to ensure people even comprehend the law of action and reaction which is an imutable law. Though, there are many attempting to legislate consequences to be illegal.
Trying to legislate personal responsibility away from the individual does them no service. Liberals like to say we are a profit and corporate driven society yet expect those same "RICH" corporations to pony up millions to idiots who spill hot coffee on their crotches.
Until we expect humans to re-accept responsibility for their stupid actions we will keep looking at life as though someone else (i.e. the government) is responsible for my happiness, financial well being, education, and protection. Then those same people expecting this take care of me mentality say the government is taking their freedoms away though they desire to have a homogenized education and earning system.
Separation of School and State.
[...] AT&T research paper published in 2001 and unearthed today by Andrew Appel at Freedom to Tinker shows how the phone company uses Hancock-coded software to crunch through tens of millions of long [...]
Personal responsibility = good. Religious moralization and losing separation of church and state however = bad.
Funny you'd mention condoms, which are an important part of satisfying urges *responsibly*, taking measures to prevent undesired consequences. Condoms and sex education are a sign of a civilization trying to be mature and responsible. Dire religious fearmongering and preaching indicate Peter Pan syndrome: a tendency to want not to have to grow up and become responsible, to cling to fairy tales and a security blanket and try to preserve the childhood state of only having to do as father tells you and good old Dad will take care of everything for you.
This has recently been becoming a “popular†methodology in the social sciences, but is called Social Network Analysis.
Drop AT&T! Not a good deal.
There are some people think they have already woken up to a sad reality and an extremely weak economy, and it isn’t because they haven’t given up enough privacy.
Does any know the ATT, Verizon and Comcast share their FTTx network Infrastructure as a service to ISP or private content?
Architectural difference between the three vendors
How do they monitor the customer traffic? monitoring tool that is used and customer traffic attributes
solo cessione
i believe that this patriot act should be abolished because the FBI already has the authority to track terrorists if they have reasonable suspicion therefore making this act pointless.
[...] read more | digg story [...]
芳芳情趣用品
生活不該一成不變，今夜芳芳給您一點不一樣的吧!!
增進閨房情趣商品芳芳，讓倆人的親密關係更升溫
芳芳網路購物提供各國進口優質情趣用品,超商取貨及宅配到府,包裝隱密,請安心選購 ...
芳芳購物網站會不定期從國外進口情趣商品 貨色齊全 芳芳讓顧客有多樣化的選擇 ..
レーシックとは、視力回復するために行う手術方法で全国でもレーシックに対応している医院は少ない。いくつかの医院を比較して、安全で安心したレーシック手術を受けましょう。生命保険を検討されている方は一括見積もりをしてみましょう。ランキングによる比較をしてオススメ生命保険を格付け!あなたに合った生命保険プランが見つかります。医療保険の見直しは医療保険 ランキングを見て検討してみましょう。いろんな医療保険 比較ができるので自分に合った医療保険 制度を調べてご利用ください。医療保険 女性に適したプランも見つかります。キャッシングのご利用はキャッシング 比較サイトで調べてみよう。キャッシング 審査の内容やキャッシング 申込の方法も簡単に比較できます。即日キャッシングができる消費者金融もご紹介しております。レーシック 手術を受ける前に、レーシック 失敗の可能性について十分な知識をつけると共に、レーシック 保険に入ることをおすすめします。レーシック 東京の医院やレーシック 大阪の医院など、全国のレーシック対応の医院が見つかります。FX 比較サイトで情報収集できます。FX 初心者の方はまずバーチャル FXで流れを掴んでみましょう。FX ブログなどを見ると様々な人の外国為替証拠金取引の内容を理解できます。クレジットカードをランキングで徹底比較しています。審査がスピーディーで簡単。クレジットローンをご利用の方におすすめ。消費者金融をご利用の方はココ。消費者金融 審査がスピーディー。消費者金融 一覧で消費者金融 比較できますので、最適なキャッシング会社が消費者金融 ランキング形式で閲覧可能です。無料 オンラインゲームは暇つぶしに最適。オンラインゲーム ランキングを見て、おもしろい人気 オンラインゲームを探しましょう。オンラインゲーム レビューもたくさんあるので、無料ゲームをするには良い条件が揃っています。FX 初心者の人はFX 取引をする前に人気の証券会社をFX 比較してみましょう。FX ランキングで自分に一番合ったFX会社が見つかります。バーチャル FXでFX体験するのもいいかもしれません。FX 投資を考えているならFX 外為をオススメします。FX 口座開設には様々な特典もあります。FX ランキングを見ながら目的に合わせたFX 比較をしてみましょう。美容整形 口コミで美容整形 失敗に関する情報を知っておくことも必要です。美容外科は全国様々ありますが、美容整形 ランキングを見て一番人気のあるサロンを選びましょう。美容整形 芸能人に関する情報もあります。美容整形外科を全国検索できます。美容に関するポータルサイトです。美容整形は現在非常に安価で手軽に利用できます。オススメの美容液を特集中。美容 体重に関する悩み相談コーナーも実施しております。インプラント 費用はみなさんが思われる以上に高くありません。インプラント 治療は随分一般的になってきていますので、インプラント 料金の低下が進んでます。リーズナブルなインプラント 価格なので、歯科 インプラントに気軽に問い合わせください。インプラント 料金は以前はかなり高く、インプラント 費用がかかるために治療が進まない人もいましたが、現在はインプラント 価格が安くなってきていますので、インプラント 値段もお手頃になってきています。インプラント 初心者の方は是非いろいろと調べてみてください。SEOとは検索エンジンで上位表示化する技術のことでSEO対策や検索エンジン最適化SEMなどの検索エンジンマーケティングとは対照的な位置づけになっています。Yahoo! SEOやGoogle SEO、MSN SEOでより効果的なSEO対策は効果的外部リンクを集めることです。SEO一番効果のある方法です。Google SEOは相互リンク、Yahoo! SEOはドメイン。SEOは各検索エンジンによって異なり、SEO対策の方法も変わってきます。無料 SEOはお任せください。キャッシング 審査が早くてキャッシング 申込が簡単なキャッシング会社をキャッシング ランキングでキャッシング 比較してみましょう。キャッシング 初心者の方もどういう流れでキャッシングできるのかがわかりやすく理解できると思います。キャッシング 東京やキャッシング 大阪で借り入れできるところを簡単検索。キャッシング 低金利の消費者金融を探してみましょう。キャッシング 選び方が大事ですが、キャッシング 簡単にご利用できます。
rug
rugs
rug
rugs
Designer rugs
Counselling Aberdeen
Hypnotherapy Aberdeen
Hypnosis Aberdeen
flexible coupling
coupling
MCSE
ccna
stair lifts
stairlifts
bath lifts
bathlifts
wheelchair ramps
access ramps
disabled ramps
mobility scooters
wheelchairs
Laser Engraving Machine
Laser Welding Machine
3D Engraving
Laser Engraver
designer bag
louis vuitton replica
handbag wholesale
replica bag
handbag replica
cnc router
laser engraver
宿州之窗
[url=http://www.sz-window.com]宿州之窗[/url]
http://www.sz-window.com