April 24, 2014


Eavesdropping as a Telecom Profit Center

In 1980 AT&T was a powerful institution with a lucrative monopoly on transporting long-distance voice communications, but forbidden by law from permitting the government to eavesdrop without a warrant. Then in 1981 Judge Greene took its voice monopoly away, and in the 1980s and 90s the Internet ate the rest of its lunch. By 1996, Nicholas Negroponte wrote what many others also foresaw: “Shipping bits will be a crummy business. Transporting voice will be even worse. By 2020 … competition will render bandwidth a commodity of the worst kind, with no margins and no real basis for charging anything.

During the 1980s and 90s, AT&T cleverly got out of any business except shipping commodity bits: in 1981 it (was forced to) split off its regional phone companies; in 1996 it (voluntarily) split off its equipment-making arm as Lucent Technologies; in 2000-2001 it sold off its Wireless division to raise cash. Now AT&T long-distance bit-shipping is just a division of the former SBC, renamed AT&T.

What profit centers are left in shipping commodity bits? The United States Government spends 44 billion dollars a year on its spy agencies. It’s very plausible that the NSA is willing to pay $100 million or more for a phone/internet company to install a secret room where the NSA can spy on all the communications that pass through. A lawsuit by the EFF alleges such a room, and its existence was implicitly confirmed by the Director of National Intelligence in an interview with the El Paso Times. We know the NSA spends at least $200 million a year on information-technology outsourcing and some of this goes to phone companies such as Verizon.

Therefore, if it’s true that AT&T has such a secret room, then it may be simply that this is the only way AT&T knows how to make money off of shipping bits: it sells to the government all the information that passes through. Furthermore, economics tells us that in a commodity market, if one vendor is able to lower its price below cost, then other vendors must follow unless they also are able to make up the difference somehow. That is, there will be substantial economic pressure on all the other telecoms to accept the government’s money in exchange for access to everybody’s mail, Google searches, and phone calls.

In the end, it could be that the phone companies that cooperated with the NSA did so not for reasons of patriotism, or because their arms were twisted, but because the NSA came with a checkbook. Taking the NSA’s money may be the only remaining profit center in bit-shipping.


AT&T Explains Guilt by Association

According to government documents studied by The New York Times, the FBI asked several phone companies to analyze phone-call patterns of Americans using a technology called “communities of interest”. Verizon refused, saying that it didn’t have any such technology. AT&T, famously, did not refuse.

What is the “communities of interest” technology? It’s spelled out very clearly in a 2001 research paper from AT&T itself, entitled “Communities of Interest” (by C. Cortes, D. Pregibon, and C. Volinsky). They use high-tech data-mining algorithms to scan through the huge daily logs of every call made on the AT&T network; then they use sophisticated algorithms to analyze the connections between phone numbers: who is talking to whom? The paper literally uses the term “Guilt by Association” to describe what they’re looking for: what phone numbers are in contact with other numbers that are in contact with the bad guys?

When this research was done, back in the last century, the bad guys where people who wanted to rip off AT&T by making fraudulent credit-card calls. (Remember, back in the last century, intercontinental long-distance voice communication actually cost money!) But it’s easy to see how the FBI could use this to chase down anyone who talked to anyone who talked to a terrorist. Or even to a “terrorist.”

Here are a couple of representative diagrams from the paper:

Fig. 4. Guilt by association – what is the shortest path to a fraudulent node?

Fig. 5. A guilt by association plot. Circular nodes correspond to wireless service accounts while rectangular nodes are conventional land line accounts. Shaded nodes have been previously labeled as fraudulent by network security associates.


Comcast and Net Neutrality

The revelation that Comcast is degrading BitTorrent traffic has spawned many blog posts on how the Comcast incident bolsters the blogger’s position on net neutrality – whatever that position happens to be. Here is my contribution to the genre. Mine is different from all the others because … um … well … because my position on net neutrality is correct, that’s why.

Let’s start by looking at Comcast’s incentives. Besides being an ISP, Comcast is in the cable TV business. BitTorrent is an efficient way to deliver video content to large numbers of consumers – which makes BitTorrent a natural competitor to cable TV. BitTorrent isn’t a major rival yet, but it might plausibly develop into one. Which means that Comcast has an incentive to degrade BitTorrent’s performance and reliability, even when BitTorrent isn’t in any way straining Comcast’s network.

So why is Comcast degrading BitTorrent? Comcast won’t say. They won’t even admit what they’re doing, let alone offer a rationale for it, so we’re left to speculate. The technical details of Comcast’s blocking are only partially understood, but what we do know seems hard to square with claims that Comcast is using the most effective means to optimize some resource in their network.

Now pretend that you’re the net neutrality czar, with authority to punish ISPs for harmful interference with neutrality, and you have to decide whether to punish Comcast. You’re suspicious of Comcast, because you can see their incentive to bolster their cable-TV monopoly power, and because their actions don’t look like a good match for the legitimate network management goals that they claim motivate their behavior. But networks are complicated, and there are many things you don’t know about what’s happening inside Comcast’s network, so you can’t be sure they’re just trying to undermine BitTorrent. And of course it’s possible that they have mixed motives, needing to manage their network but choosing a method that had the extra bonus feature of hurting BitTorrent. You can ask them to justify their actions, but you can expect to get a lawyerly, self-serving answer, and to expend great effort separating truth from spin in that answer.

Are you confident that you, as net neutrality czar, would make the right decision? Are you confident that your successor as net neutrality czar, who would be chosen by the usual political process, would also make the right decision?

Even without a regulatory czar, wheels are turning to punish Comcast for what they’ve done. Customers are unhappy and are putting pressure on Comcast. If they deceived their customers, they’ll face lawsuits. We don’t know yet how things will come out, but it seems likely Comcast will regret their actions, and especially their lack of transparency.

All of which – surprise surprise – confirms my position on net neutrality: there is a risk of harmful behavior by ISPs, but writing and enforcing neutrality regulation is harder than it looks, and non-regulatory forces may constrain ISPs enough.


Comcast Blocks Some Traffic, Won't Explain Itself

Comcast’s apparent policy of blocking some BitTorrent traffic, which has been discussed on tech sites [example] for months, has now broken out into the mainstream press. Comcast is making things worse by refusing to talk plainly about what they are doing and why. (This is an improvement over Comcast’s previously reported denials, which now appear to be inconsistent with the facts.)

To the extent that Comcast has explained itself, its story seems to be that it is slowing traffic from heavy users in order to keep the network moving smoothly. This would be a reasonable thing for Comcast to do (if they were open about it) – but it’s not quite what they’re actually doing.

For starters, Comcast’s measures are not aimed at heavy users but rather at users of certain protocols such as BitTorrent. And not even all users of BitTorrent are targeted, but only those who use BitTorrent in a particular way: uploading a file to non-Comcast users while not simultaneously downloading parts of the same file. (In BitTorrent jargon, this is called “seeding”.) To get an idea of how odd this is, consider that an uploader who is experiencing blocking can apparently avoid the blocking by adding some download traffic.

It would likely be easier for Comcast to simply measure how much traffic each user is generating and drop the heaviest users’ packets, or just to discard packets at random (a tactic that falls most heavily on those who send and receive the most packets).

Beyond its choice of what to block, Comcast is using an unusual and nonstandard form of blocking.

There are well-established mechanisms for dealing with traffic congestion on the Internet. Networks are supposed to respond to congestion by dropping packets; endpoint computers notice that their packets are being dropped and respond by slowing their transmissions, thus relieving the congestion. The idea sounds simple, but getting the details right, so that the endpoints slow down just enough but not too much, and the network responds quickly to changes in traffic level but doesn’t overreact, required some very clever, subtle engineering.

What Comcast is doing instead is to cut off connections by sending forged TCP Reset packets to the endpoints. Reset packets are supposed to be used by one endpoint to tell the other endpoint that an unexplained, unrecoverable error has occurred and therefore communication cannot continue. Comcast’s equipment (apparently made by a company called Sandvine) seems to send both endpoints a Reset packet, purporting to come from the other endpoint, which causes both endpoints to break the connection. Doing this is a violation of the TCP protocol, which has at least two ill effects: it bypasses TCP’s well-engineered mechanisms for handling congestion, and it erodes the usefulness of Reset packets as true indicators of error.

People have apparently figured out already how to defeat this blocking, and presumably it won’t be long before BitTorrent clients incorporate anti-blocking measures.

It looks like Comcast is paying the price for trying to outsmart their customers.


The ease of applying for a home loan

I’m currently in the process of purchasing a new house. I called up a well-known national bank and said I wanted a mortgage. In the space of 30 minutes, I was pre-approved, had my rates locked in, and so forth. Pretty much the only identifying information I had to provide was the employer, salary, and social security number for myself and my wife, as well as some basic stats on our investment portfolio. Interestingly, the agent said that for people in my situation (sterling credit, paying more than 20% of the down payment out of our own pocket), they believe I’m highly unlikely to ever default on the loan. As a result, they do not need me to go the trouble of documenting my income or assets beyond what I told them over the phone. They’ll take my word for it.

(In an earlier post, I discussed my name and social security number having been stolen from where they had been kept in Ohio. Ohio gave me a free subscription to Debix, which claims to be able to intercept requests to read my credit report, calling my cell phone to ask for my permission. Why not? I signed up. Well, my cell phone never buzzed with any sort of call from Debix. Their service, whatever it does, had no effect here.)

Obviously, there’s a lot more to finalizing a loan and completing the purchase of a home than there is to getting approved for a loan and locking a rate. Nonetheless, it’s striking how little personal information I had to divulge to get this far into the game. Could somebody who knew my social security number use this mechanism to borrow money against my good credit and run away to a Carribean island with the proceeds? I would have to hope that there’s some kind of mechanism further down the pipeline to catch such fraud, but it’s not too hard to imagine ways to game this system, given what I’ve observed so far.

Needless to say, once this home purchase is complete, I’ll be freezing my credit report. Let’s just hope the freezing mechanism is more useful than Debix’s notification system.

(Sidebar: an $18 charge appeared on my credit card last month for a car rental agency that I’ve never used, claiming to have a “swipe” of my credit card. I challenged it, so now the anti-fraud division is allegedly attempting to recover the signed charge slip from the car rental agency. The mortgage agent, mentioned above, saw a note in my credit report on this and asked me if I had “challenged my bank”. I explained the circumstances and all was well. However, it’s interesting to note that the “challenge”, as it apparently appears in my credit report, doesn’t have any indication as to what’s being challenged or how significant it might be. Again, the agent basically took my word for it.)


Radiohead Album Available for Free, But Fileshared Anyway

The band Radiohead is trying an interesting experiment, offering its new album In Rainbows for download and letting each customer decide how much to pay. You can name a price of zero and download the album for free, if you want, or you can pay whatever price you think is fair.

Now Andy Greenberg at Forbes is reporting that despite Radiohead’s free-if-you-choose offer, many users are downloading the album from P2P systems rather than getting it from the band’s site. Some commentators find this surprising, but in fact it should have been predictable.

Why are some people getting In Rainbows from P2P rather than the band’s site? Probably because they find P2P easier to use.

Radiohead’s site makes you click and click to get the music. First you have to click through a nearly content-free splash screen. Then you click through another splash screen telling you things you probably already knew. Then you click an “ORDER” button, and click away a dialog box telling you something you already knew. Then after some headscratching, you realize you need to click the “VIEW BASKET” button, which takes you to a form asking you to name your price, in U.K. currency. (They link you to a third-party site, offering a large collection of currency-conversion tools – several more clicks to find the one you want.) After choosing your price, you click “PAY NOW”, at which point you get to stare at a “You are currently in a queue” screen for a while, after which you set up an daccount enter some personal information (including your email address and mobile phone number) and agree to some terms of service (which are benign, but it’s more time and more clicks to verify that). Finally, you get to download the music.

It’s easy to see why somebody might prefer a P2P download. Leaving aside legal issues – and let’s face it, many people do – the moral argument against unauthorized P2P downloading seems pretty weak in this case, where downloaders aren’t depriving the band (or anyone else) of revenue.

This is an interesting natural experiment that tells us something about why people use P2P. If people normally choose P2P over authorized channels because P2P is cheaper, we would expect customers to shift toward the authorized channel when it offers a zero price. But if people choose P2P for convenience, then we’d expect a shift toward more P2P use for this album, because people have fewer moral qualms about P2P downloading this album than they would for a normal album. The clunkiness of Radiohead’s site improves the experiment by sharpening the ease-of-use factor.

It’s too early to tell how the experiment will come out, but news reports so far indicate that the ease-of-use factor is probably more important than some pundits think. This is yet more evidence that had the record industry embraced easy-to-use Internet music technologies early on, things would be very different now.

[UPDATE (Oct 21, 2007): Bill Zeller documents how technical issues completely prevent a large number of users from legally downloading In Rainbows from Radiohead's site.]


Grokster Case Lumbers On; Judge To Issue Permanent Injunction

Remember the Grokster case? In which the Supreme Court found the filesharing companies Grokster and StreamCast liable for indirect copyright infringement, for “inducing” infringement by their users? You might have thought that case ended back in 2005. But it’s still going on, and the original judge just issued an interesting ruling. (Jason Schultz has a two part summary of the ruling.)

The issue now before the judge is what relief to grant the copyright-owner plaintiffs against StreamCast, which is the only defendant still standing. It’s apparently a given that the judge will eventually assess monetary damages against StreamCast. And you’d think these damages would be enough to kill StreamCast, so it’s not clear why StreamCast hasn’t just thrown in the towel, shut its doors, and handed over all its assets to the plaintiffs. Instead, StreamCast fought on, so the judge had to decide what kind of injunction, if any, to impose on StreamCast – that is, what rules would govern StreamCast’s future behavior.

The judge first considered the question of whether he could impose on StreamCast obligations (beyond payment of damages) that go beyond what the law requires of ordinary companies. Would he just award money damages and sternly command StreamCast not to break the law again; or would he go further and impose a permanent injunction? After a detailed legal analysis, he concluded that a permanent injunction was appropriate. StreamCast had actively promoted itself as a haven for infringement and “that bell cannot be unrung”.

The copyright-owner plaintiffs had asked for an injunction requiring StreamCast to apply all feasible anti-infringement technologies and to stop all infringment. StreamCast had built its own filtering technology which it said was effective enough, and much cheaper and more practical than commercially available alternatives.

The judge first rejected the plaintiff’s proposal that StreamCast be required to stop all infringement using its software. He recognized, correctly, that that would be impossible, so that such an injunction would be a death sentence for StreamCast.

Instead, the judge will require StreamCast to set up a filtering system that reasonably balances effectiveness and cost, with the strong emphasis on effectiveness. The precise details will be worked out with the help of a special master: an independent technical expert to be appointed by the judge. Which means yet more legal process to choose the special master, wait for the special master’s advice, and then order specific action from StreamCast.

All of this may be proper from a legal standpoint, but it seems unlikely to matter in practice. It’s hard to see how StreamCast can sustain a business given the legal and financial strain they must be under, and the likely ruinous monetary damages they’re still facing. I can understand why the plaintiffs might want to keep StreamCast on life support, in the hope of getting legal rulings that prove helpful elsewhere. But why does StreamCast keep fighting?


Online Symposium: Future of Scholarly Communication

Today we’re kicking off an online symposium on The Future of Scholarly Communication, run by the Center for Information Technology Policy at Princeton. An “online symposium” is a kind of short-term group blog, focusing on a specific topic. Panelists (besides me) include Ira Fuchs, Paul DiMaggio, Peter Suber, Stan Katz, and David Robinson. (See the symposium site for more information on the panelists.)

I started the symposium with an “introductory post. Peter Suber has already chimed in, and we’re looking forward to contributions from the other panelists.

We’ll be running more online symposia on various topics in the future, so this might be a good time to bookmark the symposium site, or subscribe to its RSS feed.


attack of the context-sensitive blog spam?

I love spammers, really I do. Some of you may recall my earlier post here about freezing your credit report. In the past week, I’ve deleted two comments that were clearly spam and that made it through Freedom to Tinker’s Akismet filter. Both had generic, modestly complementary language and a link to some kind of credit card application processing site. What’s interesting about this? One of two things.

  1. Akismet is letting those spams through because their content is “related” to the post.
  2. Or more ominously, the spammer in question is trolling the blogosphere for “relevant” threads and is then inserting “relevant” comment spam.

If it’s the former, then one can certainly imagine that Akismet and other such filters will eventually improve to the point where the problem goes away (i.e., even if it’s “relevant” to a thread here, if it’s posted widely then it must be spam). If it’s the latter, then we’re in trouble. How is an automated spam catcher going to detect “relevant” spam that’s (statistically) on-topic with the discussion where it’s posted and is never posted anywhere else?


Infinite Storage for Music

Last week I spoke on a panel called “The Paradise of Infinite Storage”, at the “Pop [Music] and Policy” conference at McGill University in Montreal. The panel’s title referred to an interesting fact: sometime in the next decade, we’ll see a $100 device that fits in your pocket and holds all of the music ever recorded by humanity.

This is a simple consequence of Moore’s Law which, in one of its variants, holds that the amount of data storage available at a fixed size and price roughly doubles every eighteen months. Extrapolate that trend and, depending on your precise assumptions, you’ll find the magic date falls somewhere between 2011 and 2019. From then on, storage capacity might as well be infinite, at least as far as music is concerned.

This has at least two important consequences. First, it strains even further the economics of the traditional music business. The gap between the number of songs you might want to listen to, and the number you’re willing and able to pay a dollar each to buy, is growing ever wider. In a world of infinite storage you’ll be able to keep around a huge amount of music that is potentially interesting but not worth a dollar (or even a dime) to you yet. So why not pay a flat fee to buy access to everything?

Second, infinite storage will enable new ways of building filesharing technologies, which will be much harder for copyright owners to fight. For example, today’s filesharing systems typically have users search for a desired song by contacting strangers who might have the song, or who might have information about where the song can be found. Copyright owners’ technical attacks against filesharing often target this search feature, trying to disrupt it or to exploit the fact that it involves communication with strangers.

But in a world of infinite storage, no searching is needed, and filesharers need only communicate with their friends. If a user has a new song, it will be passed on immediately to his friends, who will pass it on to their friends, and so on. Songs will “flood” through the population this way, reaching all of the P2P system’s participants within a few hours – with no search, and no communication with strangers. Copyright owners will be hard pressed to fight such a system.

Just as today, many people will refuse to use such technologies. But pressure on today’s copyright-based business models will continue to intensify. Will we see new legal structures? New business models? Or new public attitudes? Something has to change.