Comcast’s apparent policy of blocking some BitTorrent traffic, which has been discussed on tech sites [example] for months, has now broken out into the mainstream press. Comcast is making things worse by refusing to talk plainly about what they are doing and why. (This is an improvement over Comcast’s previously reported denials, which now appear to be inconsistent with the facts.)
To the extent that Comcast has explained itself, its story seems to be that it is slowing traffic from heavy users in order to keep the network moving smoothly. This would be a reasonable thing for Comcast to do (if they were open about it) – but it’s not quite what they’re actually doing.
For starters, Comcast’s measures are not aimed at heavy users but rather at users of certain protocols such as BitTorrent. And not even all users of BitTorrent are targeted, but only those who use BitTorrent in a particular way: uploading a file to non-Comcast users while not simultaneously downloading parts of the same file. (In BitTorrent jargon, this is called “seeding”.) To get an idea of how odd this is, consider that an uploader who is experiencing blocking can apparently avoid the blocking by adding some download traffic.
It would likely be easier for Comcast to simply measure how much traffic each user is generating and drop the heaviest users’ packets, or just to discard packets at random (a tactic that falls most heavily on those who send and receive the most packets).
Beyond its choice of what to block, Comcast is using an unusual and nonstandard form of blocking.
There are well-established mechanisms for dealing with traffic congestion on the Internet. Networks are supposed to respond to congestion by dropping packets; endpoint computers notice that their packets are being dropped and respond by slowing their transmissions, thus relieving the congestion. The idea sounds simple, but getting the details right, so that the endpoints slow down just enough but not too much, and the network responds quickly to changes in traffic level but doesn’t overreact, required some very clever, subtle engineering.
What Comcast is doing instead is to cut off connections by sending forged TCP Reset packets to the endpoints. Reset packets are supposed to be used by one endpoint to tell the other endpoint that an unexplained, unrecoverable error has occurred and therefore communication cannot continue. Comcast’s equipment (apparently made by a company called Sandvine) seems to send both endpoints a Reset packet, purporting to come from the other endpoint, which causes both endpoints to break the connection. Doing this is a violation of the TCP protocol, which has at least two ill effects: it bypasses TCP’s well-engineered mechanisms for handling congestion, and it erodes the usefulness of Reset packets as true indicators of error.
People have apparently figured out already how to defeat this blocking, and presumably it won’t be long before BitTorrent clients incorporate anti-blocking measures.
It looks like Comcast is paying the price for trying to outsmart their customers.
All I know is that I *pay for* a certain speed, I *want* that speed. It’s a fookin contract.
If *they* can’t keep things moving at the speed they *advertise*, then it is false advertisement. I think it is just about time for a lawsuit.
I see an awful of of UDP traffic with Wireshark when BitTorrent is running, apparently I mistook it for file transfers. In the document Andrew links, there is an admission that BT is designed to defeat Slow Start:
Choking is done for several reasons. TCP congestion control behaves very poorly when sending over many connections at once. Also, choking lets each peer use a tit-for-tat-ish algorithm to ensure that they get a consistent download rate.
There are several criteria a good choking algorithm should meet. It should cap the number of simultaneous uploads for good TCP performance. It should avoid choking and unchoking quickly, known as ‘fibrillation’. It should reciprocate to peers who let it download. Finally, it should try out unused connections once in a while to find out if they might be better than the currently used ones, known as optimistic unchoking.
The TCP Slow Start algorithm was designed to prevent ftp from de-stablilizing the Internet, and the easiest way to defeat it is my opening multiple TCP connections and switching from those being held down by Slow Start to those that aren’t. And it doesn’t handle bandwidth hogs in any case.
So the fact remains that ftp and BitTorrent present different challenges to traffic engineering, and hence mandate different solutions. BT isn’t a delay-sensitive real-time app, so it’s perfectly fine for the network to defer its transactions until it has the bandwidth to satisfy them.
@Richard Bennett: What Jon said — BitTorrent does not transfer files with UDP. You might benefit from a quick review of the BitTorrent protocol specification, which can be found here: http://www.bittorrent.org/protocol.html
That’s news to me. The only instances I know of that Bittorrent uses UDP is for DHT and ‘UDP trackers’, both of which only obtain meta-information about peers. Neither is used to actually transfer the file. Neither is designed to escape traffic shaping. Neither uses significant bandwidth (UDP trackers are in fact designed to reduce bandwidth usage).
I would say that that is a pretty non-standard interpretation of the words ‘block’ and ‘delay’.
Ed says: “Neither objection seems on point. (A) seems beside the point, given that we’re talking about how to throttle TCP traffic, and we’re comparing to a mechanism (forged TCP Resets) which also applies only to TCP. (B) also seems to miss the point, because Comcast seems to be targeting BitTorrent seeders, rather than directly targeting heavy bandwidth users.”
BitTorrent sets up sessions with TCP, and then carries them out with either UDP or TCP as appropriate to escape traffic shaping. So the TCP reset is efficient at preventing an uncontrollable UDP session from starting. UDP, as you know, doesn’t play nice with Slow Start.
TCP RSTs are a feature found in every firewall, including the one your university uses for many of the same reasons that Comcast limits traffic from heavy users of the upstream channel, all of whom happen to be using BitTorrent.
If you read the latest from the AP advocate/reporter who broke the story, the Comcast metering is traffic-sensitive and doesn’t block connections permanently; it simply delays then until traffic can support them without degrading other users.
It’s a much better scheme than Slow Start because it operates on a more rational timeline and doesn’t depend on bursty loads from each user.
Further, it cannot be ignored that the distribution of linux and FOSS to dismantle the centralized power structures of large corporations is itself an act with a political dimension. So, any attempt to dismantle or disrupt Bit torrent traffic is a de facto act of political repression.
Thanks Andrew! That makes sense now.
@Brent Nordquist: There is no contradiction here. The point is that the Sandvine heuristic appears to be trying to block BitTorrent seeding (which it detects by looking for high upload rates without corresponding high download rates), but not BitTorrent downloading (which it detects by looking for simultaneous high upload AND download rates).
The Torrentfreak article you cite suggests trying to maintain simultaneous download and upload activity for the entire duration for which you are participating in the torrent, so as to make the Sandvine heuristic believe that you are downloading the whole time and not seeding. The best way to do this and still reach an upload/download “ratio goal” of 1.0 is to limit the download speed to the upload speed.
Above you say, “consider that an uploader who is experiencing blocking can apparently avoid the blocking by adding some download traffic”. I’m curious about this point–do you have a reference, or can you clarify how that works?
The torrentfreak article you link to seems to say just the opposite, under “What is working”: “When downloading, make sure that you have met your uploading goal by the time that the download completes. The easiest way to accomplish this is to set a download rate slower than the uploading rate.”
Richard writes, “The TCP traffic mechanisms aren’t much help here because they: A) Only regulate single streams of TCP traffic (not UDP); and B) Don’t directly target bandwidth hogs.”
Neither objection seems on point. (A) seems beside the point, given that we’re talking about how to throttle TCP traffic, and we’re comparing to a mechanism (forged TCP Resets) which also applies only to TCP. (B) also seems to miss the point, because Comcast seems to be targeting BitTorrent seeders, rather than directly targeting heavy bandwidth users.
I realize DOCSIS can congest before the TCP network is not fully utilized. But it’s a long way from there to the conclusion that forging Reset packets is a better way to cope with the congestion.
Note that dropping packets at random does not ‘[fall] most heavily on those who send and receive the most packets”–it affects _everybody_. If you drop 1% of packets, that’s 1% of everybody’s packets, not just those of the heaviest users.
Also, aren’t forged TCP resets the same mechanism used by the Great Firewall of China? So here we have the Great Firewall of Comcast…
I think you’re missing the point, Ed. You say: “There are well-established mechanisms for dealing with traffic congestion on the Internet” which is indeed correct, but beside the point. The mechanisms you mention are meant to prevent the Internet as a whole from becoming unstable, and they’re more or less adequate for the purpose, minus some cycling problems; see the research on RED for examples of their shortcomings. What Comcast is dealing with however, is something similar in that they’re trying to avoid collision collapse, but more localized as they’re concerned with their own residential network, not the Internet as a whole.
So it’s more relevant to look at the weaknesses of the DOCSIS protocol inside Comcast’s network and how one goes about preventing it from de-stabilizing from high upload requests. The TCP traffic mechanisms aren’t much help here because they: A) Only regulate single streams of TCP traffic (not UDP); and B) Don’t directly target bandwidth hogs. BitTorrent is designed to achieve target rates of upload and download, and its willing to open as many TCP or UDP sessions as it needs in order to achieve its goal, which is steady, sustained transfers. So all the TCP traffic management in the world is not effective against an application that’s willing to create as many TCP streams as it needs to achieve a pre-determined bandwidth saturation goal.
If the discussion comes down to the question of which throttling mechanisms are legitimate and which aren’t, the average citizen has to tune out and leave the discussion to the professionals, but that’s frustrating for the net neutrality dogmatists who depend on controversy to make a living.
But I do agree that Comcast hasn’t represented itself well in this debate, and suspect their general cluelessness is going to cost them in the long run; not that it should, but life not being fair, it probably will.
I wish people would stop mentioning these ineffective ways of circumventing the Comcast blocking; none of them work.
Dropping TCP resets is a bad idea (there are innocent reasons TCP resets are sent) and is useless unless both endpoints do it (extremely unlikely).
Protocol encryption does not work because Sandvine is using heuristic analysis of network behavior, like number of open connections. (These heuristics are apparently triggering false positives on things like Lotus notes). Trying to foil the heuristics is probably self-defeating.
Tunnelling does work, but you are essentially downloading from another computer so you need access to a box outside Comcast with a high-throughput near-symmetrical connection. This is effectively a suggestion of “don’t use Comcast for Bittorrent” and doesn’t seem like much of a bypass to me.
Ed,
I’d like to read a knowledgable comparison between midpoint-initiated resets and ICMP source quench as congestion control mechanisms.
Obviously, there are gross differences. But, at this point, I think pretty much everone believes “source quench considered harmful”, without really thinking through the reasons why that is.
Sending packets that falsely claim to come from a network user’s computer should be a crime, if it isn’t already one.
“It would likely be easier for Comcast to simply measure …”
Sigh … I have to repeat to myself:
My stance on “Net Neutrality” – I’m staying OUT
Also of note: whatever heuristic the Sandvine system uses to determine which TCP connections to terminate appears to be interfering with non-P2P traffic, notably Lotus Notes and SSH, as well.
http://kkanarski.blogspot.com/2007/09/comcast-filtering-lotus-notes-update.html
http://slashdot.org/comments.pl?sid=333521&threshold=4&commentsort=0&mode=thread&cid=21046205
@ignorance: yes, but at least you used to be able to count on the fact that an ISP would not send packets pretending to be you (i.e. using your IP address as the packet’s source IP address).
zoinks! so networks are no longer passive, or maybe never were? that’s an eye opener and no mistake.