Today I want to continue last week’s discussion of how network discrimination might actually occur. Specifically, I want to talk about packet reordering.
Recall that an Internet router is a device that receives packets of data on some number of incoming links, decides on which outgoing link each packet should be forwarded, and sends packets on the appropriate outgoing links. If, when a packet arrives, the appropriate outgoing link is busy, the packet is buffered (i.e., stored in the router’s memory) until the outgoing link is available.
When an outgoing link becomes available, there may be several buffered packets that are waiting to be transmitted on that link. You might expect the router to send the packet that has been waiting the longest – a first-come, first-served (FCFS) rule. Often that is what happens. But the Internet Protocol doesn’t require routers to forward packets in any particular order. In principle a router can choose any packet it likes to forward next.
This suggests an obvious mechanism for discriminating between two categories of traffic: a network provider can program its routers to always forward high-priority packets before low-priority packets. Low-priority packets feel this discrimination as an extra delay in passing through the network.
Recall that last week, when the topic was discrimination by packet-dropping, I distinguished between minimal dropping, which drops low-priority packets first but only drops a packet when necessary, and non-minimal dropping, which intentionally drops some low-priority packets even when it is possible to avoid dropping anything. The same kind of distinction applies to today’s discussion of discrimination by delay. A minimal form of delay discrimination only delays low-priority packets when it is necessary to delay some packet – for example when multiple packets are waiting for a link that can only transmit one packet at a time. There is also a non-minimal form of delay discrimination, which may delay a low-priority packet even when the link it needs is available. As before, a net neutrality rule might want to treat minimal and non-minimal delay discrimination differently.
One interesting consequence of minimal delay discrimination is that it hurts some applications more than others. Internet traffic is usually bursty, with periods of relatively low activity punctuated by occasional bursts of packets. If you’re browsing the Web, for example, you generate little or no traffic while you’re reading a page, but there is a burst of traffic when your browser needs to fetch a new page.
If a network provider is using minimal delay discrimination, and the high-priority traffic is bursty, then low-priority traffic will usually sail through the network with little delay, but will experience noticeable delay whenever there is a burst of high-priority traffic. The technical term for this kind of on-again, off-again delay is “jitter”.
Some applications can handle jitter with no problem. If you’re downloading a big file, you care more about the average packet arrival rate than about when any particular packet arrives. If you’re browsing the web, modest jitter will cause, at worst, a slight delay in downloading some pages. If you’re watching a streaming video, your player will buffer the stream so jitter won’t bother you much.
But applications like voice conferencing or Internet telephony, which rely on steady streaming of interactive, realtime communication, can suffer a lot if there is jitter. Users report that VoIP services like Vonage and Skype can behave poorly when subjected to network jitter.
And we know that residential ISPs are often phone companies or offer home phone service, so they may have a special incentive to discriminate against competing Internet phone services. Causing jitter for such services, whether by minimal or non-minimal delay discrimination, could be an effective tactic for an ISP that wants to drive customers away from independent Internet telephone services.
There is some anecdotal evidence to suggest that Comcast’s residential Internet customers may be having trouble using the Vonage Internet phone service because of jitter problems.
Let’s assume for the sake of argument that these reports are accurate – that Comcast’s network has high jitter, and that this is causing problems for Vonage users. What might be causing this? One possibility is that Comcast is using delay discrimination, either minimal or non-minimal, with the goal of causing this problem. Many people would want rules against this kind of behavior.
(To be clear: I’m not accusing Comcast of anything. I’m just saying that if we assume that Comcast’s network causes high jitter, and if we assume that high jitter does cause Vonage problems, then we should consider the possibility that Comcast is trying to cause the jitter.)
Another possibility is that Comcast isn’t trying to cause problems for Vonage users, and Comcast’s management of its network is completely reasonable and nondiscriminatory, but for reasons beyond Comcast’s control its network happens to have higher jitter than other networks have. Perhaps the jitter problems are temporary. In this case, most people would agree that net neutrality rules shouldn’t punish Comcast for something that isn’t really its fault.
This most challenging possibility, from a policy standpoint, (still assuming that the jitter problem exists) is that Comcast didn’t take any obvious steps to cause the problem but is happy that it exists, and is subtly managing its network in a way that fosters jitter. Network management is complicated, and many management decisions could impact jitter one way or the other. A network provider who wants to cause high jitter can do so, and might have pretextual excuses for all of the steps it takes. Can regulators tell this kind of strategem apart from fair and justified engineering decisions that happen to cause a little temporary jitter?
Surely some discriminatory strategies are so obvious, and the offered engineering pretexts so weak, that we could block or punish them without worrying about being wrong. But there would be hard cases too. Net neutrality regulation, even if justified, will inevitably lead to some difficult line-drawing.
Robin Battey writes: “Something that I haven’t seen yet in all these talks about “net neutrality†is how the upcoming IPv6+IPSec technologies will make it almost irrelevant.”
I wouldn’t hold much hope for that. Have you tried to get IPv6 dialtone from a major ISP in North America lately? I have. Furthermore, I think IPv6+IPsec is likely to help major ISP’s engineer traffic for non-neutral discrimination.
One of the interesting ways the “net neutrality” dispute could end up evolving is by non-minimal discrimination against IPv6-in-IPv4 packets. Incidentally, the “flow label” field in the IPv6 header is another avenue of [potentially non-neutral] network discrimination.
Something that I haven’t seen yet in all these talks about “net neutrality” is how the upcoming IPv6+IPSec technologies will make it almost irrelevant. At a point “Really Soon Now” (somewhere less than 10 years from now, I’d guess), nearly all traffic will be encrypted using some combination of IPSec or SSL-like technology (private, authenticated network streams). With the advent of IPv6 routing headers, or anonymous routing protocols like TOR (The Onion Router), even the final destination may be hidden from the intervening links, including the ISP’s. When no one can tell what service the packets are destined for, there will be no way of doing service-type discrimination.
You could still discriminate based upon traffic patterns, of course — like keeping track of how much data has come from which hosts for the past certain amount of time, or a smart traffic-pattern-matching algorithm yet to be developed. Right now, that would take more compute power (per packet) than the high-end routers are capable of. It would also require developing a new matching algorithm for every new P2P system that came out, which turns into a race. When P2P designers start mimicking “legitimate” high-bandwidth services like (paid) streaming radio, and the ISP’s can no longer differentiate the “good” users from the “bad” ones, the system would fall apart.
Of course, there isn’t really anything anyone (or anything short of world-wide cooperation, for that matter) can do to stop it from happening; it can only be slowed down. Secured streams will become more and more common, and unencrypted network traffic will go the way of telnet, rsh, and plaintext passwords. To make things go even more quickly, so far as adoption is concerned, I hear rumors (and read articles) saying that Microsoft is shipping Vista with IPSec authentication enabled by default — in a “best effort” fashion — on all local network traffic. It’s not encrypted, and I don’t expect the world will convert overnight, but it’s still a huge jump-start.
Additionally, if ISP’s start trying to discriminate based upon type of service, it’s just one more incentive for bandwidth-using companies to set up secure, anonymous streams. Any sort of discrimination based on anything other then pure bandwidth usage is certain to be short-lived. I don’t see what all this hoopla is about.
Just a quick comment on the question asked by “V” and some of the answers that reference the new ECN bit and the ToS/QoS fields in the IP packet header:
First about the ToS/QoS fields – providers tend to look on those bits with suspicion. They reasonaby ask why, if those bits are to be honored, why any person would not mark all of their packets as requiring the highest levels of processing. Consequently use of these bits tends to raise a lot of technical concern about admission control, “policing”, and policy. And, to make things worse, as packets are handed from provider to provider, there is a reluctance for the downstream provider to honor the packet markings received from the upstream provider.
It has been my sense that the next bit area of internet governance will concern the question of how those who want to run transmission-quality sensitive applications will be able to obtain adequate and fully end-to-end assurances (assurances, not guarantees).
As for the ECN flag – it is new and I have not seen in deployed very widely, much less have I seen TCP engines reacting to ECN flags. But TCP flows, at least when using most well engineered TCP engines (including Microsoft’s), tend to behave with proper network etiquitte when they perceive loss or changes in round-trip time. That does not mean that there aren’t TCP engines that try to be bullies, just as there used to be Ethernet chipsets that tended to grab the wire too quickly in order to dominate slower players. But the problems with net congestion generally aren’t from TCP streams, rather they are from high volume or time sensitive UDP streams, such as RTP/RTCP based voice or video.
Regarding the question about recognizing which traffic is time sensitive: As I mentioned above, the ToS/QoS bits aren’t really all that useful because they are often overwritten or simply erased by admission control and policing engines in edge and inter-provider routers.
That tends to leave us with UDP port numbers as differentiators. That is usually an adequate mechanism but for RTP/RTCP based VOIP (which is pretty much all non-proprietary VOIP) there are no well known UDP port numbers involved – rather the port numbers are established via the SIP (or equivalent protocols using a sub-protocol called “SDP”) It substantially complicates life for a normally stateless router to track SIP conversations, assuming that the router can even see those conversations, in order to obtain UDP port information. The situation is not good in this regard.
There are other signatures that are being used by providers to detect VOIP flows (often in order to impose predatory impairments). One such mechanism looks for sequences of smallish (~256 byte or less) UDP packets between a four-tuple of IP addresses and ports that are metronomed at roughly 50 millisecond intervals.
I’m doing a lot of work these days with traffic shaping have found that, when it is under the user’s control, it can actually improve the way the net “feels” to the user both for time-insensitive traffic (e.g. HTTP), time sensitive traffic (e.g. NTP and VOIP), and intermediate traffic (e.g. SSH or telnet.) It all comes down to the question of who is in control and who gets to decide. My feeling is that the user’s, not the providers, choices must be the determinative factor. The question that remains in my mind is how we pay for the costs, if any (there may be no costs).
Parijat,
What market forces, exactly? Here is what I see: Cable Internet providers are not required by law to allow someone else to offer their service, so they do not. DSL companies are only required by law to open their central offices (CO’s) to competing providers, but have no legal requirement to open up their remote terminals in the same way. So guess what? They don’t. This means that if you live somewhere where you are too far from the CO to get DSL, but the phone company has put in a remote terminal, then your ONLY broadband choices are all companies that offer telephone service. There are no market forces here. Both the cable Internet and the DSL provider have the same incentive: block other VOIP providers so people will pay for theirs. When you have exactly two choices like this, experience shows that there is no real competition.
The problem here is that these companies may be able to tie one service (phone service) to another serivce (Internet) for which they are they only high speed provider.
One solution is structural: Break up Internet service from phone service so the same company CANNOT offer both. Then these market forces you speak of will have a chance to function.
I don’t see what is wrong with a service provider discrimating against certain kinds of traffic. Specifically, if the service provider wants to charge higher rates for say VoIP traffic, there is nothing wrong with it per se. Either governmental regulation can keep the prices down or market forces can be allowed to play with these prices. As far as technological possibility is concerned, QoS solutions in place today can fairly easily allow separate billing and monitoring for different types of traffic, viz. ordinary data traffic (file transfers) as opposed to delay/jitter-sensitive data (VoIP, videoconf, etc).
It’s not in Comcast’s interest to have high jitter across the board. It is in their interest to have high jitter for Vonage VOIP calls and low jitter for Comcast VOIP calls. More generally, it’s only in an ISP’s interest to discrimate against traffic for which they offer a competing service. To the extent that similar services are implemented with similar traffic patterns such discrimination might be tricky to do in a way that looks accidental.
The appropriate way to handle congestion for ALL other cases and traffic types is to drop packets […]
Dave,
RFC 3168 “The Addition of Explicit Congestion Notification (ECN) to IP” is a proposed standard.
Any consumer broadband solution is going to have some sort of per-user limits. Many DSL providers have offered tiered pricing, but most cable companies are stuck in a single gear. Usually, however, there are additional unstated limits to the service. Here’s what I’d like to see:
1) The carriers should open up about the hidden customer limits for each tier of service they offer. For example, you might get 6000/384kbps service, but they might silently apply the brakes using traffic shaping if you are consistently using more than half that limit for more than 15 minutes. (Yes, your max download is 6000kbps but watch what happens if you try to download 15 terabits in a month.)
2) Traffic shaping or limiting should only be applied to a customer as a consequence of the policies in #1, not as some network-wide policy to discriminate against particular traffic types.
3) The appropriate way to handle congestion for ALL other cases and traffic types is to drop packets, which will cause the sender to throttle back. If this is happening on a regular basis the carrier most likely needs to add capacity or adjust their customer limits in #1.
With these policies in place it’s up to the customer’s network equipment to prioritize the use of their own broadband connection. A “dumb” router will give equal weight to all traffic, which will make it hard to hold a VoIP conversation if you’re also doing a bunch of P2P downloads at the same time to the point that you’re running up against the carrier’s bandwidth limiting policy. OTOH a smart router will prioritize the VoIP traffic and everything will be fine.
We don’t need the carrier to get involved in this prioritization. They should just carry the bits and have clear rules about how those bits are carried for the customer.
Let’s say that an ISP wants to discriminate against a particular third party application. Rather than admitting it, however, the ISP simply chooses a traffic shaping technology that incidentally *happens* to seriously degrade the application in question. The ISP claims, as a pretext, that traffic shaping is necessary in order to reduce infringing P2P file sharing.
Could government regulators effectively regulate this kind of pretextual network manipulation? I’d like to hear from people familiar with the FCC’s experience trying to regulate Ma Bell in the 70s and the RBOCs in the 90s.
We may all agree that network neutrality is a good thing, and still not know whether FCC regulations will, in the end, achieve it (and even if it does, what the collateral costs might be).
Would it be technically possible for the ISP to distinguish between packets […] ?
V,
The standard for IPv4, RFC 791 (STD 5), defines a “Type of Service” (ToS) field in the IPv4 header (p.12).
RFC 2474 (Proposed Standard) attempts to replace that header field with a “Differentiated Services” (DS) field (p.7). This proposed DS field definition is “incompatible” with the TOS field definition (p.8).
So, the simple answer to your question is “yes.”
One other thing that comes immediately to my mind is that with cable internet all users on the same cable segment use the same physical pipe to the ISP. With DSL the pipe sharing starts in the ISP-side aggregator. Having a few heavy-duty P2P users on the block could make a difference. But then that should affect all traffic.
Regarding phony pretexts – a recent WSJ article quoted a phone/DSL company executive saying that they are considering blocking third-party VOIP traffic “because it consumes so much bandwidth.” Hogwash – the real bandwidth killers are heavy-duty P2P systems like BitTorrent. Obviously this is just an excuse to impede competition against their own VOIP products.
HTTP and VOIP both have very different needs in terms of delay, pulse, etc etc etc. Would it be technically possible for the ISP to distinguish between packets from different protocols and optimize treatment accordingly?
I wonder if this will truly be an area where malice turns to be indistinguishable from incompetence…
Minimizing slack bandwidth will tend to increase jitter, which means that companies who upgrade their networks only gradually, or who occasionally screw up their routing tables, will reap some benefits from the resulting discrimination. Bit insofar as jitter also depends (to some extent) on the kind and source of other traffic on (segments of) the network this raises the question of whether an ISP can manage its customer base and its content-provider partners to make its network more or less friendly to applications such as VOIP with which it wants to compete.
(Just by the way, for managed VOIP, it’s impossible to do fair traffic-shaping tests because bits will necessarily travel on a different path between ultimate endpoints depending on who is providing the management services.)
The existing cable Internet networks are all highly asymmetric, designed with the assumption that very little upstream bandwidth was needed. That works fine for web browsing but it doesn’t wash for P2P and VoIP. Cable companies can either spend time and money to reengineer their networks or figure out how to control upstream demand. Guess which option they choose. Some cable companies have been using “traffic shaping” to further limit the amount of bandwidth a single user can get. Here’s an example for a Cisco router:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios113ed/113na/1139na/traffic.htm
Notice that their method is to buffer the packets in the router for a period of time to hold the speed down, without exceeding the TCP timeouts. That’s different from the maximim rate limiting which is a pretty blunt instrument; the router simply drops some packets to force the sender and receiver to shrink their receive windows. Although buffering is not a bad strategy for P2P traffic, it would play havoc with VoIP traffic because of the jitter issue.
According to this WSJ article, Shaw cable in Canada is one of the companies already using this technology:
http://www.postel.org/pipermail/end2end-interest/2005-October/005387.html
That article claims that their rate limiting is “smart” and looks at the type of traffic being sent to determine how it should be treated. If Rogers offers its own VoIP service, though, they may think it’s smart to use rate limiting for other VoIP providers exactly because of what it does.
It should be possible to build tools that would detect this type of traffic shaping pretty easily, by sending different types of traffic between the same two endpoints and measuring differences in their delay characteristics, dropped packets, and receive windows.