[Update: I gave a brief interview to Marketplace Tech Report]
The last 48 hours has given rise to a fascinating dispute between Level 3 (a major internet backbone provider) and Comcast (a major internet service retailer). The dispute involves both technical principles and fuzzy facts, so I am writing this post more as an attempt to sort out the details in collaboration with commenters than as a definitive guide. Before we get to the facts, let’s define some terms:
Internet Backbone Provider: These are companies, like Level 3, that transport the majority of the traffic at the core of the Internet. I say the “core” because they don’t typically provide connections to the general public, and they do the majority of their routing using the Border Gateway Protocol (BGP) and deliver traffic from one Autonomous System (AS) to another. Each backbone provider is its own AS, but so are Internet Service Retailers. Backbone providers will often agree to “settlement free peering“ with each other in which they deliver each others’ traffic for no fee.
Internet Service Retailers: These are companies that build the “last mile” of internet infrastructure to the general public and sell service. I’ve called them “Retailers” even though most people have traditionally called them Internet Service Providers (the ISP term can get confusing). Retailers sign up customers with the promise of connecting them to the backbone, and then sign “transit” agreements to pay the backbone providers for delivering the traffic that their customers request.
Content Delivery Networks: These are companies like Akamai that provide an enhanced service compared to backbone providers because they specialize in physically locating content closer to the edges (such that many copies of the content are stored in a part of the network that is closer to end-users). The benefit of this is that the content is theoretically faster and more reliable for end-users to access because it has to traverse less “hops.” CDNs will often sign agreements with Retailers to interconnect at many locations that are close to the end-users, and even to rent space to put their servers in the Retailer’s facilities (a practice called co-location).
Akamai and LimeLight Networks have traditionally provided delivery of Netflix content to Comcast customers as CDNs, and paid Comcast for local interconnection and colocation. Level 3, on the other hand, has a longstanding transit agreement with Comcast in which Comcast pays Level 3 to provide its customers with access to the internet backbone. Level 3 signed a deal with Netflix to become the primary provider of their content instead of the existing CDNs. Rather than change its business relationship with Comcast to something more akin to a CDN, in which it pays to locally interconnect and colocate, Level 3 hoped to continue to be paid by Comcast for providing backbone connectivity for its customers. Evidently, it thought that the current terms of its transit agreement with Comcast provided sufficient speed and reliability to satisfy Netflix. Comcast realized that they would simultaneously be losing the revenue from the existing CDNs that paid them for local services, and it would have to pay Level 3 more for backbone connectivity because more traffic would be traversing those links (apparently a whole lot). Comcast decided to try to instead charge Level 3, which didn’t sound like a good deal to Level 3. Level 3 published a press release saying Comcast was trying to unfairly leverage their exclusive control of end-users. Comcast sent a letter to the FCC saying that nothing unfair was going on and this was just a run-of-the-mill peering dispute. Level 3 replied that it was no such thing. [Updates: Comcast told the FCC that they they really do originate a lot of traffic and should be considered a backbone provider. Level 3 released their own FAQ, discussing the peering issue as well as the competitive issues. AT&T blogged in support of Comcast, Level 3 said that AT&T “missed the point completely.”]
Comcast’s attempt to describe the dispute as something akin to a peering dispute between backbone providers strikes me as misleading. Comcast is not a backbone provider that can deliver packets to an arbitrary location on the internet (a location that many other backbone providers might also be able to deliver to). Instead, Comcast is representing only its end-users, and it is doing so exclusively. What’s more, it has never had a settlement-free peering agreement with Level 3 (always transit, with Comcast paying). [Edit: see my clarification below in which I raise the possibility that it may have had both agreements at the same time, but relating to different traffic.] Indeed, the very nature of retail broadband service is that download quantity (or the traffic going into the Comcast AS) far exceeds upload quantity. In Comcast’s view of the world, therefore, all of their transit agreements should be reversed such that the backbone providers pay them for the privilege of reaching their users.
Why is this a problem? Won’t the market sort it out? First, the backbone market is still relatively competitive, and within that market I think that economic forces stand a reasonable chance of finding the optimal efficiency and leave relatively less room for anti-competitive shenanigans. However, these market dynamics can fall apart when you add to the mix last-mile providers. Last mile providers by their nature have at least a temporary monopoly on serving a given customer and often (in the case of a provider like Comcast) a local near-monopoly on high-performance broadband service altogether. Historically, the segmentation between the backbone market and the last-mile market has prevented shenanigans in the latter from seeping into the former. Two significant changes have occurred that alter this balance: 1) Comcast has grown to the size that it exerts tremendous power over a large portion of the broadband retail customers, with far less competition than in the past (for example the era of dial-up) and 2) Level 3 has sought to become the exclusive provider of certain desirable online content, but without the same network and business structure as traditional CDNs.
The market analysis becomes even more complicated in a scenario in which the last-mile provider has a vertically integrated service that competes with services being provided over the backbone provider with which it interconnects. Comcast’s basic video service clearly competes with Netflix and other internet video. In addition, Comcast’s TV Everywhere service (in partnership with HBO) competes with other computer-screen on-demand video services. Finally, the pending Comcst/NBCU merger (under review by the FCC and DoJ) implicates Hulu and a far greater degree of vertical integration with content providers. This means that in addition to its general incentives to price-squeeze backbone providers, Comcast clearly has incentive to discriminate against other online video providers (either by altering speed or by charging more than what a competitive market would yield).
But what do you all think? You may also find it worthwhile to slog through some of the traffic on the NANOG email list, starting roughly here.
[Edit: I ran across this fascinating blog post on the issue by Global Crossing, a backbone provider similar to Level 3.]
[Edit: Take a look at this fantastic overview of the situation in a blog post from Adam Rothschild.]
I think this shows we need an anti-monopoly solution for cable similar to what we got with AT&T and Bell South.
A solution that allowed multiple companies to sell service on the same lines would prevent any sort of monopoly from forming and has created an environment that lead to innovation in the telecommunications industry for decades.
What this will not do is make the companies stock more popular with quick trading investors. But that is no big deal really. Companies such as these need to focus on building up networks and serving customers as best possible before looking toward profit taking and quite frankly I would not trust such a vital service to the will of the market anyway, where a rumor or less can tank your stock’s value for no reason what so ever other than someones opinion.
I have read all the arguments against this type of regulation for the cable industry and none of them make sense in the greater scheme of things. It would not be as beneficial to the company’s bottom line, but when has it ever when you break up a monopoly and the company has to compete like the rest of businesses do?
All in all I see this as sorta like 2 dogs fighting over scraps and the consumer will lose out if they get their way. Your average consumer could care less if the execs of either company get a new benz or jet this year and don’t want to pay more for a service in a clearly contracting economy. Raising prices of goods and services will not stimulate economic consumption, that requires consumer confidence that this job shedding country cannot deliver on at this time. Contraction will continue to occur until the bubble completely bursts and then will level out and start to grow in small ways, hopefully in a more reasonable and sane manner than has happened in the last decade.
One thing these companies MUST do is increase their bandwidth capacity because in twenty years they will probably need 100 times what they have now to meet demand. This so going to cause a lot of crying and moaning and desire to raise prices by companies who’s leadership is used to a lifestyle most Americans can only dream about. Perhaps they should be thinking more of what they as a company need to be doing to promote the common good and a lot less about padding their own lifestyle while doing it.
The following is based on my admittedly sparse understanding of how CDN’s work. I’d appreciate it if someone who knows more would chime in to say whether or not I’ve got the right idea here.
——-
The difference between co-located CDNs and “remote” CDNs has been articulated in this thread, but I think the initial post should have drawn a brighter line between them.
As I understand it Akamai’s CDN and Level 3’s CDN are fundamentally different. Akamai’s caches are located *in* a retail ISPs network, close to the retailer’s customer-edge. Level 3’s caches, on the other hand, are are located on the edge of Level 3’s network, which is close to the retail’s ISPs “backbone” edge.
From the perspective of a CDN customer (Netflix in this case) and the perspective of end user’s (Netflix subscribers, in this case), the difference between these two types of CDNs is significant, but marginal. Both of them result in lower bandwidth costs for Netflix, and better performance for end-users. My understanding is that Akamai provides better performance for end-users than a remote CDN would, but is generally more expensive for Netflix. It’s then up to Netflix to balance these costs against performance differences.
However, from the perspective of a retail ISP (Comcast in this case), this difference is *everything*. Co-located CDNs result in drastically less load on the retail ISPs network. On the other a remote CDN might actually *increase* the load on the retail ISPs network (as compared to Netflix not using a CDN at all).
To see why this is, consider this scenario. Say Comcasts users request X bytes/second of Netflix video, where X is the size of the stream producing “ideal” quality. If Netflix was not using a CDN and instead served all it’s data out of a handful of data centers, it’s outgoing traffic would encounter bottlenecks from all over the Internet (like any other Internet traffic). Thus, Netflix will have degrade the quality of it’s streams somewhat, and serve only Z bytes/second of data to Comcast customers (where Z < X). Now say that Netflix employs a remote CDN (like Level 3). This CDN will serve traffic closer to Comcast's network, and thus, will encounter fewer bottlenecks. Netflix then serves higher quality streams to Comcast customers at Y bytes/second (where Z < Y Z traffic.
Before this whole spat, however, Netflix was using Akamai, a co-located CDN. Since these caches are even closer to end-users, the stream quality with Akamai is even better say A byes/second (where Z < Y < A < X). But (and here's the key point), those A bytes/second travel over a small fraction of Comcasts network (say f). This means Comcasts costs are more like f*A. My understanding is that f*A (co-lo'd CDN) << Z (no CDN) < Y (remote CDN).
That whole analysis is obviously a gross simplification, but I think it captures the right trends. The overall point is this, retail ISPs seems to be greatly affected by an economic decision they have no stake in: Netflix's CDN provider. Note this means Comcast would have had plenty of reason to be upset about the shift to a remote CDN, unrelated to it's potential conflicts of interest regarding it's position as a competitor to Netflix.
I'm still not at all sure how any of this bears on the economics of peering and the correct regulatory approach to the situation, but it seemed worth breaking down.
Obviously the cost of internal transit will depend on Comcast’s network design. Remember that Comcast serves a very wide geographic area and is free to design its network in any way that it chooses. As far as I know, it isn’t prohibited from setting up caching proxies in areas of concentrated customers.
But, even if the internal transit cost of serving content from remote CDNs were substantially greater than that of co-located CDNs, should Comcast be able to discriminate against remote content providers simply because they chose not to co-locate? Or to extort remote content providers for compensation in order to prevent their packets from being slow-walked through the Comcast network while co-located traffic is put on the fast lane?
Put another way, if X bytes/second produces optimum video quality, and Comcast’s customers want that quality, and a remote CDN is capable of delivering that bandwidth to the edge of Comcast’s network, then failure of Comcast to deliver that bandwidth to its customers across its internal network should be chalked up to some internal Comcast failure — whether from poor network design, inadequate network capacity, etc.
Through all the talk about internet transit providers, I found myself wondering, what about Netflix? By swapping CDN, Akamai with Level3 – the deal is supposed to shift some of Netflix costs to Comcast. Yet, these costs are not simply cut, like streamlined away by a more efficient process – Akamai provides some real service. Comcast wants to recoup these costs. It will do by taxing users. If in the U.S. its like in Europe, where I live, it will do so wholesale.
If Netflix creates such huge amounts of traffic on Comcast’s network, then taxing Akamai/Level3 to me looks like Comcast discriminating between its own users via indirection instead of by traffic volume. (Leaving aside that Comcast and Netflix are competing in the same market.)
As a customer of an internet access provider, I like flat rates and oppose price discrimination that makes offers hard to compare. (I am in a place where I can choose IAPs). So Comcast’s strategy makes sense to me: I’d say, let Netflix customers pay both content (copyright royalties) AND delivery. I don’t watch netflix, why should I pay into their delivery?
Thinking about said discrimination some more, the only fair way I see is tiered by traffic volume internet access offers. Combined costing will make a hell out of it: competition law is to sort it out.
Internet Service Provider is a broad, and sometimes unhelpful, term, but Internet Service Retailer is not much better. Perhaps functional terms would be more useful than market terms. Content Delivery Network is a useful functional term. So are Internet Access Provider, Internet Transit Provider, Internet Mail Service Provider, Internet Hosting Provider, etc.
With respect to peering, it exists not only between backbones. Any two networks (including Internet Access Providers) can peer if they see the benefit in doing so. Peering between Internet Access Providers within the same region will save them both on transit fees.
Which brings me to a thought. In its letter to the FCC, Comcast stated that “Settlement-free peering is typically appropriate when each carrier sends a roughly balanced level of traffic to the other’s network,” and added the footnote that “[i]n this context, network operators routinely consider traffic at a 2:1 ratio to be roughly balanced.” Comcast went on to say that CDNs “are not Internet backbone providers. Their business involves sending significantly more traffic than they receive. For that reason they typically purchase services (‘paid interconnection’) from Internet backbone providers.”
Comcast’s point, it would seem, is that networks are financially responsible for the traffic they originate, and peering is appropriate if there is a rough balance between outbound traffic and inbound traffic.
The “roughly balanced level of traffic” test makes sense in the context of Internet Transit Providers peering with one another — this is traffic for which alternate routes are available, and that would not be routed across an Internet Transit Provider’s network, but for the peering agreement.
However, the “roughly balanced level of traffic” test just doesn’t make any sense in the context of an Internet Access Provider.
Assuming that all of the traffic passed by an Internet Access Provider is either originated or terminated for the benefit of its customers — in other words, that the Internet Access Provider is not acting as a transit provider — then it makes no sense to distinguish between traffic originated and traffic terminated. Excepting spam and such, every packet terminated was “requested” by one of the Internet Access Provider’s customers and was for that customer’s benefit. Because the Internet Access Provider is being paid by its customers both to originate traffic and to terminate traffic, all inbound as well as outbound traffic is for the benefit of the Internet Access Provider.
As long as an Internet Access Provider is acting in that capacity, why should it charge a fee for accepting inbound traffic which its customers are already paying it to accept for their benefit? Who pays his or her Internet Access Provider to only send packets?
Back to the dispute here between Comcast as an Internet Access Provider for retail customers and Level 3 in its dual role as a Content Delivery Network and an Internet Transit Provider. Reading Comcast’s letter, it is clear that Comcast thinks that Level 3 is “trying to game the process” in some way. Level 3’s undercutting the Netflix bids of Comcast’s CDN customers seems to have struck a nerve with Comcast, for obvious reasons.
How is Level 3 gaming the system, though? If NetFlix were using a remote (i.e., not co-located) CDN other than Level 3, and Level 3 were only acting as an Internet Transit Provider, would this dispute exist between Comcast and the remote CDN? Or would Comcast still have a problem with Level 3 as an Internet Transit Provider?
If the former, does that implicate net neutrality issues? Does any remote content provider “owe” an Internet Access Provider compensation for accepting and delivering traffic requested by that Internet Access Provider’s customer?
If the latter, does that also implicate net neutrality issues? A consumer-oriented Internet Access Provider’s inbound traffic will always be greater than its outbound traffic, and that ratio will only increase as more audio, video and other large file content is consumed by the Internet Access Provider’s customers. If an Internet Access Provider insists that it won’t accept “unbalanced” traffic without payment, isn’t it essentially saying that someone (other than its customers) is going to have to pay a toll in order for it to accept and deliver traffic requested by, and destined for, its customers?
In its letter to the FCC, Comcast says that the dispute doesn’t involve net neutrality.
“The dispute between Comcast and Level 3 relates to how Level 3 wants Comcast (and presumably other networks) to treat their new influx of CDN traffic, but it has nothing whatsoever to do with any content, application, or service that Level 3 is transmitting. This concerns only the vast increase in the amount of traffic Level 3 told Comcast that it wants to send to Comcast’s network on a ‘peering’ basis.”
Comcast’s argument here is essentially “we aren’t discriminating against content, just against a particular speaker, whose volume of speech is so great because of the demand for it by our customers.”
In its letter, Comcast also says that the dispute doesn’t involve tolls on delivering content.
“Despite Level 3’s complaints, Comcast is neither resisting carrying Internet video traffic nor imposing new ‘tolls’ on Internet video traffic. The simple fact is that Comcast terminates huge amounts of online video traffic to our high-speed Internet customers, most of it pursuant to longstanding, mutually acceptable commercial arrangements we have in place with the leading CDN companies. Our customers get access to all the online video they want, along with any other Internet content, application, or service they choose — regardless of its source.”
Comcast essentially makes the novel argument that this dispute is not about imposing tolls on Internet video traffic, as evidenced by the fact that Comcast already delivers huge amounts of video — from sources that pay it tolls under “mutually acceptable commercial arrangements.”
Steve, you’re correct that one of the reasons for this dispute is that Comcast is trying to apply a rule designed for peering between Internet Transit Providers — the “roughly balanced level of traffic” test — to a relationship between an Internet Transit Provider and an Internet Access Provider, pretending that the Internet Access Provider derives benefit only from outbound traffic.
There are other interesting questions to be sure:
Did Level 3 assume that its dual role as Internet Transit Provider and remote Content Delivery Network would give it an economic advantage over Akamai and Limelight, and calculate its fees to Netflix accordingly? Did Level 3 (as CDN) omit to pay transit fees to Level 3 (as Internet Transit Provider)?
More generally, why can’t any common carrier resist the temptation to transform itself into a gatekeeper, when all its customers really want is a reliable common carrier?
Very well put.
I was discussing this lately in a private mailing list. It was pointed out that because of the way routing usually works, traffic can be expected to cross from the L3 network to the Comcast network near the source of the traffic, rather than near the destination. That is, Comcast would wind up paying most of the cost of shipping the data from the nearest server to the customer, even if those are in different cities. It doesn’t seem unreasonable for them to ask to be compensated for doing so.
There’s still a natural limit to how much they can charge, because L3 could always send the traffic via other backbone provider(s) at standard rates, so I don’t think there’s a monopoly issue in this regard. Also, to demonstrate a network neutrality issue you’d have to establish that Comcast were discriminating against types of traffic, rather than just based on volume.
If you want to break this down to its simplest level. Akamai paid Comcast to connect (or peer), then why should Level 3 not pay as a CDN?
From the Comcast letter Level 3 requested (or demanded) 24-27 extra ports (for extra bandwidth). Comcast was not obliged to provide those interconnection ports. If Comcast did not (or does not) provide these ports in the future, Level 3 would be in a bad position as a CDN trying to send data to Comcast subscribers. It wouldn’t be Comcasts fault, it would be Level 3’s. In business if one person wants something from another, the one that wants it exchanges something of value for it. If Comcast wanted something from Level 3, Comcast would have to pay for it.
Indeed, and typically the government would do best to stay out of it unless there is some evidence that one company is exercising market power in an anticompetitive way… which is what the mismatched level of competition on the different sides of this two sided market indicates could be happening, not to mention the incentives that vertical integration generates.
Settlement-free peering, which is what Comcast and Level3 have/had, means that two backbones interconnect at multiple locations, each paying their own costs, with no exchange of monies between providers.
Comcast published a URL to their settlement-free peering policy in one of their public posts.
Settlement-free peering is practical because it allows backbones to interconnect with technical agreements and minimal business agreements, because no money changes hands.
Over time, as Internet service companies evolved their services, settlement-free peering got more complicated. The advent of content distribution networks, like Akamai, caused a lot of change. There is no technical reason to charge a content-distribution network an interconnect fee if the CDN has the right architecture, meaning that all traffic exchange is local and the backbone is not providing transit. That was the old days.
CDNs pay for access to consumer Internet users, while Netflix does not. Cable companies pay broadcast networks now, when they didn’t before. Go figure. (One reason is to give the broadcast networks a reason NOT to move content to the Internet.)
Level3 pulls a fast one on Akamai and steals Netflix traffic, hoping its peering agreement with Comcast (and everyone else) will hold up. It didn’t. Level3 is being assholish about it and it’s going to redound to their chagrin as all the other players re-assess their relations to Level3. Comcast shrugs and explains how it is and always has been.
Not really much of a story. Just a little of the scene behind the curtains.
But it’s not clear to me that Level 3 was in fact a settlement-free peer of Comcast. See my reply to the first commenter above.
Well, they did in the past, indirectly via Akami and LimeLight, as I described in the post.
I’m not sure that’s “how it is and always has been.” Prior to the advent of CDNs, last-mile providers never got paid for anything about the traffic generated by their customers. As for how it’s always been between Comcast and Level 3, as far as I can tell it’s always been the case that Comcast paid Level 3 (per their own letter to the FCC).
I realize you were focusing mainly in the ISP aspects of the dispute but Comcast’s conflict of interest in allowing its customers to access Netflix’s video services and possibly drop their cable TV plan should be a part of the discussion.
[apologies – this is before the first cup of coffee]
There is an error in the starting premise which affects the whole of this discussion. It is bogus to presume that is that there is some natural hierarchy where “backbone only” networks not only exist but are in some way necessary and required. It tends to go hand in hand with people who talk about “connecting to the Internet” instead of “the rest of the Internet”.
While a useful evolutionary step away from the NSFnet days, the dinosaur-like long-haul telecom carriers have learned the lesson of every long-haul service (trains, planes, barges, etc): it is very expensive and subsidies are needed to keep you afloat. This is why we’ve seen “retail” consume long-haul in the US: SBC (+ameritch + pacbell + bellsouth + snet + ….) consumed the former AT&T; Verizon consumed the former UUnet; Nextel ate Sprint; CenturyLink is trying to consume Qwest (already USWest) … Level3 had been trying to sell itself well before the recent flirtation with delisting. Some folks build infrastructure instead, especially when they have crew and material and only need to sort out medium-size spans to stitch together existing customer-service areas; that is how many small ISPs grew into regional and national entities. Comcast has taken that build-and-lease approach. If you replace “medium” with “long” then that also describes Level3’s entrance with the use of Kiewit construction assets.
As more infrastructure has been built globally, the ‘net has become VERY well meshed, utterly unlike the top-down hierarchy described here. It is one reason why actual network engineers are still upset at the current state of IPv6; the IETF dropped any concerns for scalability in managing the vast swaths of address space and assumed a top-down model, already dead before v6 was fully baked.
There are very few networks which claim to be transit free, and almost none which are 100% transit free; “settlements” occur on many levels, including decent rates on undersea cables, capital expenditure (buying hardware for your partner), terrestrial rights of way, etc.
And to be blunt few is still too many as those folks actually bring less value to the table while they claw to maintain what is essentially a marketing title. Their fragility and co-dependence can be seen anytime there is a peering dispute which actually partitions the global Internet. In many respects their lack of a path-of-last-resort requires them to act in concert, not solely focusing on their business’ long-term direction but rather on maintaining that status.
Sadly, there are folks who continue to buy into a legacy model and believe that there is some non-marketing reward for climbing an illusory ladder. Given modern traffic analysis tools, a network can better determine effective interconnections (peering, transit) for their business and customers. Many find that regional deployment strategies rather than shotgun “collect them all” peering works. It also allows one to better see what transit networks are appropriate to service your traffic.
“All fine” you say, “but what about this dispute?” The background above is important to understand that *no*single*network* is a “backbone provider that can deliver packets to an arbitrary location on the internet”. Some networks are stronger in some regions; years ago for example Level3 made the decision to sell off its Asia-Pacific holdings and to date has not built into Latin America. Yet all the while it maintains the mantle of a “global backbone”. Comcast took the decision to not cross the Atlantic and invest closer to their customers; how different are they in effective footprint or reach? Why should the normal dynamics of traffic carriage cost be set aside for perceived benefits of market position?
I don’t have a clear-cut answer, but hate to see the discussion start off with faulty footing.
You got me. I was aware that I was over-simplifying some of these distinctions in the post. I agree with you that it is a myth that there are a few true “tier 1” backbone providers that can reach any arbitrary location on the net, and that “retailers” have been steadily growing their networks and at times merging with backbone providers. The way I tried to massage this was referring to large backbone providers “that can deliver packets to an [not any] arbitrary location on the internet (a location that many other backbone providers might also be able to deliver to).” The difference between Comcast and Level 3 is that Level 3 delivers to a much more diverse set of locations, and that Comcast holds exclusive delivery control over every one of its destinations. These are the dynamics that I think ultimately make the difference in my market analysis (in addition to the vertical integration concerns).
I do agree that the blurring of lines between some of these providers is an issue, as I described in point #4 over here (including the Verizon/MCI/UUNET example you also used).
Someone pointed me to an interesting SIGCOMM paper:
Internet Inter-Domain Traffic
by Craig Labovitz (Arbor Networks), Scott Iekel-Johnson (Arbor Networks), Danny McPherson (Arbor Networks), Jon Oberheide (University of Michigan), Farnam Jahanian (University of Michigan)
Are there really that many high-bandwidth long-haul connections around, or is it just the aggregate of all the shorter-haul stuff that provides more bandwidth than the long-haul routes? (And if the second, what are the implications for routing and QoS issues, since it seems you would be walking about way more hops?)
I see some of the same expansion-of-big-locals phenomenon as you, but put most of the basis for it on the effective monopoly that the locals/retailers have on end customers. Even in areas with multiple ostensible service options, there’s typically nothing like a real competitive market.
“really that many” = yes, in many of the places where n>0, n>1.
“just the aggregate” = no, except in places where the distinction never existed and the question is meaningless (e.g., monolithic national telcos)
“typically nothing like” = bingo
Re: “What doesn’t make sense is that Comcast claimed in its letter that it exchanged traffic with Level 3 under a settlement-free peering agreement. Specifically, it said it paid for interconnection facilities, but then had a settlement-free peering agreement.”
Suffice it to say that there are a variety of contractual alternatives between the stereotypical “pure” peering/SFI and “pure” IP transit. In some cases, aspiring or current or previously interconnected parties that are sizing up/down substantially relative to each other may find that neither of the more familiar contractual options is desirable or feasible. This is especially true at the upper end of the interconnection spectrum, where both parties have already achieved a scope and scale of operations sufficient to negotiate a mix of direct/indirect reachability with most of (the rest of) the Internet.
In such cases, the stakes involved in the outcome can be very big and also quite durable. This fact, coupled with the natural tendency of opposing parties to each interpret (sometimes loudly) ambiguous facts in whatever terms happen to be most favorable to their own interests, occasionally results in just this kind of flap.
If you can still be surprised that “retailers are now big enough to boss backbone providers around,” it can only mean that you haven’t been paying enough attention for the last 5-6 years. Or perhaps this is just a symptom of the mis-identification of the industry segments involved. The divide is not between “retailers” and others (“wholesalers”?), but rather between the relatively few companies that hold large swaths of (in general, economically incontestible) “turf,” and the vastly larger and more diverse mix of “extra-territorials” that operate in and between the 10k or so big, well-connected downtown buildings and Internet data centers around the world. For the overwhelming majority of the latter, their current existence and future survival (at least as independent companies) is contingent on past or present laws that compel the turf holders to facilitate access to “retail” end-users, or at minimum prohibit them from actively impeding reciprocal efforts by the other two parties to reach each other. This divergence in competitive environments has been exacerbated greatly by advances in optical multiplexing technologies, which have all but eliminated barriers to entering the “extra-territorial” market segment. Meanwhile, in relative terms turf has become even less contestible in many places, and absent some unlikely countervailing development this trend will only continue.
What happens when interdependent market-segments come to be subject to radically and increasingly asymmetrical entry and competitive barriers? For a hint at the answer, try counting the number of large backbone providers that are “independent” (i.e., not owned by one of the turf holders) today vs. a decade go, then ask yourself: what happened to the companies that changed status? Now try that same calculation for the “turf” based segment of the industry.
I’m having trouble clearly understanding the difference between your description of the market segmentation and my original description. Can you put some company names on the examples you describe? What are examples of this “turf” you describe other than last-mile retail service?
The confusion may be a result of the gap that opened up between the question that I quoted and intended to respond to (Joe’s “Quick question” @ December 2nd, 2010 at 2:40 am), and the current downfield location of that response.
But just in case it’s not, I will clarify as best I can:
In your original description, your use of the term “retailers” is largely orthogonal to the underlying mechanics + economics of Internet traffic exchange, which is only context in which discussions about the present subject can be usefully undertaken; it’s the only context where terms like the ones being debated here have any concrete meaning. If that’s not clear enough, consider that there are a wide variety of commercial Internet service “retailers” around the world that bear little or no resemblance to your description, some of which lack the technical prerequisites to even want to “peer” (granted, the latter are a nearly extinct breed in the US). At the same time, there are also many completely vertically integrated “retailers” that retail much more than just Internet access — and for whom interconnection decisions and contractual mechanism preferences are based on a variety of factors, some of which may have nothing to do with the present cost or quality of IP transit or content delivery mechanisms. Also, there would seem to be a large number of what you might call “retailers” that don’t in fact “build the ‘last mile’,” including some that never did, while it’s not easy to find places where one could say that anyone is building “last mile” outside of those places where it wasn’t already built.
My comments about turf were intended to point out yet another critical dimension of variance that has already almost completely upended the kind of conceptual categories used here and in many other places (note: JZP did an excellent job explaining this already, so I won’t repeat), and which is only going to become more determinative over time. I’m not going to give out the answers to my earlier questions either, but your subsequent pointer to “point #4 over here” leads to me believe that you could compile a long list of examples easily enough yourself with a little effort.
Perhaps I should chalk it up to simply being dense, but I still don’t see your specific explanation of how my description of retailers is orthogonal to the actual mechanics of the market. Indeed, although I readily admit that there are cases where the last mile has pushed into the backbone and vice-versa, and cases where the peering arrangements are not as cut and dry as my generalization, the characteristics of the retailer at hand seems to be clearly determinative of the market dynamics (namely, high degree of market power in the last-mile and vertical integration into the video market by last-mile providers). If there is a specific reason why it is not, I am all ears.
Great post — one of best explanations I’ve seen. Quick question though:
You say that Comcast has always paid Level 3 as a transit customer. That makes perfect sense to me. What doesn’t make sense is that Comcast claimed in its letter that it exchanged traffic with Level 3 under a settlement-free peering agreement. Specifically, it said it paid for interconnection facilities, but then had a settlement-free peering agreement.
I’m just trying to make of sense of that. Because you say: “What’s more, it has never had a settlement-free peering agreement with Level 3 (always transit, with Comcast paying).” Again, that’s what I had thought too – but that’s not what Comcast said in its letter, unless I’m missing something.
Was just hoping you could straighten that part out for me. Thx
But you’re absolutely right that the big story here is that retailers are now big enough to boss backbone providers around
Yes, it is a bit confusing. Let me first quote from the Comcast letter to the FCC:
What I take from this quote is that Comcast was a Level 3 transit customer. When Comcast says “they exchanged their on-net traffic on a settlement-free basis,” I am somewhat confused, and indeed it seems almost contradictory with the preceding sentence. I don’t understand whether “on-net traffic” is somehow distinguished from other traffic, or if Comcast payments for “interconnection facilities” really wasn’t a transit agreement after all (I’m fairly sure it was/is). Maybe someone here can clarify.
Upon further investigation, it appears that the “on-net” traffic is precisely what it sounds like: traffic that originates and terminates on the same network (ie within the same AS or at least within AS’s run by the same entity). This is what a slide deck from a guy who works for Global Transit says:
Comcast evidently defines it as traffic originating from the network in question, but perhaps terminating on their network. According to their peering agreement:
This of course begs the question, raised by Anonymous below:
When network administrators talk about on-net traffic or on-net services in general, the idea is that it originates and terminates within a pair of peer networks. Comcast’s historical agreement with L3 has been for Comcast to pay L3 for transit to L3’s customers, but for L3 and Comcast to exchange on-net for no fee. That means traffic that originates on Comcast and terminates on L3 or vice versa was exempt from the transit agreement.
L3 is making a tricky move here in claiming that the Netflix traffic is on-net. In a very narrow technical sense, it is, but only because L3 has added a CDN service to their network to bring it onto L3’s network. L3 is *logically* moving traffic from a customer of theirs – Netflix, who pays them to handle the traffic – to a peering partner who they expect will complete the delivery for free.
The question I have about this is why Netflix should pay L3 at all, given that L3 apparently expects Comcast to do most of their work under their agreement with Netflix for free.
I think your description of how they are defining on-net traffic is correct. However, I think you might have things backward after that:
It seems clear that Comcast is making this claim, whereas L3 is claiming it’s transit. That’s why L3 is saying it’s not a settlement-free peering issue at all. L3’s position seems like the most logical position to me (and equally so by your reasoning).
Comcast is claiming it’s a peering issue, but just that the balance of traffic exchanged overall has shifted to the point where L3 should start paying them. This makes no logical sense to me, because Comcast is an eyeball network and the traffic is being requested by their customers. This also makes no market sense to me (or at least, it does not make sense that such behavior would lead to optimal market allocation) because Comcast has significant (with a small “s”) market power whereas L3 does not.
I would assume that there are at least two reasons. First, L3 gets the traffic to the interconnect point with Comcast which undoubtedly incurs costs and requires expertise that Netflix does not have. Second, L3 delivers to the rest of the internet… not just Comcast.