While many of us were discussing Comcast’s partial blocking of BitTorrent Traffic, and debating its implications for the net neutrality debate, a more clear-cut neutrality violation was apparently taking place on Verizon’s network – a redirection of Verizon customers’ failed DNS lookups, to drive traffic to Verizon’s own search engine.
Here’s the background. Suppose you’re browsing the web and you mistype an address – say you type “fredom-to-tinker”. Your browser will try to use DNS, the system that maps textual machine names to numeric IP addresses, to translate the name you typed into an address it can actually connect to across the Net. DNS will return an error, saying that the requested name doesn’t exist. Your browser (if it’s a recent version of IE or Firefox) will respond by doing a search for the text you typed, using your default search engine.
What Verizon did is to change how DNS works (for their residential subscribers) so that when a customer’s computer looks up a DNS name that doesn’t exist, rather than returning the name-doesn’t-exist error DNS says that the (non-existent) name maps to Verizon’s search site. This causes the browser to go to the Verizon search site, which shows the user search results (and ads) related to what they typed.
(This is the same trick used by VeriSign’s ill-fated SiteFinder service a few years ago.)
This is a clear violation of net neutrality: Verizon is interfering with the behavior of the DNS protocol, in order to drive traffic to its own search site. And unlike the Comcast scenario which might possibly have been justifiable as legitimate network management, in this case Verizon cannot claim to be helping its network run more smoothly.
Verizon’s actions have two effects. The obvious effect is to drive traffic from the search engines users chose to Verizon’s own search engine. That harms users (by overriding their choices) and harms browser vendors (by degrading their users’ experiences).
The less obvious effect is to break some other applications. DNS lookups that have nothing to do with browsing will still be redirected, because the DNS infrastructure has no way of knowing which requests relate to browsing and which don’t. So if some other application does a DNS lookup and the result should be a not-found error, Verizon will cause the result to point to a Verizon server instead. If a non-browser program expects to see not-found errors sometimes and has a strategy for dealing with them, it won’t be able to carry out that strategy because it won’t see the errors it should be seeing. This will even cause browsers to misbehave in some circumstances.
The effects of Verizon’s neutrality violation can be summarized simply: they interfer with a standard technical protocol; they cause harm on the whole, in part by breaking unrelated services; and they do this in order to override consumer choice by shifting traffic from consumer-chosen services to Verizon’s own services. This is pretty much the definition of a net neutrality violation.
This example contradicts at least two of the standard arguments against net neutrality regulation. First, it shows that violations do happen, and they do cause harm. Second, it shows that at least sometimes it’s easy to tell a harmful violation apart from legitimate network management.
But it doesn’t defeat all of the arguments against net neutrality regulation. Even though violations do occur, and do cause harm, it might turn out that the regulatory cure is worse than the disease.
I used the dns opt out as directed on verizon page. however, it seems that every time i set up the opt out, my net connection is nothing short of problems. connections were established, but traffic going out is slow. if it is not slow, it dies completely and periodical at a high frequencies. the dns redirection is nothing short of hair polling….
Look, net neutrality, while a nice thought, can’t happen. everyone wants his or her own piece of the internet and pays a great deal for it. Your browser, whatever it is, does the same thing. Think about it..Internet explorer, by default, brings up the msn search page when it can’t find your expected site; netscape does the same thing. ISPs have been doing this for years. Verizon’s very late in the game and why not? it’s a great way to drive more traffic to your business. it doesn’t cause any harm to your computer and, frankly, is it that difficult to find a browser hack to over-ride any isp’s dns landings? no, of course not. google’s been doing it for a long time now. remember the IE search engine hack? even the fed. government is pushing net neutrality. you can’t seriously believe for one second that the government doesn’t want to control the internet can you??
verizon’s own portal provides very explicit instructions on opting of the dns assistance if you just can’t stand it.
Thanks for calling this out. My “ISP,” EarthLink, does the same thing. Currently I’m running my own nameserver just to avoid this, which is obvious suboptimal.
I’m not entirely sure that this is a net neutrality issue. Verizon is not redirecting or blocking any connections, however, they are breaking the DNS protocol.
If I write a program that does a nslookup on any given TLD, then does something else with the results of the nslookup, my program will produce unexpected results if I use Verizon’s DNS servers. As many have stated, this will also break functionality in some popular web browsers.
However, it is not (technically) hindering the neutrality of their network, only connections that would normally fail have been re-directed. This does not mean that their tactic is any less crummy.
I prefer to run my own DNS server (locally) that digs the roots directly. Other (clean) public name servers such as 4.2.2.2 and 4.2.2.3 would be viable alternates to using Verizon’s DNS.
Many people don’t realize that you are not married to your ISP’s DNS servers, any Verizon customer who is annoyed by this behavior can easily circumvent it.
Ned,
Your right. I do have a rather narrow and very specific view of how network neutrality should be defined.
I do believe that at some point (maybe not now, but probably soon) there will need to be some type of regulation to enforce net neutrality. Personally I do not trust our government (or any other government for that matter) to write legislation that doesn’t do more harm than good. But since I think that the shortsightedness and greed of the network providers will require government intervention, I’m resigned to getting the government involved.
Needless to say, I don’t want any legislation that does get passed to be too broad or require too many things. The only thing I would want that legislation to ensure would be the neutrality of the packets sent across the network. I don’t feel that what is done with the packets at the destination or originating point has anything to do with the neutrality issue and I don’t think it should be regulated. As long as the packets and the destination are not tampered with or discriminated against by the network provider, then I’m satisfied.
[…] packets […]
Jamie,
It seems to me that you’re firmly committed to what we might call a “narrow view” of network neutrality. I’m distinguishing that in contrast to what we might call a “broad view” of network neutrality, as articulated by Dr. Milton Mueller in his recent paper &ldquo:Net Neutrality as Global Principle for Internet Governance”, presented at the 2007 GigaNet Annual Symposium last Sunday.
In his paper, Dr. Mueller advocates that the network neutrality principle should be read as “Nondiscriminatory, universal access to Internet
resources.”
I take it you disagree with Dr. Mueller’s “broad view”, and insist upon a “narrow view” of network neutrality.
@Ned Ulbricht
I do agree that it is a problem. And it may need to be addressed, but I don’t agree that it is a Net Neutrality issue.
Again, they are not intercepting, degrading, or redirecting packets sent on their network.
You are not required to use their DNS servers. You are welcome to use DNS servers from another source. This would only be a net neutrality issue if they were redirecting all DNS requests on the network to their servers.
Is this a problem? Yes. Is it a Net neutrality violation? No.
posted this over on bbr the other day …
I will further this argument by asking what right does an ISP have to co-opt my subdomains? Say I own billsbigcars.com, I choose not to use wildcards on the nameservers for that domain for some technical reason, and one of my potential customers tries to go to toyottas.billsbigcars.com, with an obvious misspelling of toyotas. The user (at least on Insight … I am sure others) will STILL be redirected to the ISP search page, even though billsbigcars.com exists. What’s worse, they may immediately see an ad for bobsbigcars.com, a fierce competitor, click on their site, and buy a car … leaving me holding the bag while Insight makes money off of the ad.
This practice reminds me of the old story of the competing tow truck companies (or similar service) … COMPANY A fooled the phone company into setting up remote access call forwarding FOR COMPANY B… and often during rush hour would forward COMPANY B’s calls to COMPANY A’s phone number. This was outright fraud … and to me the redirect situation described above is exactly the same thing!
All they are doing is modifying an application that runs on the network […]
Jamie,
At the risk of sounding like I’m arguing from authority, let me quote from Nominet’s recent position paper (28 Oct 2007; p.1):
To my ears, “the received technical wisdom” sounds awfully pompous, but then I’m not from the UK. Pompous or not, though, I’d hope you’d agree that DNS isn’t just any old application that runs on the ‘net. It’s something that gets used by most other network applications.
And in case it might seem that this paper is a little off to one side of the debate, let me quote a little further (p.2), under the heading: DNSSEC, the basic purpose:
Verizon is mounting a classic MITM attack—tampering with data in transit. Now you could argue that this information isn’t all that valuable, and tampering with it doesn’t cause excessive harm. But to the extent that Verizon profits from tampering with DNS traffic, that gives them an incentive to avoid deploying DNSSEC.
The ultimate control over the DNS servers used still lives with the customer. Unfortunately they have to be somewhat technically savvy to exercise that control.
One caveat: The following assumes that neither your home network router, Verizon, nor any ISP upstream of Verizon are redirecting DNS queries. For example, there are plenty of home network routers, including the Linksys WRT45G product, that redirect all DNS queries by default (a common term for this is dnsmasq, the name of some software that does such magic) to the default server(s) specified by your ISP in its DHCP data used by your router.
Your computer’s IP network configuration includes the ability to specify the DNS servers your computer uses to resolve its DNS queries. For example, within Windows, you are looking for the IP Protocol’s Advanced properties, DNS servers setup. With Linux, this is either via DHCP client configuration, or via your /etc/resolv.conf configuration file.
If you don’t specify your DNS servers, and few do, they default to the DNS servers provided by your ISP. This is usually via your network router’s IP interface, as setup by ISP’s DHCP servers.
For my home network, I choose to configure the DNS servers myself. In fact, I use a commercially provided DNS service, instead of the servers provided by my ISP. Why? In my experience they are faster, and much more reliable than my ISP’s.
In fact, various commercial entities commonly offer such DNS services (because ISPs can suck so badly in this regard). Some of those DNS services are pay based, others acquire their money in a manner that is similar to Verizon. I.E. they offer advertisements when DNS query failure occurs. In any event, I find them more honest (they are very open about their terms of service, as related to how they make money) than ISPs.
Keywords to goggle (I am not endorsing these services): opendns.org, or dyndns.org, to name two such commercial entities.
As usual, given the IP network architecture, the user has lots if control, given they *own* their own computers. However, it can sometimes take plenty of technical savvy, and persistence to exercise that control over Internet experience.
I’m not sure I agree with the basic premise of this article. I don’t see this as a net neutrality violation. They aren’t degrading, prioritizing, or redirecting packets on their portion of the network. All they are doing is modifying an application that runs on the network so that it behaves in an unexpected, at least to the user/application, fashion. The user is not required to use Verizon’s dns servers.
If they were forcing all dns requests to pass through their dns servers, then it would be a clear violation. But since they aren’t, it’s just a rather shady and underhanded thing to do. It’s something the users of Verizon’s service should complain about, but not something that needs the government to get involved in.
Net neutrality is a big issue and something I’m quite worried about, but this issue has nothing to do with it. In this case, the network itself is still neutral(at least as much as its ever been).
Jim,
Propagation delays for DNS zone changes shouldn’t surprise too many people. And, although I’m not trying to minimize the importance of TTL management, all the same, it’s not reasonable to expect DNS to be fully consistent at all times. It’s a large, distributed database—changes occur frequently—and it’s optimized for performance over consistency.
Nevertheless, when different service providers hack DNS into some kind of provider-specific DWIM lookup tool, then it introduces a new dimension of inconsistency in the namespace. There are now circumstances where lookups from one resolver may not only never be expected to return the same results as lookups using another resolver, but the resource records returned are different at a user-visible level. This is undesirable.
And I haven’t even mentioned the problems with specializing DNS for the IntarWeb. You already mentioned that.
(Btw, and fwiw, if you haven’t been paying attention to the Rio governance circus, there appears to be a new high-level policy code-phrase: “Critical internet resources.” Interrogate the usual suspects.)
Verizon redirects typos that don’t return a valid domain name to a search page (as does Earthlink, and as Global Crossing has also tested, and you can find my position on the matter expressed as a comment at that link). Most web browsers have similar functionality to redirect failed domain lookups to search engines–is that also a violation of net neutrality, or is consenting to use an application on your machine different from consenting to use a provider’s DNS server? If unexpected new (yet possibly helpful) behavior from a provider’s DNS server is objectionable, is the same true for behavior from, say, a search engine itself, such as when Google changes its ranking rules? Or is the issue here that there’s an RFC violation?
What about the case where DNS redirection is only performed for failed lookups that are directed to MSN’s “no default search engine selected” domain by IE?
While you may be correct that a regulatory “cure” could be worse than the disease; I nevertheless get quite irritated by those who continue to advocate (in the face of repeated neutrality offensives) that a mysterious market force will somehow correct the situation.
The simple fact, even in the face of repeated public exposure, offending companies still continue to find innovative ways ways to manipulate the user’s experience in an underhanded manner to benefit the offending corporation. So I don’t think, based on anecdotal evidence, that “market forces” truly prevent this type of abuse.
Unfortunately, for me – as a non-technical person – I have no way to know if I am getting screwed or not. I have to depend on websites, such as this one, to expose this type of abusive behavior. Thank-you for clearly presenting the facts and making this issue available for public review.
The telcos in the US are trying to convert the Internet into an AOL model. That is why they redirect DNS requests, tamper with torrent traffic, block inbound traffic to customers, sell “web space”, etc.. The list is getting longer and longer every month. Most AOL customers told AOL to go forth and multiply. Unfortunately we now have to deal with monopolies (sometimes duopolies of the same spawn). That means the telcos can dictate terms. And they do, and here goes the Internet.
Yeah, when I was on Charter this was the thing that bugged me the most (even more than the unreliability of their “service”). Worse than Verizon, their “opt-out” option doesn’t actually opt you out–it just sets a cookie that instructs their servers to redirect to the default Internet Explorer search page! (This is even more useless than Charter’s page, which is, at least, Yahoo instead of MSN.) I actually wrote a Greasemonkey script that would blank out this page, allowing me to retry the URL (because Charter’s page would appear before a slowish server had been able to respond).
Charter Communications (3rd largest US cable operator) does the same thing. Try a few bogus DNS queries against 24.197.160.17 or 24.197.160.18
Here are DNS Opt Out instructions pages:
FIOS: http://netservices.verizon.net/portal/link/help/item?case=dns_assist&partner=verizon&product=fios
High Speed Internet: http://netservices.verizon.net/portal/link/help/item?case=dns_assist&partner=verizon&product=dsl
In the Verizon result page, there is a link to page about this DNS redirection service and that page includes instructions to ‘opt out of DNS redirection’ by changing the DNS settings in the Verizon home routers to use an alternative set of DNS servers that still provide the normal DNS not found errors. Verizon provides detailed instructions for each type of home router.
Mark
Isn’t DNS more akin to an application-level service?
Tim,
That’s something of a matter of perspective….
First, if I was advising protocol or application designers, I’d advise them to design their protocol or application so as to avoid complete dependence on DNS. That’s important for robustness reasons: you want to be able to troubleshoot or fix using IP layer addresses. It’s also important for portability and interoperability: you want to work using another name-to-address mapping service (e.g. yp). And it’s important for upgradability: you want to keep working with any successor protocol to DNS.
Otoh, there are some protocols that, even though they layer over another protocol, are still closely tied to lower layers. Think ICMP, for instance. These layers don’t quite fit into a neat vertical model. Simplifications —like the OSI reference model— don’t quite capture all the nuances and inter-relationships of a real-world stack.
A number of people would suggest that DNS is one of those essential layers in the Internet protocol suite. In fact, I think a lot of the philosophical attachment to a ‘single root’ flows from a perceived equivalence between the public DNS namespace and ‘The Internet’ (The capital ‘I’ Internet).
Umm, isn’t the following circular reasoning: “First, it shows that violations do happen. …”. That is, if you define a term so broadly and vaguely so that you can claim “violation” using the broad and vague sense, that’s not saying much.
I can confirm Jambarama’s description of OpenDNS behavior, because I have used it with both Verizon and Comcast, and that is what I see.
I think, also, that the possible failure modes of search-intercepted DNS are overstated; if there’s weird behavior, it is not noticeable (and since my OpenDNS references are made from my firewall, I should notice a difference whenever I use my laptop outside the house). That said, I much prefer that it is my choice; if I decide that OpenDNS has screwed up, I can easily turn them off.
I’m not so sure that this is a clear violation of net neutrality. I imagine Verizon considers this a feature – it can be useful to have a search when you try to access a domain name. I do believe that Verizon should make this feature optional (but I think it is OK for it to be on by default). A clear violation of net neutrality would be Verizon redirecting all lookups to their DNS servers, blocking the use of services like OpenDNS. Personally I believe the Comcast case is a more serious violation than Verizon’s.
BTW, OpenDNS allows you to turn off the typo correction service (if you create an account and claim your IP address).
Isn’t DNS more akin to an application-level service? I.e. if my ISP provides an email server that fails to implement SMTP correctly, that’s certainly annoying, but it’s not a network neutrality violation. I’m free to (and in fact, do) ignore my ISP’s email server and connect to another email server of my choosing. By the same token, unless they’re actually blocking or redirecting DNS queries to servers other than their own, Verizon isn’t stopping anyone from getting the DNS services they want, they’re just offering a default DNS server that behaves in a less-than-ideal manner.
Of course, given that DNS servers are typically provided by DHCP and most users wouldn’t have the first clue how to manually change them, maybe this is a distinction without a difference. But conceptually, at least, I always thought of network neutrality as an issue of routing policies at the IP layer. And it doesn’t sound like they’re doing that.
@Shun: Basically you get the DNS from the DHCP server that also gives you your IP address. To use OpenDNS you ignore the default DNS server you get from the DHCP server, and use the provided OpenDNS one.
OpenDNS essentially already does this though. They direct you as best they can, and they can handle most basic spelling mistakes. But when your site request is ambiguous, OpenDNS sends you a list of search results with ads – just like verizon.
The only real difference I see is that you don’t have to use OpenDNS. You don’t have to use Verizon’s DNS either (you could switch to OpenDNS, FreeDNS, or any other DNS you can find), but we seem to be some weight to “default” status if there is some barrier to changing the default.
When IE was default on Macs people kicked and screamed. When IE7 used MSN search as default, google claimed it was too hard to change this, and as such was unfair. There are dozens of examples of this.
Anyway, so to your question, if you’ve setup OpenDNS, you will not be affected by this.
How does this work with OpenDNS? Do queries to OpenDNS servers automatically get routed to Verizon DNS servers? That would be more of a coup. I’m curious because I run Linux at home, and mess with my resolv.conf file every once in a while. It would be disappointing to know that this would do me no good in a Verizon-dominated service area.
I like Hal’s idea, btw. The problem is, no one has a beef directly with Verizon to make this attack feasible. “Man that last bill was too high…release the zombies!” only goes so far. You wouldn’t necessarily need customer support to do this. A network of compromised computers would be sufficient.
Could you have a little program that would throw out a zillion DNS lookups a second to made-up names? I wonder if that could make the Verizon servers fill up with state, especially if a lot of Verizon customers could be persuaded to run it occasionally.
In some web browsers, if you merely type the domain name (i.e., leave off the hostname and the TLD), the failure of DNS to resolve the domain name as a hostname under the default domain signals the browser to try again by slapping www. and .com on each end. Wouldn’t Verizon’s DNS misuse actually break this browser functionality by resolving the first attempt to their own search engine?
I think this actually goes beyond the impact of VeriSign’s DNS misuse, because at least by the time a DNS request gets to VeriSign’s servers, the browser had already queried the local DNS without the .com suffix.