Quad-A blocking in DNS

Hugo Salgado - NIC Chile - hsalgado@nic.cl - July 2011.

Summary
In these days of transition to IPv6, one of the problems that has been detected is the slowness in the load time of some websites. The analysis detected that is caused by a phenomenon called "Quad-A blocking in DNS", which causes delay in the resolution step of a name in the users that have IPv6 connectivity, even when the website only have IPv4. The sites with this error have a problem of configuration in one of their DNS servers, which causes the lack of answer for queries of AAAA type (Quad-A). Even when there's no evidence of the root cause of this problem, there's some data that points to some firewall implementations, IDS or load balancers doing query filtering, only allowing the response of known records types, and discarding everything else. The recommendation is to review all the name servers its behavior for AAAA type queries, even although still no use services in IPv6, because will the customers who it begin to use, and sent consultations DNS by these records in all the sites web that visit.

Analysis
The analysis indicate that the domains with problems are doing load balancing through DNS. The fact that there's a right answer for A records, and some fingerprinting checks, suggests that the DNS servers are adequate, but some kind of middleware in front of them are doing filtering, discarding the queries or the answers when the type of records are different to A. The worst problem is that although these services do not implement IPv6 and by thus never have AAAA records, their customers are the ones that will begin to use IPv6, and who will be affected its experience to connect to these sites. A browser that is in a appliance with IPv6 connectivity will attempt to get both A and AAAA records of any website, and the lack of response will cause a delay, retries, and finally a timeout that can achieve several seconds before connecting definitely by IPv4. In the following figure is a sample of an attempt of resolution of website "www.canal13.cl". In the window of the left is a sample of a DNS query with the tool "dig", directed to one of the authoritative servers that has to answer for the name (after of a chain CNAME redirection), while in the right is, at the same time, the output of network packets with the attempts of resolution, and the answers when there's any:

Is observed that a query for the A record answer correctly in less than 1 second, and a query for the AAAA record cause 3 attempts of answer with 5 seconds of delay between each one, causing a "timeout" after 15 seconds. The delay can be increased in a cascade effect if the objects of the webpage reside in different names in the same domain, because the sequence of resolution is applied to each one of them. The same website indicated in the previous example have javascript and stylesheets resources in two othe different names, so the load of the page takes fortyfive seconds in Firefox on Linux, and twentyone seconds in Explorer on Windows Vista, both browsers with IPv6 support enabled and running on a computer with IPv6 connectivity. This problem is different to the serialized connection one called "Happy Eyeballs" 1. In that case, is assumed that one have the A and AAAA valid records, and is the IP connection to those numbers IP the has to be done in parallel, using the first that works. In our case, the Quad-A blocking occurs previous to the connection, and for example a Chrome browser in its last version, which have implemented the recommendations of Happy Eyeballs, has the same problem of delay in the resolution and so the load, finally. MacOSX in its "Lion" version has resolved this problem at operating system level, and do the queries for A and AAAA also in parallel, returning with the first that responds 2. But in Linux is the function gethostbyname of the operating system which delay until receive the AAAA answer, so there is not so much to be done by browsers above that. A special remark: this is no another problem with IPv6 transition. It's a problem of the authoritative DNS servers for some websites, which is being more evident with IPv6 than before, but affects multiple services.

1 "Happy Eyeballs: Improving User Experiences with IPv6 and SCTP" Dan Wing and Andrew Yourtchenko, Cisco Systems, The Internet Protocol Forum, http://www.Ipjforum.org/?P=378 2 http://www.apple.com/archives/IPV6-dev/2011/July/msg00009.html

Diagnosis
For each authoritative name server of the zone "example.cl", a query for a record Quad-A dig @ <server> www.example.cl AAAA can give 4 valid answers: one or more AAAA records, (2001:1398:1::6003 for example) I know the label, but there is not a record of that type (NOERROR + answers = 0) I don't know the label (NXDOMAIN) II don't understand the question (RCODE!= NOERROR)

but all in a typical DNS response time (~ <200 ms). The standard RFC 1035 Section 4.1.1, indicates that if a server do not understand a question or do not understand the format, it must deliver a "Not Implemented" or "Format Error", but in no case fail in an answer. This wrong operation was already documented in the RFC 4074 "Common Misbehaviour Against DNS Queries for IPv 6 Addresses" dated 2005.

Measurements
Incidence in websites
We took a random sample of 5% of the domain names under .CL in 29 July 2011, which represents 15,162 domains, over which we run our analysis code. Of the sample, 20 websites reported this problem, which is 0.13% of total. Extrapolating, it would be close to 400 websites incorrect under .CL On 16 September of 2011 we executed the test on the full .CL domain, throwing 392 domains with problems, that correlates with the estimate before. By design of the test, is possible that there are other domain names with problems in one of their servers of name.The test gives positive if all the servers of name are incorrect. Its important to note that the problem of configuration is in the name server, not in the web server, so correcting a same DNS can solve the problem of more of a website that share this same service. But for effects of damage to the end client, it is the websites that are affected finally.

Browser timeout:
Linux 2.6.35 (Fedora 14): 15 seconds, tested with Firefox, Chrome and Opera. In Linux is confirmed that is gethostbyname() who causes the problem, when is used with IPv6 compatibility. Windows Vista: 7 seconds, tested with Explorer, Firefox, Chrome MacOSX Lion: there's no timeout, tested with Safari.

Tools
For testing analysis in browsers and operating systems, there's a name server in an test domain that does not answer for AAAA record queries, causing in the practice a timeout of resolution: www.test-aaaa-block.cl There's a simple diagnosis tool for a website, where a url indicated is tested against its name servers looking for timeout: http://www.test-aaaa-block.cl