QA/Networking

Testing IPv6 support and dualstack networking can be very tricky. The purpose of this document is to provide a live resource on testing situations arising from existance and usage of two network layer protocols.

Tracker bug dependency tree.

Configuration

IPv4 and IPv6 connectivity

Applications may behave differently based on connectivity checks. Those can be performed using the following per protocol characteristics.

Existence of any IP address (including 127.0.0.1 or ::1
Existence of non-loopback IP address (including IPv6 link-local addresses)
Existence of a global IP address
Existence of a default route (or route for the target we want to reach)

Example: To test application behavior on a host connected via IPv6-only network you should make sure you only have the loopback IPv4 address and that you have a global IPv6 address and a default IPv6 gateway as that what you would normally call an IPv6-only setup. Similarly for IPv4-only setup you would make sure that there are only IPv6 loopback and link-local addresses and that there is a global IPv4 address and a default IPv4 gateway. But there are still many corner cases that may trigger unexpected issues.

Target DNS records

You need to carefully choose a target hostname for dual-stack networking tests.

Examples of scenarios:

Hostname with a single A record or a single AAAA record
Hostname with one A record and one AAAA record
Hostname with multiple records per family, some of which are obsolete and inacceessible

Other DNS records typically point to names, not addresses and thus are protocol agnostic but it may still be useful to test e.g. SRV records so that the SRV code path is correct as well.

IPv4 and IPv6 recursive DNS servers

This is entirely orthogonal to IPv4 and IPv6 DNS records and doesn't cause many issues.

IPv4 and IPv6 authoritative DNS servers

This is a rather special case important for domain operators. A bad configuration can make otherwise working IPv6 hosts inaccessible from IPv6-only networks with their own recursive servers.

Name resolution (using libc functions)

Name resolution features are provided by the GNU C Library (glibc) which is not yet ready for proper IPv6 and dual-stack operation as you can see when performing your tests. The C library comes with its own testing tool getent that has a special database called ahosts that runs getaddrinfo(), the library function that translates names to objects with addressing information. For your testing it is best used together with tools like strace, ltrace or even gdb so that you know exactly what is happening behind the scenes.

As the getent tool is very primitive, we created a tool called getaddrinfo just as the library function that handles a larger subset of the function's API.

Name resolution input

When application requests addressing information for a hostname with an optional service name, the library returns a list of addressing information objects. The order of objects in the list is significant and depends on operating system configuration and connectivity.

From the application

nodename
servname
protocol
socktype
flags
- AI_CANONNAME
- ...

From local configuration and connectivity checks

Files in /etc/ including nsswitch, hosts, services and more
To what extend is IPv4 and IPv6 available

From the outside world

DNS information
Multicast DNS information
LDAP information

Name resolution processing

What is requested

Not all information is requested at all times. Some information like canonical name must be explicitly requested by the application via AI_CANONNAME flag. It may be desirable to suppress other requests by local configuration or connectivity checks, a notable example being suppression of DNS AAAA queries on hosts without global connectivity.

What is passed to the application

Not all information that is learnt via requests is presented to the application. It is typically filtered according to input from the application. It is sometimes also filtered according to connectivity checks but that has caused more problems than improvements.

How it is sorted

There are rules for sorting addressing information returned by getaddrinfo(). One of the basic features is to return global IPv6 destinations before global IPv4 destinations. But when the library detects that IPv6 connectivity is not available, the reverse applies.

Example tests

Dual-stack host, destination with global IPv4 and IPv6

On a host with IPv4 and IPv6 connectivity we request addressing information of another host that is announced as dual-stack in DNS.

$ getent ahosts www.nix.cz
2a02:38::1001   STREAM info.nix.cz
2a02:38::1001   DGRAM  
2a02:38::1001   RAW    
195.47.235.3    STREAM 
195.47.235.3    DGRAM  
195.47.235.3    RAW

You can see that getaddrinfo() returned six items for two unique IP addresses with the IPv6 address sorted first. If you only expected only two items, one for each IP address, see upstream bug 14990.

IPv4-only host, destination with global IPv4 and IPv6

We do the same on a host without IPv6 connectivity.

$ getent ahosts www.nix.cz
195.47.235.3    STREAM info.nix.cz
195.47.235.3    DGRAM  
195.47.235.3    RAW    
2a02:38::1001   STREAM 
2a02:38::1001   DGRAM  
2a02:38::1001   RAW

You can see that the result is the same as before except that IPv4 is sorted first.

Name resolution (using netresolve)

There is an experimental package called netresolve (TODO: not yet in Fedora) that consists of a library somewhat similar to the glibc name resolution API implementation and a set of debugging tools. Any application using libc API and a couple of other APIs can be run using wrapresolve to use the netresolve implementation instead and benefit from some advanced features and more extensive debugging. When using the libc backend, netresolve can be also used to test the glibc implementation. With the nss backend it can be used to test glibc nsswitch backends directly.

$ ./netresolve --node www.nix.cz
response netresolve 0.0.1
name info.nix.cz
ip 2a02:38::1001 any any 0 0 0 21599
ip 195.47.235.3 any any 0 0 0 12589

You can see that netresolve behaves slightly different than getent ahosts and it by default returns one item per IP address. But you can easily tweak it to behave the same way.

 
$ ./netresolve --node www.nix.cz --service ''
response netresolve 0.0.1
name info.nix.cz
ip 2a02:38::1001 stream tcp 0 0 0 21021
ip 2a02:38::1001 dgram udp 0 0 0 21021
ip 2a02:38::1001 raw any 0 0 0 21021
ip 195.47.235.3 stream tcp 0 0 0 12592
ip 195.47.235.3 dgram udp 0 0 0 12592
ip 195.47.235.3 raw any 0 0 0 12592

Note that netresolve is both the name of the command and the name of the library. The above tests are done using the command but the same results would be given to an application using the library. Unless you explicitly request, netresolve uses its internal name resolution modules and not the libc functions.

Tools

To be merged into section Packages...

Tools with issues:

Testing client applications

Client application checklist

A client application should:

avoid using obsolete functions:
- gethostbyname(), gethostbyname2()
- gethostbyaddr()
- ...
use system wide name resolution configuration
- avoid implementing name resolution code in the application
- avoid calling DNS libraries directly (there are other name resolution sources than DNS)
reduce timeouts when a fallback is available
- reduce excessive DNS timeouts, possibly using a decent system wide name resolution library
- reduce excessive TCP timeouts, possibly using a decent socket library
attempt all provided addresses until one succeeds or all fail
- possibly using a decent socket library

Testing server applications

Server applications should be capable of accepting IPv4 and IPv6 clients and logging their addresses. Some server applications make use of addresses extensively, some just need to translate them to a meaningful name. Many server applications initiate connections to other hosts and thus the section about client applications applies to them as well.

Packages

Packages that are expected to play well with IPv6

Following packages play more or less well with IPv6. You can still expect minor issues including delays when configuration of one of the protocols doesn't work as expected. Those packages are still to be tested.

Network configuration:

dhcp – Dynamic host configuration protocol software
iproute – Advanced IP routing and network device configuration tools
NetworkManager – Network connection manager and user applications
dnssec-trigger – NetworkManager plugin to update/reconfigure DNSSEC resolving

Server software:

bind – The Berkeley Internet Name Domain (BIND) DNS (Domain Name System) server
dnsmasq – A lightweight DHCP/caching DNS server
httpd – Apache HTTP Server
openssh – An open source implementation of SSH protocol versions 1 and 2

Other:

curl – A utility for getting files from remote servers (FTP, HTTP, and others)
cyrus-imapd – A high-performance mail server with IMAP, POP3, NNTP and SIEVE support
c-ares – A library that performs asynchronous DNS operations
firefox – Mozilla Firefox Web browser
libssh2 – A library implementing the SSH2 protocol
iptables – Tools for managing Linux kernel packet filtering capabilities

Packages with broken or incomplete IPv6 support

Core system:

avahi – Local network service discovery
- IPv6 turned off by default
- nss-mdns doesn't do IPv6LL due to glibc limitations
glibc – The GNU libc libraries
- limited and buggy getaddrinfo()
- upstream getaddrinfo issues

Core tools:

iputils – Network monitoring tools including ping
- ping (unreleased upstream now supports dual-stack operation)
- tracepath
- ...
quota – File system quota management
- rpc.rquotad and other program should be ported from glibc's Sun RPC to tirpc's RPC
  - User:Pavlix: Why and how is it related to IPv6?

Development:

subversion - A Modern Concurrent Version Control System
- no fallback to IPv4 when IPv6 default route fails

Packages that are intended for IPv4-only operation

Following packages do not need any IPv6 testing as they contain IPv4 specific tools.

arptables
arpwatch

Notable packages to be tested for IPv6 support

Following packages weren't categorized yet but are considered important enough to be checked for IPv6 support. Please extend the list with other high visibility packages.