Changes/systemd-resolved

systemd-resolved

Summary

Enable systemd-resolved by default. glibc will perform name resolution using nss-resolve rather than nss-dns.

Owners

• Name: Michael Catanzaro
• Email: <mcatanzaro@redhat.com>
• Name: Zbigniew Jędrzejewski-Szmek
• Email: <zbyszek@in.waw.pl>

Current status

• Targeted release: Fedora 33
• Last updated: 2021-05-15
• FESCo issue: #2381
• Tracker bug: #1834846
• Release notes tracker: #507

Detailed Description

We will enable systemd-resolved by default.

1. We will change the fedora-release presets to enable systemd-resolved instead of disable it.
2. systemd-libs currently has a %post scriplet to enable nss-myhostname and nss-systemd by either (a) modifying authselect's user-nsswitch.conf template, if authselect is in use, or (b) directly modifying /etc/nsswitch.conf otherwise. We will work with the systemd maintainers to enable nss-resolve here as well by adding resolve [!UNAVAIL=return] to the hosts line.
3. We will work with the authselect maintainers to update authselect's minimal and nis profiles to enforce nss-resolve. These profiles modify the hosts line of /etc/nsswitch.conf. (By default, Fedora uses authselect's sssd profile, which does not modify the hosts line and therefore does not have this problem.)
4. We will remove our downstream patch to systemd that prevents systemd from symlinking /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf on new installs. For system upgrades, a systemd-libs %post scriptlet will be used to reassign the symlink during upgrade. Upon detecting this symlink, NetworkManager will automatically enable its systemd-resolved support and configure split DNS as appropriate.

systemd-resolved has been enabled by default in Ubuntu since Ubuntu 16.10, but please note we are doing this differently than Ubuntu has. Ubuntu does not use nss-resolve. Instead, Ubuntu uses the traditional nss-dns provided by glibc upstream, so glibc on Ubuntu continues to read /etc/resolv.conf, as is traditional. This extra step is not useful and not recommended by upstream. We want to follow upstream recommendations in using nss-resolve instead.

If you do not wish to use systemd-resolved, then manual intervention will be required to disable it:

• Disable and stop systemd-resolved.service.
• Delete /etc/resolv.conf
• Restart the NetworkManager service. NetworkManager will then create a traditional /etc/resolv.conf. (If you are not using NetworkManager, you may create your own /etc/resolv.conf and configure it to your liking.)

Benefit to Fedora

Standardization

Fedora will continue its history of enabling new systemd-provided services whenever it makes sense to do so. Standardizing on upstream systemd services is beneficial to the broader Linux ecosystem in addition to Fedora, since standardizing reduces behavior differences between different Linux distributions. Sadly, Fedora is no longer leading in this area. Ubuntu has enabled systemd-resolved by default since Ubuntu 16.10, so by the time Fedora 33 is released, we will be three years behind Ubuntu here.

resolvectl

resolvectl has several useful functions (e.g. resolvectl status or resolvectl query) that will be enabled out-of-the-box.

Caching

systemd-resolved caches DNS queries for a short while. This can dramatically improve performance for applications that do not already manually cache their own DNS results. (Generally, only web browsers bother with manually caching DNS results.)

Split DNS

When systemd-resolved is enabled, users who use VPNs will notice that DNS requests are now sent to the correct DNS server by default. Previously, using a VPN that receives traffic only for resources on its network would result in embarrassing "DNS leaks." If multiple VPNs are in use, then, depending on the order that the VPN connections were established, it could result in failure to resolve private resources. These scenarios will now work properly. For example, consider the scenario of Distrustful Denise, who (quite reasonably) does not trust her ISP. Denise uses a public VPN service, public-vpn.example.com, to hide her internet activity from her ISP, but she also needs to use her employer's corporate VPN, corporation.example.com, in order to access internal company resources while working from home. Using the Network panel in System Settings, Denise has configured her employer's VPN to "use this connection only for resources on its network." Distrustful Denise expects that her employer's VPN will receive all traffic for corporation.example.com, and no other traffic. And while this mostly works in Fedora 32, she discovers that it does not work properly for DNS:

• If Denise connects to public-vpn.example.com first and corporation.example.com second, she is unable to access internal company resources. All DNS requests are sent to public-vpn.example.com's DNS server, so she is unable to resolve names for internal company websites.
• If Denise connects to corporation.example.com first and public-vpn.example.com second, then she is able to access internal company resources. However, it only works because all her DNS requests are sent to corporation.example.com's DNS server. Sadly for Distrustful Denise, her employer discovers that she has been making some embarrassing DNS requests that she had expected to go through public-vpn.example.com instead.

Whichever VPN Denise connects to first receives all DNS requests because glibc's nss-dns module is not smart enough to split the requests. /etc/resolv.conf is just a list of DNS servers to try in order, and NetworkManager has no plausible way to decide which to list first, because both ways are broken, so it just prefers whichever was connected first. And if one server fails to respond, then the next server in the list will be tried, likely resulting in a DNS leak. In contrast, when systemd-resolved is enabled, it will send each DNS request only to the correct DNS server. The DNS server that will be used for each tun interface can be inspected using the resolvectl tool. These benefits are not specific to users who use multiple VPNs at the same time; they apply even if only the corporate VPN is in use, since systemd-resolved will ensure requests outside the corporate network are not sent to the corporate DNS server. (If, however, the VPN was configured to receive all traffic, then it would of course receive all the DNS requests. systemd-resolved will send the DNS requests to whichever network will actually receive the traffic.)

Migrating away from /etc/resolv.conf will also avoid an annoying footgun with this legacy file: only the first three listed nameservers are respected. All further nameservers are silently ignored. NetworkManager adds a warning comment when writing more than three nameservers to this file, but it cannot do any better than that.

DNS over TLS

systemd-resolved supports DNS over TLS (different from DNS over HTTPS). Although this feature will not initially be enabled by default, using systemd-resolved will enable us to turn on DNS over TLS in a future Fedora release, providing improved security if the user's DNS server supports DNS over TLS.

Scope

• Proposal owners: We will update Fedora presets to enable systemd-resolved by default (#127, #133).

• Other developers: This change requires coordination with the systemd (PR) and authselect maintainers.

• Release engineering: #9367

• Policies and guidelines: none required

• Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

systemd-resolved will be enabled automatically when upgrading to Fedora 33. After upgrade, /etc/resolv.conf will be managed by systemd and symlinked to /run/systemd/resolve/stub-resolv.conf. glibc will no longer look at /etc/resolv.conf when performing name resolution. Instead, glibc will communicate directly with systemd-resolved via nss-resolve. systemd adds a large warning comment to the top of /etc/resolv.conf to warn system administrators that changes to this file will be ignored; however, scripts that edit this file manually will break. Because this file is usually managed by NetworkManager, impact to Fedora users will be limited to users who have manually disabled NetworkManager; such users are expected to be experienced system administrators who should be comfortable adapting to the change (or disabling systemd-resolved).

Any applications that bypass glibc and read /etc/resolv.conf directly will still work because /etc/resolv.conf will point to systemd-resolved's stub resolver running on 127.0.0.53. See next subsection.

In short, applications that read /etc/resolv.conf will continue to work as before. Applications that write to it will no longer work as expected, but this only previously worked if NetworkManager is disabled, a non-default configuration. It remains possible to disable systemd-resolved if desired. It is also possible to continue using a traditional /etc/resolv.conf with systemd-resolved by removing the symlink and creating a traditional /etc/resolv.conf as a regular file. In this case, systemd-resolved will respect the traditional configuration in /etc/resolv.conf. Any custom system administration scripts that manage /etc/resolv.conf will need to manually remove the symlink and recreate /etc/resolv.conf as a regular file, or they will not work as expected.

A slightly different case is applications or users which want to access the "real" list of remote servers, for example for debugging. In the past they could look at /etc/resolv.conf. That file will now only contain the address of the local stub server. The full list of servers may be queried over D-Bus (similarly to what resolvectl dns does). A flattened list of DNS servers is also available in /run/systemd/resolve/resolv.conf when systemd-resolved is running.

Local stub resolver on port 53

systemd-resolved provides a stub resolver listening on the address 127.0.0.53:53 on the loopback interface. Only processes running on the local machine may connect. This allows processes which bypass the glibc nss stack and connect to a resolver of UDP or TCP to talk to systemd-resolved. Nevertheless, /etc/resolv.conf is provided only for compatibility purposes, and applications should prefer to use either glibc or the systemd-resolved D-Bus API instead; see systemd-resolved(8) for details.

If a different server should listen on port 53 (for example dnsmasq), the stub may be disabled by creating

 # /etc/systemd/resolved.conf.d/stub-listener.conf
 [Resolve]
 DNSStubListener=no

DNSSEC

systemd-resolved's DNSSEC support is known to cause compatibility problems with certain network access points (Reference #1, Reference #2). Per recommendation from the systemd developers, we will change the default value of this setting in Fedora from the upstream default DNSSEC=allow-downgrade to DNSSEC=no by building systemd with the build option -Ddefault-dnssec=no. The upstream default attempts to use DNSSEC if it is working, but automatically disable it otherwise, allowing man-in-the-middle attackers to disable DNSSEC. Sadly, even the allow-downgrade setting suffers known compatibility problems. Because Fedora is not prepared to handle an influx of DNSSEC-related bug reports, we will disable this feature altogether. We anticipate that enabling DNSSEC by default will not be possible in the foreseeable future, or perhaps ever. Instead, enabling DNS over TLS (when supported by the DNS server) seems likely in the near future.

Local multicast DNS

systemd-resolved's full MulticastDNS (mDNS) support conflicts with Avahi. Per recommendation from the systemd developers, we had originally planned to change the default setting in Fedora from the upstream default MulticastDNS=yes to MulticastDNS=resolve. Multicast DNS resolving would have been enabled, but responding would have been disabled. This was to be done using a new systemd build option to control the default value of the MulticastDNS setting, similar to the existing default-dnssec and default-dns-over-tls build options. However, due to implementation issues, we instead will use MulticastDNS=no and allow avahi to continue handling mDNS.

Regardless, the default LLMNR mode will be set to LLMNR=resolve.

Opting out of /etc/resolv.conf that points to the localhost stub resolver

Simply recreate the file with different contents or pointing somewhere else. In particular it may be useful to point to /run/systemd/resolve/resolv.conf, which contains a flattened list of servers used by systemd-resolved:

 sudo ln -sfv /run/systemd/resolve/resolv.conf /etc/resolv.conf

Fully opting out of systemd-resolved use

Before the upgrade, create a local preset for systemd-resolved.service:

 sudo bash -c 'mkdir -p /etc/systemd/system-preset && echo "disable systemd-resolved.service" >/etc/systemd/system-preset/20-systemd-resolved-disable.preset'

How To Test

Load any website in a web browser. If you succeed, then name resolution probably works.

Try using resolvectl status and, for example, resolvectl query fedoraproject.org, to see how they work and sanity-check their output.

Users who use multiple VPNs at the same time are encouraged to test DNS in a multiple VPN scenario, to ensure that DNS requests are sent to the expected DNS server.

User Experience

See the Benefit to Fedora section, above, for direct benefits to users who use VPNs.

Users will be able to use the resolvectl tool and the functionality it provides.

/etc/resolv.conf will now be managed by systemd rather than by NetworkManager. As before, this file must not be modified directly when it is managed.

Dependencies

In Fedora, /etc/nsswitch.conf is managed by authselect. By default, authselect uses the sssd profile to generate configuration compatible with sssd. In this mode of operation, it does not modify the hosts line in /etc/nsswitch.conf. This is also true if using the winbind profile instead of the sssd profile. However, authselect's minimal and nis profiles do modify the hosts line. These authselect profiles must be updated to enable nss-resolved. If you are using authselect in one of these modes, it will not be possible to cleanly disable systemd-resolved because the hosts line in /etc/nsswitch.conf will be clobbered whenever 'authselect apply-changes' is run. If you wish to disable systemd-resolved and you are using authselect in one of these modes, then you should stop using authselect. This is not expected to cause many problems because virtually all Fedora users will be using the default sssd profile.

We do not need to directly make any changes to the /etc/nsswitch.conf shipped by glibc. Changes will be applied in the systemd-libs %post scriptlet.

Contingency Plan

The contingency plan, in the unlikely event of unexpected problems, is simply to revert any changes and not enable systemd-resolved.

• Contingency deadline: beta freeze
• Blocks release? No
• Blocks product? No

Documentation

• systemd-resolved is documented in several manpages: resolvectl(1), resolved.conf(5), nss-resolve(8), systemd-resolved(8).
• Arch Wiki documentation
• NetworkManager DNS documentation

See also

Previous attempt to add a caching resolver in F24: Changes/Default_Local_DNS_Resolver

Release Notes

systemd-resolved is now enabled by default. systemd-resolved provides a system-level DNS cache that can substantially improve performance for applications that do not cache their own DNS results, allows correct handling of split DNS scenarios such as when VPNs are in use, and will allow Fedora to enable DNS over TLS in the future. /etc/resolv.conf will now be managed by systemd-resolved rather than by NetworkManager. /etc/resolv.conf will no longer be read when performing name resolution using glibc; however, it is still provided for compatibility with applications that manually read this file to perform name resolution. Writing to /etc/resolv.conf will no longer work as expected.