Changes/SystemdSecurityHardening

Enable systemd service hardening features for default system services

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

Improve security by enabling some of the high level systemd security hardening settings that isolate and sandbox default system services.

Owner

• Name: Rahul Sundaram
• Email: metherid@gmail.com

• Targeted release: Fedora 40
• Last updated: 2023-12-03
• Announced
• Discussion thread
• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

systemd provides a number of settings that can harden security for services. We are selecting a few high level ones to enable by default on a service by service basis as suitable for that particular service.

• PrivateTmp=yes
• ProtectSystem=yes/full/strict
• ProtectHome=yes/read-only
• ProtectClock=yes
• ProtectHostname=yes
• ProtectControlGroups=yes
• ProtectHostname=yes
• ProtectKernelLogs=yes
• ProtectKernelModules=yes
• ProtectKernelTunables=yes
• ProtectProc=invisible
• PrivateDevices=yes
• PrivateNetwork=yes
• NoNewPrivileges=yes
• User=

If we want to go further, we could also consider:

• CapabilityBoundingSet=
• DevicePolicy=closed
• KeyringMode=private
• LockPersonality=yes
• MemoryDenyWriteExecute=yes
• PrivateUsers=yes
• RemoveIPC=yes
• RestrictAddressFamilies=
• RestrictNamespaces=yes
• RestrictRealtime=yes
• RestrictSUIDSGID=yes
• SystemCallFilter=
• SystemCallArchitectures=native

We will aim to cover all the default system services as well as some of the high profile services such as Nginx or PostgreSQL. All of these settings need to be configured on a per service basis instead of using a global override to facilitate fine tuning the settings based on service requirements and limit the impact for users on upgrades. Certain services have a very targeted scope. For instance, a service that only needs to read or write from only one directory could leverage more fine grained settings to restrict access even further. We will enable as many of these as feasible for the services but not every knob is going to be applicable to every service. For example, PrivateNetwork=yes can only be used for services that does not need network connectivity by default. We have to choose between DynamicUser=yes or User if either is feasible for the service to use. As a base starting point, from Fedora 39 workstation, we have the following system services installed by default which should considered within the scope of the change (excluding systemd associated ones which already have a number of these security settings enabled). We may also consider doing this for some of the high profile services including say Nginx and PostgreSQL permitting time considerations and other contributors if any joining this effort. We will prioritize critical or long running services.

• abrtd.service
• abrt-journal-core.service
• abrt-oops.service
• abrt-pstoreoops.service
• abrt-vmcore.service
• abrt-xorg.service
• accounts-daemon.service
• alsa-restore.service
• alsa-state.service
• anaconda-direct.service
• anaconda-fips.service
• anaconda-nm-config.service
• anaconda-nm-disable-autocons.service
• anaconda-noshell.service
• anaconda-pre.service
• anaconda.service
• anaconda-sshd.service
• arp-ethers.service
• auditd.service
• auth-rpcgss-module.service
• avahi-daemon.service
• blivet.service
• blk-availability.service
• bluetooth.service
• bolt.service
• brltty.service
• canberra-system-bootup.service
• canberra-system-shutdown-reboot.service
• canberra-system-shutdown.service
• chronyd-restricted.service
• chronyd.service
• chrony-wait.service
• colord.service
• console-getty.service
• cups-browsed.service
• cups.service
• dbus-broker.service
• dbus-daemon.service
• dbus-org.freedesktop.hostname1.service
• dbus-org.freedesktop.import1.service
• dbus-org.freedesktop.locale1.service
• dbus-org.freedesktop.login1.service
• dbus-org.freedesktop.machine1.service
• dbus-org.freedesktop.portable1.service
• dbus-org.freedesktop.timedate1.service
• debug-shell.service (opens a user shell that must be able to do arbitrary stuff)
• dm-event.service
• dnf-makecache.service
• dnf-system-upgrade-cleanup.service
• dnf-system-upgrade.service
• dnsmasq.service
• dracut-cmdline.service
• dracut-initqueue.service
• dracut-mount.service
• dracut-pre-mount.service
• dracut-pre-pivot.service
• dracut-pre-trigger.service
• dracut-pre-udev.service
• dracut-shutdown-onfailure.service
• dracut-shutdown.service
• emergency.service (opens a user shell that must be able to do arbitrary stuff)
• fedora-third-party-refresh.service
• firewalld.service
• flatpak-add-fedora-repos.service
• flatpak-system-helper.service
• fprintd.service
• fsidd.service
• fstrim.service
• fwupd-offline-update.service
• fwupd-refresh.service
• fwupd.service
• gdm.service
• geoclue.service
• grub-boot-indeterminate.service
• gssproxy.service
• htcacheclean.service
• httpd.service
• hypervfcopyd.service
• hypervkvpd.service
• hypervvssd.service
• iio-sensor-proxy.service
• import-state.service
• initrd-cleanup.service
• initrd-parse-etc.service
• initrd-switch-root.service
• initrd-udevadm-cleanup-db.service
• instperf.service
• ipp-usb.service
• iscsid.service
• iscsi-init.service
• iscsi-onboot.service
• iscsi.service
• iscsi-shutdown.service
• iscsi-starter.service
• iscsiuio.service
• kdump.service
• kmod-static-nodes.service
• ldconfig.service
• libvirtd.service
• libvirt-guests.service
• livesys-late.service
• livesys.service
• loadmodules.service
• logrotate.service
• low-memory-monitor.service
• lvm2-lvmdbusd.service
• lvm2-lvmpolld.service
• lvm2-monitor.service
• man-db-cache-update.service
• man-db-restart-cache-update.service
• mcelog.service
• mdcheck_continue.service
• mdcheck_start.service
• mdmonitor-oneshot.service
• mdmonitor.service
• ModemManager.service
• ndctl-monitor.service
• netavark-dhcp-proxy.service
• NetworkManager-dispatcher.service
• NetworkManager.service
• NetworkManager-wait-online.service
• nfs-blkmap.service
• nfsdcld.service
• nfs-idmapd.service
• nfs-mountd.service
• nfs-server.service
• nfs-utils.service
• nftables.service
• nis-domainname.service
• nm-priv-helper.service
• numad.service
• nvmefc-boot-connections.service
• nvmf-autoconnect.service
• ostree-boot-complete.service
• ostree-finalize-staged-hold.service
• ostree-finalize-staged.service
• ostree-prepare-root.service
• ostree-remount.service
• packagekit-offline-update.service
• packagekit.service
• pam_namespace.service
• pcscd.service
• plocate-updatedb.service
• plymouth-halt.service
• plymouth-kexec.service
• plymouth-poweroff.service
• plymouth-quit.service
• plymouth-quit-wait.service
• plymouth-read-write.service
• plymouth-reboot.service
• plymouth-start.service
• plymouth-switch-root-initramfs.service
• plymouth-switch-root.service
• podman-auto-update.service
• podman-clean-transient.service
• podman-restart.service
• podman.service
• polkit.service
• power-profiles-daemon.service
• psacct.service
• qemu-guest-agent.service
• qemu-pr-helper.service
• quotaon.service
• raid-check.service
• rc-local.service (this can do arbitrary stuff)
• realmd.service
• rescue.service
• rpcbind.service
• rpc-gssd.service
• rpc-statd-notify.service
• rpc-statd.service
• rpmdb-migrate.service
• rpmdb-rebuild.service
• rtkit-daemon.service
• saslauthd.service
• selinux-autorelabel-mark.service
• selinux-autorelabel.service
• selinux-check-proper-disable.service
• speech-dispatcherd.service
• spice-vdagentd.service
• spice-webdavd.service
• sshd.service
• ssh-host-keys-migration.service
• sssd-autofs.service
• sssd-kcm.service
• sssd-nss.service
• sssd-pac.service
• sssd-pam.service
• sssd.service
• sssd-ssh.service
• sssd-sudo.service
• switcheroo-control.service
• system-update-cleanup.service
• tcsd.service
• thermald.service
• udisks2.service
• unbound-anchor.service
• upower.service
• uresourced.service
• usbmuxd.service
• vboxclient.service
• vboxservice.service
• vgauthd.service
• virtinterfaced.service
• virtlockd.service
• virtlogd.service
• virtnetworkd.service
• virtnodedevd.service
• virtnwfilterd.service
• virtproxyd.service
• virtqemud.service
• virtsecretd.service
• virtstoraged.service
• vmtoolsd.service
• wpa_supplicant.service
• zfs-fuse-scrub.service
• zfs-fuse.service
• zvbid.service

For a concrete example, Nginx in Fedora uses only PrivateTmp because of https://fedoraproject.org/wiki/Features/ServicesPrivateTmp implemented in early 2012.

https://src.fedoraproject.org/rpms/nginx/blob/rawhide/f/nginx.service#_19

Over the decade since then, systemd has introduced a large number of additional directives. We could consider the following changes there:

[Service]

LockPersonality=true

MemoryDenyWriteExecute=true

NoNewPrivileges=true

PrivateDevices=true

ProtectClock=true

ProtectControlGroups=true

ProtectHostname=true

ProtectKernelLogs=true

ProtectKernelModules=true

ProtectKernelTunables=true

ProtectProc=invisible

ProtectSystem=full

RestrictRealtime=true

RestrictSUIDSGID=true

SystemCallArchitectures=native

Feedback

• Updated the upstreaming guidance to take into account minimum supported version of systemd based on feedback in https://discussion.fedoraproject.org/t/96423/2. Daniel still feels that these changes are better done upstream exclusively. Others noted that Fedora does enable a number of compiler flags and additional security features including SELinux by default and systemd sandboxing features can follow that pattern. Package maintainers should be encouraged to contribute these changes upstream. IMO, however Fedora should be "First" and adopt these "Features" to be true to it's mission. Fedora shouldn't limit itself to passively following whatever upstream happens to include as many may not even include a systemd service file and do not enable the vast majority of these features even when they do. Fedora is better positioned to provide more comprehensive coverage of these features by default given that Fedora always included the very latest systemd releases by default and act as an integration point for newer systemd sandboxing features.
• Added a concrete example in the form of Nginx based on feedback in https://discussion.fedoraproject.org/t/96423/11 and followup at https://discussion.fedoraproject.org/t/96423/18 reiterated that all the settings will not be applicable to all the services.
• There was a suggestion to user drop-in config snippets instead of changing the service files directly to make the hardening settings readily visible at https://discussion.fedoraproject.org/t/96423/6 and another suggestion to do it in /usr/lib since Fedora already follows that pattern in https://discussion.fedoraproject.org/t/96423/8
• There was some discussions about scope and I have added my rationale at https://discussion.fedoraproject.org/t/96423/15
• There was some discussions on updating the packaging guidelines and making the changes advertised well. I have proposed some initial draft for both the packaging guidelines and release notes, both of which will evolve as we firm up our approach (drop-in vs direct service changes etc).
• Systemd does not support a general mechanism of resetting a directive back to default by setting it to an empty value. You must instead explicitly set the value depending on the setting and this was noted in https://discussion.fedoraproject.org/t/96423/17

Open questions for FESCo:

• Mechanism: Do we support adding the idea of using more of systemd security settings by default and recommending them in the packaging guidelines? What specific mechanism should we use to accomplish this a) per service settings as proposed b) default overrides with per service opt outs ex: default override can set ProtectClock=yes but services like Chronyd will specifically disable this (Fedora uses this mechanism to set a timeout via /usr/lib/systemd/system/service.d/10-timeout-abort.conf c) drop-ins in /usr/lib (as Fedora already uses in some cases)
• Scope: Is limiting the scope to default system services in this release the right approach? Do we also want to target other high profile services like Nginx and PostgreSQL that are not included by default?
• Impact: How should we should consider the impact for RHEL 10 since it is expected to be based on Fedora 40?

Benefit to Fedora

Fedora services will get a significant security boost by default by avoiding or mitigating any unknown security vulnerabilities in default system services. Since Fedora will include the very latest version of systemd and other components and has the visibility and control of the default configuration of the services, it can go well beyond what upstream can support directly based on their minimum version of systemd. Since Fedora already has the reputation of being security focused (SELinux enabled by default, system wide compiler flags that enable a number of security features etc), it is in a good position to act as a coordination and integration point.

It can be the first mainstream distribution that enables more of these systemd hardening features by default and push that upstream wherever feasible. This serves the first, features and friends part of the Fedora mission respectively.

Scope

• Proposal owners: Individual per service pull requests to enable various security features as applicable.
• Other developers: Review PRs as needed
• Release engineering: https://pagure.io/releng/issue/11785
• Policies and guidelines:

Packaging guidelines will have to be modified to add recommendations to use more of the systemd security features by default. In particular, we should add a security settings section in https://fedoraproject.org/wiki/Packaging:Systemd. Current the guidance only recommends a couple of settings for long running services. Sample text:

Systemd services included in Fedora are recommended to use as many of the following security settings as applicable while maintaining the default functionality of the service.

• PrivateTmp=yes
• ProtectSystem=yes/full/strict
• ProtectHome=yes
• PrivateDevices=yes
• ProtectKernelTunables=yes
• ProtectKernelModules=yes
• ProtectKernelLogs=yes
• ProtectControlGroups=yes
• NoNewPrivileges=yes
• PrivateNetwork=yes

The full list of sandboxing features are available in https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#Sandboxing. Note that if you are submitting changes to upstream as recommended in https://docs.fedoraproject.org/en-US/packaging-guidelines/PatchUpstreamStatus/, systemd will warn and ignore any of these features it doesn't support. So while the service itself won't break, these warnings can add to the support burden. Please take into account the minimum required version of systemd that upstream supports and only include those settings or provide build system logic to conditionally build the default unit file when submitting these patches upstream. The specific version of systemd required for any of these settings is documented in the systemd exec man page.

• Trademark approval: N/A

Upgrade/compatibility impact

Packages will automatically get additional security features enabled by default transparently. In limited circumstances, they may need to override the defaults. Refer to user experience section for details.

How To Test

You can use tools like systemd-analyze security and systemctl cat to verify that specific security features are enabled by default. Default services with the default features should have no adverse impact and users shouldn't have to do anything beyond using the software as intended and report any regressions. High profile services not installed by default that gain these security features would benefit from more targeting testing to spot any unintended consequences especially for niche or advanced functionality. If advanced non-default functionality requires overrides default settings, we can document those in the release notes to provide guidance.

User Experience

This should be largely transparent change for users. The goal is to have the services work as expected with the default functionality but to potentially require tweaking the settings if the configuration is changed by users after installation. For instance, if we add ProtectHome=yes to Apache httpd.service and the user wishes to serve files out of their home directory, they will need to override the systemd setting to ProtectHome=read-only to allow for the service to read from the user home directory in addition to changing the service specific configuration files to enable this feature.

Dependencies

None. We are merely enabling some of systemd security features by default for default system services and potentially some high profile services.

Contingency Plan

• Contingency mechanism: These settings can be enabled/disabled at a per service level. No wholesale reverts is necessary. If we don't finish the work for all the services, we can follow up in future releases.
• Contingency deadline: N/A
• Blocks release? No

Documentation

• https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#Sandboxing
• https://docs.arbitrary.ch/security/systemd.html
• https://www.redhat.com/sysadmin/systemd-secure-services
• https://www.redhat.com/sysadmin/mastering-systemd

Release Notes

systemd security hardening features are enabled for default system services and following high profile services.

• PostgreSQL
• Apache Httpd
• Nginx
• MariaDB

....

If you wish to turn off any particular settings, you can follow the standard systemd method of overriding the config. For example,

$ cat /etc/systemd/system/httpd.service.d/override.conf

[Service]

ProtectHome=no

$ sudo systemctl daemon-reload

$ sudo systemctl restart httpd.service

$ systemctl status httpd.service

● httpd.service - The Apache HTTP Server

    Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Drop-In: /etc/systemd/system/httpd.service.d
            └─override.conf
    Active: active (running) since Mon 2023-11-15 18:29:25 EST; 3min 30s ago