Enable systemd service hardening features for default system services

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

Improve security by enabling some of the high level systemd security hardening settings that isolate and sandbox default system services.

Owner

Name: Rahul Sundaram
Email: metherid@gmail.com

Targeted release: Fedora 40
Last updated: 2023-12-04
Announced
Discussion thread
FESCo issue: #3117
Tracker bug: <will be assigned by the Wrangler>
Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

systemd provides a number of settings that can harden security for services. We are selecting a few high level ones to enable by default on a service by service basis as suitable for that particular service.

PrivateTmp=yes
ProtectSystem=yes/full/strict
ProtectHome=yes/read-only
ProtectClock=yes
ProtectHostname=yes
ProtectControlGroups=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectProc=invisible
PrivateDevices=yes
PrivateNetwork=yes
NoNewPrivileges=yes
User=

If we want to go further, we could also consider:

CapabilityBoundingSet=
DevicePolicy=closed
KeyringMode=private
LockPersonality=yes
MemoryDenyWriteExecute=yes
PrivateUsers=yes
RemoveIPC=yes
RestrictAddressFamilies=
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
SystemCallFilter=
SystemCallArchitectures=native

We will aim to cover all the default system services as well as some of the high profile services such as Nginx or PostgreSQL. All of these settings need to be configured on a per service basis instead of using a global override to facilitate fine tuning the settings based on service requirements and limit the impact for users on upgrades. Certain services have a very targeted scope. For instance, a service that only needs to read or write from only one directory could leverage more fine grained settings to restrict access even further. We will enable as many of these as feasible for the services but not every knob is going to be applicable to every service. For example, PrivateNetwork=yes can only be used for services that does not need network connectivity by default. We have to choose between DynamicUser=yes or User if either is feasible for the service to use. As a base starting point, from Fedora 39 workstation, we have the following system services installed by default which should considered within the scope of the change (excluding systemd associated ones which already have a number of these security settings enabled). We may also consider doing this for some of the high profile services including say Nginx and PostgreSQL permitting time considerations and other contributors if any joining this effort. We will prioritize critical or long running services.

abrtd.service
abrt-journal-core.service
abrt-oops.service
abrt-pstoreoops.service
abrt-vmcore.service
abrt-xorg.service
accounts-daemon.service
alsa-restore.service
alsa-state.service
anaconda-direct.service
anaconda-fips.service
anaconda-nm-config.service
anaconda-nm-disable-autocons.service
anaconda-noshell.service
anaconda-pre.service
anaconda.service
anaconda-sshd.service
arp-ethers.service
auditd.service
auth-rpcgss-module.service
avahi-daemon.service
blivet.service
blk-availability.service
bluetooth.service
bolt.service
brltty.service
canberra-system-bootup.service
canberra-system-shutdown-reboot.service
canberra-system-shutdown.service
chronyd-restricted.service
chronyd.service
chrony-wait.service
colord.service
console-getty.service
cups-browsed.service
cups.service
dbus-broker.service
dbus-daemon.service
dbus-org.freedesktop.hostname1.service
dbus-org.freedesktop.import1.service
dbus-org.freedesktop.locale1.service
dbus-org.freedesktop.login1.service
dbus-org.freedesktop.machine1.service
dbus-org.freedesktop.portable1.service
dbus-org.freedesktop.timedate1.service
~~debug-shell.service~~ (opens a user shell that must be able to do arbitrary stuff)
dm-event.service
dnf-makecache.service
dnf-system-upgrade-cleanup.service
dnf-system-upgrade.service
dnsmasq.service
dracut-cmdline.service
dracut-initqueue.service
dracut-mount.service
dracut-pre-mount.service
dracut-pre-pivot.service
dracut-pre-trigger.service
dracut-pre-udev.service
dracut-shutdown-onfailure.service
dracut-shutdown.service
~~emergency.service~~ (opens a user shell that must be able to do arbitrary stuff)
fedora-third-party-refresh.service
firewalld.service
flatpak-add-fedora-repos.service
flatpak-system-helper.service
fprintd.service
fsidd.service
fstrim.service
fwupd-offline-update.service
fwupd-refresh.service
fwupd.service
gdm.service
geoclue.service
grub-boot-indeterminate.service
gssproxy.service
htcacheclean.service
httpd.service
hypervfcopyd.service
hypervkvpd.service
hypervvssd.service
iio-sensor-proxy.service
import-state.service
initrd-cleanup.service
initrd-parse-etc.service
initrd-switch-root.service
initrd-udevadm-cleanup-db.service
instperf.service
ipp-usb.service
iscsid.service
iscsi-init.service
iscsi-onboot.service
iscsi.service
iscsi-shutdown.service
iscsi-starter.service
iscsiuio.service
kdump.service
kmod-static-nodes.service
ldconfig.service
libvirtd.service
libvirt-guests.service
livesys-late.service
livesys.service
loadmodules.service
logrotate.service
low-memory-monitor.service
lvm2-lvmdbusd.service
lvm2-lvmpolld.service
lvm2-monitor.service
man-db-cache-update.service
man-db-restart-cache-update.service
mcelog.service
mdcheck_continue.service
mdcheck_start.service
mdmonitor-oneshot.service
mdmonitor.service
ModemManager.service
ndctl-monitor.service
netavark-dhcp-proxy.service
NetworkManager-dispatcher.service
NetworkManager.service
NetworkManager-wait-online.service
nfs-blkmap.service
nfsdcld.service
nfs-idmapd.service
nfs-mountd.service
nfs-server.service
nfs-utils.service
nftables.service
nis-domainname.service
nm-priv-helper.service
numad.service
nvmefc-boot-connections.service
nvmf-autoconnect.service
ostree-boot-complete.service
ostree-finalize-staged-hold.service
ostree-finalize-staged.service
ostree-prepare-root.service
ostree-remount.service
packagekit-offline-update.service
packagekit.service
pam_namespace.service
pcscd.service
plocate-updatedb.service
plymouth-halt.service
plymouth-kexec.service
plymouth-poweroff.service
plymouth-quit.service
plymouth-quit-wait.service
plymouth-read-write.service
plymouth-reboot.service
plymouth-start.service
plymouth-switch-root-initramfs.service
plymouth-switch-root.service
podman-auto-update.service
podman-clean-transient.service
podman-restart.service
podman.service
polkit.service
power-profiles-daemon.service
psacct.service
qemu-guest-agent.service
qemu-pr-helper.service
quotaon.service
raid-check.service
~~rc-local.service~~ (this can do arbitrary stuff)
realmd.service
rescue.service
rpcbind.service
rpc-gssd.service
rpc-statd-notify.service
rpc-statd.service
rpmdb-migrate.service
rpmdb-rebuild.service
rtkit-daemon.service
saslauthd.service
selinux-autorelabel-mark.service
selinux-autorelabel.service
selinux-check-proper-disable.service
speech-dispatcherd.service
spice-vdagentd.service
spice-webdavd.service
sshd.service
ssh-host-keys-migration.service
sssd-autofs.service
sssd-kcm.service
sssd-nss.service
sssd-pac.service
sssd-pam.service
sssd.service
sssd-ssh.service
sssd-sudo.service
switcheroo-control.service
system-update-cleanup.service
tcsd.service
thermald.service
udisks2.service
unbound-anchor.service
upower.service
uresourced.service
usbmuxd.service
vboxclient.service
vboxservice.service
vgauthd.service
virtinterfaced.service
virtlockd.service
virtlogd.service
virtnetworkd.service
virtnodedevd.service
virtnwfilterd.service
virtproxyd.service
virtqemud.service
virtsecretd.service
virtstoraged.service
vmtoolsd.service
wpa_supplicant.service
zfs-fuse-scrub.service
zfs-fuse.service
zvbid.service

For a concrete example, Nginx in Fedora uses only PrivateTmp because of https://fedoraproject.org/wiki/Features/ServicesPrivateTmp implemented in early 2012.

https://src.fedoraproject.org/rpms/nginx/blob/rawhide/f/nginx.service#_19

Over the decade since then, systemd has introduced a large number of additional directives. We could consider the following changes there:

[Service]

LockPersonality=true

MemoryDenyWriteExecute=true

NoNewPrivileges=true

PrivateDevices=true

ProtectClock=true

ProtectControlGroups=true

ProtectHostname=true

ProtectKernelLogs=true

ProtectKernelModules=true

ProtectKernelTunables=true

ProtectProc=invisible

ProtectSystem=full

RestrictRealtime=true

RestrictSUIDSGID=true

SystemCallArchitectures=native

Feedback

Updated the upstreaming guidance to take into account minimum supported version of systemd based on feedback in https://discussion.fedoraproject.org/t/96423/2. Daniel still feels that these changes are better done upstream exclusively. Others noted that Fedora does enable a number of compiler flags and additional security features including SELinux by default and systemd sandboxing features can follow that pattern. Package maintainers should be encouraged to contribute these changes upstream. IMO, however Fedora should be "First" and adopt these "Features" to be true to it's mission. Fedora shouldn't limit itself to passively following whatever upstream happens to include as many may not even include a systemd service file and do not enable the vast majority of these features even when they do. Fedora is better positioned to provide more comprehensive coverage of these features by default given that Fedora always included the very latest systemd releases by default and act as an integration point for newer systemd sandboxing features.
Added a concrete example in the form of Nginx based on feedback in https://discussion.fedoraproject.org/t/96423/11 and followup at https://discussion.fedoraproject.org/t/96423/18 reiterated that all the settings will not be applicable to all the services.
There was a suggestion to user drop-in config snippets instead of changing the service files directly to make the hardening settings readily visible at https://discussion.fedoraproject.org/t/96423/6 and another suggestion to do it in /usr/lib since Fedora already follows that pattern in https://discussion.fedoraproject.org/t/96423/8
There was some discussions about scope and I have added my rationale at https://discussion.fedoraproject.org/t/96423/15
There was some discussions on updating the packaging guidelines and making the changes advertised well. I have proposed some initial draft for both the packaging guidelines and release notes, both of which will evolve as we firm up our approach (drop-in vs direct service changes etc).
Systemd does not support a general mechanism of resetting a directive back to default by setting it to an empty value. You must instead explicitly set the value depending on the setting and this was noted in https://discussion.fedoraproject.org/t/96423/17

Open questions for FESCo:

Mechanism: Do we support adding the idea of using more of systemd security settings by default and recommending them in the packaging guidelines? What specific mechanism should we use to accomplish this a) per service settings as proposed b) default overrides with per service opt outs ex: default override can set ProtectClock=yes but services like Chronyd will specifically disable this (Fedora uses this mechanism to set a timeout via /usr/lib/systemd/system/service.d/10-timeout-abort.conf c) drop-ins in /usr/lib (as Fedora already uses in some cases)
Scope: Is limiting the scope to default system services in this release the right approach? Do we also want to target other high profile services like Nginx and PostgreSQL that are not included by default?
Impact: How should we should consider the impact for RHEL 10 since it is expected to be based on Fedora 40?

Benefit to Fedora

Fedora services will get a significant security boost by default by avoiding or mitigating any unknown security vulnerabilities in default system services. Since Fedora will include the very latest version of systemd and other components and has the visibility and control of the default configuration of the services, it can go well beyond what upstream can support directly based on their minimum version of systemd. Since Fedora already has the reputation of being security focused (SELinux enabled by default, system wide compiler flags that enable a number of security features etc), it is in a good position to act as a coordination and integration point.

It can be the first mainstream distribution that enables more of these systemd hardening features by default and push that upstream wherever feasible. This serves the first, features and friends part of the Fedora mission respectively.

Scope

Proposal owners: Individual per service pull requests to enable various security features as applicable.
Other developers: Review PRs as needed
Release engineering: https://pagure.io/releng/issue/11785
Policies and guidelines:

Packaging guidelines will have to be modified to add recommendations to use more of the systemd security features by default. In particular, we should add a security settings section in https://fedoraproject.org/wiki/Packaging:Systemd. Current the guidance only recommends a couple of settings for long running services. Sample text:

Systemd services included in Fedora are recommended to use as many of the following security settings as applicable while maintaining the default functionality of the service.

PrivateTmp=yes
ProtectSystem=yes/full/strict
ProtectHome=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
NoNewPrivileges=yes
PrivateNetwork=yes

The full list of sandboxing features are available in https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#Sandboxing. Note that if you are submitting changes to upstream as recommended in https://docs.fedoraproject.org/en-US/packaging-guidelines/PatchUpstreamStatus/, systemd will warn and ignore any of these features it doesn't support. So while the service itself won't break, these warnings can add to the support burden. Please take into account the minimum required version of systemd that upstream supports and only include those settings or provide build system logic to conditionally build the default unit file when submitting these patches upstream. The specific version of systemd required for any of these settings is documented in the systemd exec man page.

Trademark approval: N/A

Upgrade/compatibility impact

Packages will automatically get additional security features enabled by default transparently. In limited circumstances, they may need to override the defaults. Refer to user experience section for details.

How To Test

You can use tools like systemd-analyze security and systemctl cat to verify that specific security features are enabled by default. Default services with the default features should have no adverse impact and users shouldn't have to do anything beyond using the software as intended and report any regressions. High profile services not installed by default that gain these security features would benefit from more targeting testing to spot any unintended consequences especially for niche or advanced functionality. If advanced non-default functionality requires overrides default settings, we can document those in the release notes to provide guidance.

User Experience

This should be largely transparent change for users. The goal is to have the services work as expected with the default functionality but to potentially require tweaking the settings if the configuration is changed by users after installation. For instance, if we add ProtectHome=yes to Apache httpd.service and the user wishes to serve files out of their home directory, they will need to override the systemd setting to ProtectHome=read-only to allow for the service to read from the user home directory in addition to changing the service specific configuration files to enable this feature.

Dependencies

None. We are merely enabling some of systemd security features by default for default system services and potentially some high profile services.

Contingency Plan

Contingency mechanism: These settings can be enabled/disabled at a per service level. No wholesale reverts is necessary. If we don't finish the work for all the services, we can follow up in future releases.
Contingency deadline: N/A
Blocks release? No

Documentation

Release Notes

systemd security hardening features are enabled for default system services and following high profile services.

PostgreSQL
Apache Httpd
Nginx
MariaDB

....

If you wish to turn off any particular settings, you can follow the standard systemd method of overriding the config. For example,

$ cat /etc/systemd/system/httpd.service.d/override.conf

[Service]

ProtectHome=no

$ sudo systemctl daemon-reload

$ sudo systemctl restart httpd.service

$ systemctl status httpd.service

● httpd.service - The Apache HTTP Server

    Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
   Drop-In: /etc/systemd/system/httpd.service.d
            └─override.conf
    Active: active (running) since Mon 2023-11-15 18:29:25 EST; 3min 30s ago

Search

Changes/SystemdSecurityHardening

Contents