Changes/FsVerityRPM

Enable fs-verity in RPM

Summary

Enable the use of fsverity for installed RPM files validation.

Owners

• Name: Davide Cavalca, Boris Burkov, Filipe Brandenburger, Michel Alexandre Salim, Matthew Almond
• Email: dcavalca@fb.com, borisb@fb.com, filbranden@fb.com, michel@fb.com, malmond@fb.com

Current status

• Targeted release: Fedora Linux 36
• Last updated: 2021-12-01
• FESCo issue: tdb
• Tracker bug: tbd
• Release notes tracker: tbd

Detailed description

fs-verity is a Linux kernel feature that does transparent on-demand integrity/authenticity verification of the contents of read-only files, using a hidden Merkle tree (hash tree) associated with the file. The mechanism is similar to dm-verity, but implemented at the file level rather than at the block device level.

When fsverity is enabled for a file, the kernel reads every block and generates a hash tree for it, which is stored within the filesystem. On subsequent reads, the kernel computes the block hash and compares it with the one stored in the tree, protecting against alterations and corruption. Because this happens at the filesystem data block read layer, it encompasses all file operations (open, mmap,exec, etc.).

In the context of rpm, there are two parts to this:

• at build time, we compute the Merkle tree for the files within a package, then sign it and ship it as part of the rpm metadata
• at run time, if the fsverity rpm plugin is enabled, rpm will install the fsverity signature key and enable fsverity on files that are installed

Note that enabling the fsverity rpm plugin by default is explicitly out of scope for this proposal.

Caveats

Merkle tree cost

The Merkle tree used by fsverity needs to be generated (once at build time, once when the package is installed) and stored on disk. The generation process involved reading all blocks and computing the hash, which has a non-trivial cost; however, it does not appear to meaningfully slow down package installs during empirical testing. Once generated, the Merkle tree will use up some disk space for its storage (about 1/127th of the original file size). Note that the Merkle tree is not shipped with the RPM itself (only its signature is) and is only generated at install time if the fsverity rpm plugin is enabled. Hence, there is no cost (neither in generation time nor in disk space usage) if the plugin is disabled.

Signature overhead cost

To leverage fsverity every rpm needs to include the hash signature as part of its metadata, which will increase its size. The signature size is roughly proportional to the number of files in the package. From empirical testing, in the vast majority of cases we expect to see minimal to no size increase thanks to RPM header packing.

Relationship with IMA

IMA is another technology meant to provide detection of file alterations. IMA and fsverity operate very differently, and are somewhat complementary.

fs-verity works by using a Merkle tree to generate a checksum for every data block in the system, and reads will fail if a single data block read fails it’s checksum. The signature of the the file is validated against a public key loaded into the kernel keyring. Because fsverity operates on block reads, its runtime cost is small (as it only needs to verify the block that is being accessed), and it can protect from alterations at any point in time.

IMA works by measuring a file as a whole and comparing its signature whenever it’s read of executed. It has a higher runtime cost than fsverity (as it needs to verify the whole file at once) and it cannot detect subsequent alternations. IMA provides a much more rich and complex policy system, allowing one to define system-wide policies around trusted files that tie into LSMs such as SELinux.

IMA and fsverity could potentially be integrated (meaning, an fsverity backend for IMA could be implemented to leverage its policy controls), but this is not currently planned or being worked on.

Relationship with native checksums

By default, btrfs already checksums each file extent, which could potentially be leveraged to implement a HMAC solution. This currently exists as a patch series but it hasn’t been merged yet. Similarly to IMA, we see this approach as complementary to fs-verity. The blog post goes into more details of the tradeoffs involved.

Feedback

Pending the devel discussion

Benefit to Fedora

The main benefit is the ability to do block-level verification of RPM-installed files. In turn, this can be used to implement usecase-specific validation and verification policies depending on the environment requirements.

Scope

• Proposal owners
  • btrfs kernel enablement work (landed in 5.15); see this blog post for more details
  • koji integration: koji will need to add the fs-verity metadata to packages when signing them
• Other developers:
  • deploy the koji integration changes to production
• Release engineering: tbd
• Policies and guidelines: N/A
• Trademark approval: N/A

Upgrade/compatibility impact

None

How to test

Install the fs-verity RPM plugin to validate package contents:

$ sudo dnf install rpm-plugin-fsverity

Note that this will only be useful if the packages being installed contain the appropriate fs-verity metadata (which, for Fedora upstream packages, requires Koji integration that is part of this Change). However, you should still be able to test this if you locally sign a package with rpmsign --addverity.

User experience

This Change is fully transparent and there is no user impact by default. If the user chooses to enable the fs-verity RPM plugin, they can then leverage the additional verification features provided by fs-verity.

Dependencies

• fs-verity support is available in RPM as of 4.17, which is available as of Fedora 35 and is already enabled in rpm-4.17.0-0.rc1.0.fc36
• CONFIG_FS_VERITY in the kernel config; this is already enabled
• fs-verity requires filesystem support; currently support for ext4 and f2fs is already available; support for btrfs landed in 5.15
• there is no filesystem dependency on the builders, only at runtime (and only if the rpm fsverity plugin is installed and one wishes to use it)

Contingency plan

Revert the changes to koji.

Documentation

• https://www.kernel.org/doc/html/latest/filesystems/fsverity.html
• https://developers.facebook.com/blog/post/2021/10/19/fs-verity-support-in-btrfs/
• The proposal owners plan to document the fsverity plugin and integration in RPM (https://github.com/rpm-software-management/rpm/issues/1849)

Release Notes

The RPM package manager now supports validation of file contents using fs-verity.