Changes/Drop Delta RPMs

Drop Delta RPMs

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

Stop producing Delta RPMs during the compose process, and disable deltarpm support in the default configuration of DNF / DNF5.

Owner

• Name: Fabio Valentini
• Email: decathorpe@gmail.com

Current status

• Targeted release: Fedora Linux 40
• Last updated: 2023-09-11
• [<will be assigned by the Wrangler> devel thread]
• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

Delta RPMs are an optimization that reduces the amount of data that needs to be downloaded for package updates, by only downloading the parts of the package that have changed between the locally installed version and the new version, and reassembling a complete RPM package for the new version locally.

Due to the way they are generated during the compose process, it is not possible to produce Delta RPMs for all packages that are involved in an upgrade (since only one previous version is considered for delta generation). This can lead to upgrades that involve hundreds of packages, but only a tiny fraction of them (or none at all) also have appropriate Delta RPMs available in the repository.

Additionally, reconstruction of the new version of the RPM from the currently installed version and the Delta RPM can fail, causing an additional download of the complete RPM for the new version, wasting bandwidth.

On top of these issues, the presence of Delta RPMs in package repositories also inflates the size of the repository metadata, which needs to be downloaded by all users, whether the actual upgrade involves Delta RPMs or not. This most affects users that upgrade via GUI utilities like GNOME Software - because the PackageKit libdnf backend has no support for Delta RPMs at all, or users of OSTree based variants / spins of Fedora - where Delta RPMs are not used either.

According to early feedback from Release Engineering, it looks like it will not be feasible to address the shortcomings of Delta RPMs as they are currently implemented, and since they often no longer result in a net reduction in download sizes for upgrades, this Change proposes both to disable the generation of DeltaRPMs during the compose process, and to disable the deltarpm support in dnf / dnf5 by default.

Some statistics

I have collected data for system upgrades with dnf across three different Fedora installs (Fedora 38 Workstation, Fedora 37 KDE, and Fedora 38 KDE, respectively, all with the "updates-testing" repository enabled) for two months, with varying intervals between upgrades (between one day and one month) in an attempt to capture different situations. To summarize the results:

• Delta RPMs were available for only 25 out of 42 upgrades
• Delta RPMs saved 7.5 MB / 8% of downloads on average
• Delta RPMs saved 22.5 MB of download on average (if there were no failures)
• Delta RPMs wasted 52.7 MB of download on average (if there were failures)

As an anecdote, most of the savings are attributable to one package (firefox).

For reference, the current sizes of repository metadata for Fedora 38 (as of 2023-09-07):

• fedora: 86.8 MB
• updates: 33.2 MB
• updates-testing: 13.3 MB

Note that refreshing repository metadata only causes deltas to be downloaded, not a full re-download (due to using zchunk for repodata).

Feedback

N/A

Benefit to Fedora

Benefits for Fedora infrastucture:

• simplification of the compose process for "updates" and "updates-testing" repositories (generation of Delta RPMs will be skipped)
• reduction of storage requirements in Fedora infrastructure and on repository mirrors (both due to smaller metadata and dropped Delta RPMs)
• reduction in bandwith use for repository metadata updates

Benefits for Fedora users:

• more reliable upgrades (i.e. failures to re-assemble RPMs from Delta RPMs are eliminated)
• reduction in bandwith use for repository metadata updates
• reduction in CPU usage for package upgrades (local re-assembly of RPMs from on-disk data and Delta RPMs is eliminated)

Scope

• Proposal owners:
  • modify compose configuration to skip generation of Delta RPMs for Fedora 40+
  • modify default configuration for dnf / dnf5 to disable deltarpm support
• Other developers:
  • coordination with dnf and dnf5 developers and / or package maintainers
• Release engineering: releng#11665
  • merge compose changes to skip generation of Delta RPMs for Fedora 40+
• Policies and guidelines: N/A (not needed for this Change)
• Trademark approval: N/A (not needed for this Change)
• Alignment with Community Initiatives: N/A (no currently active initiatives)

Upgrade/compatibility impact

Installations that get upgraded to Fedora 40 should not be negatively affected by this Change. The configuration change for dnf / dnf5 will only be automatically applied if there were no modifications to /etc/dnf/dnf.conf. However, since the repositories will no longer contain Delta RPMs, this configuration change is not strictly required.

Alternatively, the default dnf / dnf5 configuration could be changed at the code level instead of changes to configuration files.

How To Test

Users should be able to verify that support for Delta RPMs is disabled in dnf on Fedora 40, for example by checking for deltarpm = 0 in the output of dnf config-manager --dump | grep deltarpm.

User Experience

Users who upgrade their systems with GNOME Software (or other similar tools) should not be affected by this change, other than faster and smaller repository metadata downloads.

Users who upgrade via dnf / dnf5 should see a similar improvement. Additionally, there will be no more wasted downloads due to failed reassembly of RPMs from Delta RPMs.

Dependencies

• Release Engineering changes (modifications of the compose process)
• dnf / dnf5 package changes (disabling deltarpm support, either via configuration changes, or changing the default from true to false in the code itself)

Contingency Plan

• Contingency mechanism:

If the compose configuration cannot be modified to skip generation of Delta RPMs in time for Fedora 40, changing the default configuration in dnf / dnf5 will still cause some of the benefits outlined above (except for those caused by the smaller repository metadata sizes).

If the default dnf configuration cannot be changed in time for Fedora 40, the benefits outlined above should still apply in almost all circumstances.

If neither changes can be implemented in time, the change can be postponed to the next Fedora release without causing problems.

• Contingency deadline: Fedora 40 Beta Freeze

• Blocks release? No

Documentation

N/A

Release Notes

As of Fedora 40, package repositories no longer contain Delta RPMs, and package managers (dnf, dnf5) have disabled support for deltarpms by default. They are not useful in many common circumstances (i.e. upgrades via GNOME Software or on OSTree based variants), and often no longer even cause less data to be downloaded for upgrades.