Changes/Zchunk Metadata

Zchunk Metadata

Summary

All dnf repository metadata will be compressed with the zchunk format in addition to xz or gzip.

Owner

• Name: Jonathan Dieter, Neal Gompa
• Email: jdieter@gmail.com, ngompa13@gmail.com
• Release notes owner:

Current status

• Targeted release: Fedora 30
• Last updated: 2019-02-25
• Tracker bug: #1601595
• Release notes tracker: #285

Detailed Description

Currently Fedora's repository metadata is compressed using the xz and gzip formats. Zchunk is a new compression format designed to allow for highly efficient deltas. When Fedora's metadata is compressed using zchunk, dnf will download only the differences between any earlier copies of the metadata and the current version.

Benefit to Fedora

Dnf and related tools will see significant reductions in the size of the metadata they download, especially if they are run on a regular basis.

Scope

• Proposal owners:
  • Package zchunk for Fedora - Done, see https://bugzilla.redhat.com/show_bug.cgi?id=1597287
  • Get the pull requests to enable zchunk in dnf, libdnf, librepo, libsolv and createrepo_c merged upstream
    • https://github.com/rpm-software-management/librepo/pull/127
    • https://github.com/openSUSE/libsolv/pull/270
    • https://github.com/rpm-software-management/dnf/pull/1107 No longer needed
    • https://github.com/rpm-software-management/libdnf/pull/478
    • https://github.com/rpm-software-management/createrepo_c/pull/92
  • Create a new package for Fedora's zchunk dictionaries - See https://bugzilla.redhat.com/show_bug.cgi?id=1656627

• Other developers:
  • Fedora Infrastructure needs to start creating zchunked metadata Generated since 2018-12-27
  • Build zchunk-enabled libdnf (see PR at https://github.com/rpm-software-management/libdnf/pull/663)
  • Build librepo with patch to properly calculate download size for zchunk metadata (see PR at https://github.com/rpm-software-management/librepo/pull/138)
  • Build createrepo_c with patch to generate zchunked metadata for all records (see PR at https://github.com/rpm-software-management/createrepo_c/pull/115)

• Release engineering: #7600
  • List of deliverables: Zchunk repository metadata

• Policies and guidelines: Packaging guidelines are not affected by this change.

• Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

Old-style xz and gzip metadata will continue to be generated by createrepo_c, so old versions of Fedora will use the xz/gzip metadata while upgrading to the new version, and then use the new zchunk metadata after the upgrade.

How To Test

Check that zchunk metadata is being downloaded:

1. Remove /var/cache/dnf/*
2. Run dnf update
3. Check that the files in /var/cache/dnf/rawhide-*/repodata have .zck extensions

Check that zchunk metadata won't be redownloaded when it matches the current metadata:

1. Remove /var/cache/dnf/rawhide-*/repodata/repomd.xml
2. Run dnf update
3. Check that the repodata download size matches the size of just repomd.xml

Check that only different chunks are being downloaded:

1. Run dnf update
2. Wait 24-48 hours
3. Run dnf update again
4. Validate that per-repository metadata download is less than 60MB (should be less than 5MB)

User Experience

This change will allow Fedora users to see significant reductions in the size of the metadata they download. If they check for updates daily, they should see a savings of up to 95%.

Dependencies

Upstream needs to merge the zchunk pull requests, and Fedora 29's dnf, libdnf, librepo, libsolv and createrepo_c will need to pull the patches in.

Contingency Plan

• Contingency mechanism:
  • If there are problems generating zchunked metadata, Fedora Infrastructure will need to disable zchunk metadata generation
  • If there are problems with the zchunk dnf/libdnf integration, we will:
    • First disable the downloading of zchunk metadata
    • If that doesn't fix it, we will revert the zchunk patches
• Contingency deadline: 2019-03-05 (Beta Freeze)
• Blocks release? No, as long as the contingency is enacted
• Blocks product? No

Documentation

Introduction to zchunk
What is zchunk and how does it work?
How to use the zchunk utilities
The zchunk file format