From Fedora Project Wiki


Changes of defaults in createrepo_c-1.0.0

Summary

Update createrepo_c to 1.0.0, new release will include change of default compression to zstd, no longer generating metadata in sqlite database format by default and simplified comps xml type in repodata.

Owner


Current status

Detailed Description

  • When adding groups.xml to repodata createrepo_c currently adds two variants to repomd.xml. The specified file as is, uncompressed, with the type "group" and also a compressed variant with type "group_XX", where XX is compression suffix. This is atypical and unexpected. We propose to include just one variant of groups.xml using specified compression and repomd.xml type "group". This is not compatible with yum in RHEL 7. If required users will still be able to create repositories with the old layout using modifyrepo_c.

    Further information: https://bugzilla.redhat.com/show_bug.cgi?id=2056318
  • To reflect API and overall stability increase major version above 0.

Feedback

Benefit to Fedora

  • Using zstd compression provides smaller metadata that are faster to decompress.
  • Generating sqlite metadata is expensive, its slowing createrepo_c runs and it takes up space on mirrors. (For F37 updates repo: primary_db: 12.8 mb, filelists_db: 22.0 mb, other_db: 4.2 mb.)
  • More consistent metadata, groups use the same logic as other metadata types. Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1904360

Scope

  • Proposal owners: Implement specified changes, mostly changes of default.
  • Other developers: There are no requirements on other developers.
  • Policies and guidelines: N/A (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with Objectives:

Upgrade/compatibility impact

Repodata created with createrepo_c's new defaults will not be compatible with fedora < 30 and RHEL < 8.4 (including RHEL 7, 6). This is mainly due to zstd compression but old YUM also cannot handle the changed mdtype of groups (it is compatible only with uncompressed - "group" and compressed - "group_XX" not compressed - "group").

Regarding the sqlite databases if some other users would require them they can always download the xml metadata and use sqliterepo_c to generate them locally.

How To Test

Repositories generated with updated createrepo_c have to work with dnf. Specifically we should ensure that groups are still loaded.

User Experience

The only noticeable effect should be smaller metadata thanks to zstd compression resulting in faster downloads (when zchunk is not used). The (de)compression speed up is not very noticeable.

For example for F37 fedora repo:

  • primary.xml.gz: 18M -> primary.xml.zst: 14M
  • filelists.xml.gz: 52M -> filelists.xml.zst: 41M
  • other.xml.gz: 6.9M -> other.xml.zst: 5.4M

Dependencies

Updating createrepo_c does not require any other change requests to complete first.

Metadata generated by the following tools could be affected by the described changes if they use createrepo_c's default configuration. It is hard to tell what effect will this have on the tools. If they create metadata for dnf it will work fine. In case there are any problems it is possible to manually setup the old behavior (as described above) or revert the changes. 

bodhi-server
copr-backend
libguestfs
modulemd-tools
prunerepo
pungi 
retrace-server
swid-tools
anaconda-install-env-deps
cobbler
ddiskit
koji-builder
mock
module-build
perl-Fedora-Rebuild 
python3-dlrn
python3-dnf-plugin-local
python3-dnf-plugin-modulesync
python3-rpmfluff
retrace-server

Contingency Plan

  • Contingency mechanism: If we cannot finish the changes in time there would be no action necessary. We can even ship only part of the changes as they don't depend on each other. In case there are additional unexpected issues with the changes we can revert them in createrepo_c.
  • Contingency deadline: 2023-08-01
  • Blocks release? No

Documentation

Createrepo_c documentation will be updated accordingly.

Release Notes

Retrieved from "https://fedoraproject.org/w/index.php?title=Changes/createrepo_c_1.0.0&oldid=673517"