From Fedora Project Wiki
(Created page with "= Switch RPMs to zstd compression = == Summary == Binary RPMs are currently compressed with xz level 2. Switching to zstd would increase decompression speed significantly. =...")
 
 
(7 intermediate revisions by 3 users not shown)
Line 26: Line 26:
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
-->
-->
* Tracker bug: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1720729 #1720729]
* Release notes tracker: <will be assigned by the Wrangler>
* Release notes tracker: [https://pagure.io/fedora-docs/release-notes/issue/350 #350]


== Detailed Description ==
== Detailed Description ==
Line 92: Line 92:
* Proposal owners: submit a patch to redhat-rpm-config
* Proposal owners: submit a patch to redhat-rpm-config
* Other developers: redhat-rpm-config maintainer: include the patch and make a new build
* Other developers: redhat-rpm-config maintainer: include the patch and make a new build
* Release engineering: [https://pagure.io/releng/issues #Releng issue number] mass rebuild is needed
* Release engineering: [https://pagure.io/releng/issue/8395 #8395] mass rebuild is needed
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
Line 116: Line 116:


== Contingency Plan ==
== Contingency Plan ==
* Contingency mechanism: Not needed, Fedora will stay at current compression.
* Contingency mechanism: Revert to the previous compression and do a new mass rebuild.
* Contingency deadline: N/A
* Contingency deadline: 2019-10-01 (a week before the Final Freeze)
* Blocks release? No
* Blocks release? Yes
* Blocks product? N/A
* Blocks product? N/A


Line 129: Line 129:
Users that build their packages will experience slightly longer build times.
Users that build their packages will experience slightly longer build times.


[[Category:ChangePageIncomplete]]
Users using older systems (even RHEL8 era) may not be able to inspect these Fedora31 RPMs with those versions of rpm / rpm2cpio.
 
[[Category:ChangeAcceptedF31]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
Line 136: Line 138:


<!-- Select proper category, default is Self Contained Change -->
<!-- Select proper category, default is Self Contained Change -->
[[Category:SelfContainedChange]]
[[Category:SystemWideChange]]
<!-- [[Category:SystemWideChange]] -->

Latest revision as of 18:27, 26 March 2020

Switch RPMs to zstd compression

Summary

Binary RPMs are currently compressed with xz level 2. Switching to zstd would increase decompression speed significantly.

Owner

Current status

Detailed Description

  • The change requires setting a new compression algorithm in rpm macros. Then a mass rebuild of all packages is required.
  • The macro for setting the compression is: %define _binary_payload w19.zstdio
  • The recommended compression level is 19. The builds will take longer, but the additional compression time is negligible in the total build time and it pays off in better compression ratio than xz lvl2 has.
  • SRPM payload compression should stay at gzip (there's almost no benefit in changing the compression, because SRPM's contents is compressed already)

Use case: Firefox installation

I rebuilt firefox-66.0.5-1.fc30 with zstd level19. Then I compared installation times with the original (xz compressed) package:

Compression Target File System Time
xz level 2 tmpfs 8s
xz level 2 ext4 on nvme 11s
zstd level 19 tmpfs 2s
zstd level 19 ext4 on nvme 4s


Comparison of compression algorithms and levels

Following table shows cpio and compressed cpio extraction times into a tmpfs. Actual times in decompressing RPMs will differ due to extracting on an actual disk and also some overhead in the RPM tool (checks, scriptlets).

Compression Level Size B Size GiB Compression time Compression time, 4 threads Decompression time Comment
CPIO - 5016785692 4,7 - - -
xz 2 1615017616 1,6 9m55s - 1m36s slow decompression
pxz 2 1631869880 1,6 - 6m11s 1m38s slow decompression
gzip 9 2086354992 2,0 10m23s - 31s insufficient compression ratio
bzip2 9 1889161565 1,8 8m - 2m50s very slow decompression; compression ratio could be better
zstd 3 1913536587 1,8 31s 29s 6,5s
zstd 10 1737928978 1,7 3m27s 2m34s 6,3s
zstd 15 1717303256 1,7 9m37s 6m34s 6,3s identical compression speed to xz; fast decompression; slightly worse compression ratio than xz
zstd 17 1635525492 1,6 16m16s 11m20s 6,7s
zstd 19 1575843696 1,5 24m2s 18m55s 7,7s

Benefit to Fedora

  • Faster installations/upgrades of user systems
  • Faster koji builds (installations in build roots)
  • Faster container builds
  • Lower bandwidth on mirrors if we choose the highest compression level

Scope

  • Proposal owners: submit a patch to redhat-rpm-config
  • Other developers: redhat-rpm-config maintainer: include the patch and make a new build
  • Release engineering: #8395 mass rebuild is needed
  • Policies and guidelines: not needed for this Change
  • Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

  • RPM in Fedora supports zstd compression already (from Fedora 28, rpm-4.14.0-0.rc2.5.fc28). No impact on Fedora users is expected.
  • Fedora <= 27 and some other distros will not be able to decompress zstd-compressed RPMs.

How To Test

  • dnf install <package>
  • rpm -q --qf "%{PAYLOADCOMPRESSOR} %{PAYLOADFLAGS}\n" <package>
  • expected output: zstd 19

Also the overall system installation time should decrease significantly.

User Experience

See Benefit to Fedora

Dependencies

N/A

Contingency Plan

  • Contingency mechanism: Revert to the previous compression and do a new mass rebuild.
  • Contingency deadline: 2019-10-01 (a week before the Final Freeze)
  • Blocks release? Yes
  • Blocks product? N/A

Documentation

N/A

Release Notes

RPMs have switched to zstd compression level 19. Users will benefit from faster package decompression. Users that build their packages will experience slightly longer build times.

Users using older systems (even RHEL8 era) may not be able to inspect these Fedora31 RPMs with those versions of rpm / rpm2cpio.

Retrieved from "https://fedoraproject.org/w/index.php?title=Changes/Switch_RPMs_to_zstd_compression&oldid=569610"