From Fedora Project Wiki
mNo edit summary
mNo edit summary
 
(8 intermediate revisions by 2 users not shown)
Line 1: Line 1:
= Default podman created images to zstd:chunked =
= Default podman created images to zstd:chunked =


{{Change_Proposal_Banner}}
{{Change_Rejected_Banner}}


== Summary ==
== Summary ==


Modify `podman push`, `buildah push` and `skopeo copy` to default to creating images with `zstd:chunked`. Users can modify `containers.conf` to continue to use the legacy `gzip` image format.
Modify `podman push`, `buildah push` and `skopeo copy` to default to creating container images compressed with `zstd:chunked`. Users can modify `containers.conf` to continue to use the legacy `gzip` image format.


== Owner ==
== Owner ==
Line 22: Line 22:
[[Category:SelfContainedChange]]
[[Category:SelfContainedChange]]


* Targeted release: [https://docs.fedoraproject.org/en-US/releases/f41/ Fedora Linux 41]
* Targeted release: [https://docs.fedoraproject.org/en-US/releases/fxx/ Fedora Linux xx]
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}  
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
Line 30: Line 30:
ON_QA -> change is fully code complete
ON_QA -> change is fully code complete
-->
-->
* [<will be assigned by the Wrangler> devel thread]
* [https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/ULPLKTES265ZZSBVWSFPKR6NZTOLULHR/ Announced]
* FESCo issue: <will be assigned by the Wrangler>
* [https://discussion.fedoraproject.org/t/f41-change-proposal-default-podman-created-images-to-zstd-chunked-self-contained/125540 discussion thread]
* FESCo issue: [https://pagure.io/fesco/issue/3252 #3252]
* Tracker bug: <will be assigned by the Wrangler>
* Tracker bug: <will be assigned by the Wrangler>
* Release notes tracker: <will be assigned by the Wrangler>
* Release notes tracker: <will be assigned by the Wrangler>
Line 40: Line 41:


== Feedback ==
== Feedback ==
It is not possible to directly push container images with both compression formats. See:
* https://github.com/moby/moby/issues/28394#issuecomment-1822741770
* https://github.com/opencontainers/image-spec/issues/803#issuecomment-745335277
Instead, it should be possible to create a manifest list with containers in both formats. See the discussion in https://discussion.fedoraproject.org/t/switch-fedora-container-images-to-support-zstd-chunked-format-by-default/123712 for more details.


This change was pushed by mistake to Fedora 41 and Fedora 40 with:
This change was pushed by mistake to Fedora 41 and Fedora 40 with:
Line 51: Line 59:
To avoid further disruptions, this change has not been reverted in Fedora 41 yet. It will be reverted if it is rejected.
To avoid further disruptions, this change has not been reverted in Fedora 41 yet. It will be reverted if it is rejected.


This change also impacts all Bootable Container variants of Fedora. Initial (non-optimized) support for zstd:chunked images is available in rpm-ostree since [https://github.com/coreos/rpm-ostree/releases/tag/v2024.6 v2024.6].
This change also impacts all Bootable Container variants of Fedora. Initial (non-optimized) support for `zstd:chunked` images is available in rpm-ostree since [https://github.com/coreos/rpm-ostree/releases/tag/v2024.6 v2024.6].


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 63: Line 71:
* Other developers:
* Other developers:
** Test if their containers still work with the new format.
** Test if their containers still work with the new format.
* Release engineering: [https://pagure.io/releng/issues #Releng issue number]
* Release engineering: [https://pagure.io/releng/issues #Releng issue number] (To Do)
** Verify that the Fedora infra can use `zstd:chunked` container images.
** Verify that the Fedora infra can use `zstd:chunked` container images.
* Policies and guidelines: N/A (not needed for this Change)
* Policies and guidelines: N/A (not needed for this Change)
Line 81: Line 89:
This change is implemented by updating the `/usr/share/containers/containers.conf` configuration file to set `compression_format = "zstd:chunked"` by default.
This change is implemented by updating the `/usr/share/containers/containers.conf` configuration file to set `compression_format = "zstd:chunked"` by default.


Users would want to keep the existing format can enforce it with the following commands:
Users that would like to keep using the existing format can enforce it with the following commands:


  $ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/
  $ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/
Line 137: Line 145:


If we find issues with `zstd:chunked` while turned on in Rawhide, we can easily reverse direction and go back to the original default `gzip` format.
If we find issues with `zstd:chunked` while turned on in Rawhide, we can easily reverse direction and go back to the original default `gzip` format.
All the container images and tags pushed with the `zstd:chunked` format will have to be pushed again with the `gzip` compression.


== Documentation ==
== Documentation ==

Latest revision as of 15:23, 17 October 2024

Default podman created images to zstd:chunked

This is a rejected Change proposal for Fedora Linux.
This document represents a proposed Change that has been rejected or withdrawn as part of the Changes process. This page is preserved for historical record.

Summary

Modify podman push, buildah push and skopeo copy to default to creating container images compressed with zstd:chunked. Users can modify containers.conf to continue to use the legacy gzip image format.

Owner

Current status

Detailed Description

The goal is to speed up the pulling of container images, via using zstd:chunked. zstd:chunked format has shown itself to speed up the pulling of images by as much as 90%. This change has been waiting for change for over 3 years, waiting for Docker to support the format.

Feedback

It is not possible to directly push container images with both compression formats. See:

Instead, it should be possible to create a manifest list with containers in both formats. See the discussion in https://discussion.fedoraproject.org/t/switch-fedora-container-images-to-support-zstd-chunked-format-by-default/123712 for more details.

This change was pushed by mistake to Fedora 41 and Fedora 40 with:

This change has been reverted for Fedora 40 in:

To avoid further disruptions, this change has not been reverted in Fedora 41 yet. It will be reverted if it is rejected.

This change also impacts all Bootable Container variants of Fedora. Initial (non-optimized) support for zstd:chunked images is available in rpm-ostree since v2024.6.

Benefit to Fedora

Fedora would be the first distribution to support building zstd:chunked format by default. Container engines (Podman, Buildah, Skopeo, CRI-O) which support the format natively would see a great speed up in pulling images. Other container engines including Docker and Containerd would see some speed up do the the compression algorithms of zstd, but not to the same degree as container engines with full support.

Scope

  • Proposal owners: Dan Walsh, Giuseppe Scrivano
    • Switch the default to zstd:chunked
  • Other developers:
    • Test if their containers still work with the new format.
  • Release engineering: #Releng issue number (To Do)
    • Verify that the Fedora infra can use zstd:chunked container images.
  • Policies and guidelines: N/A (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with Community Initiatives: N/A

Upgrade/compatibility impact

The following versions of each package have support for zstd:chunked:

  • podman 5.1
  • buildah 1.36
  • skopeo 1.15

Moby doesn't support zstd:chunked but it handles the container image as a standard zstd layer since Moby 23.0. containerd supports zstd since 1.5.

This change is implemented by updating the /usr/share/containers/containers.conf configuration file to set compression_format = "zstd:chunked" by default.

Users that would like to keep using the existing format can enforce it with the following commands:

$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/
$ echo 'compression_format = "gzip"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf

How To Test

To test this change, you can create the following config file:

$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/
$ echo 'compression_format = "zstd:chunked"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf

See: https://github.com/containers/common/blob/main/docs/containers.conf.5.md

You can also use the --compression-format=zstd:chunked option with podman push or the --dest-compress-format=zstd:chunked with skopeo.

See: https://docs.podman.io/en/latest/markdown/podman-push.1.html

--compression-format=gzip | zstd | zstd:chunked
Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. The default is gzip unless overridden in the containers.conf file.

--compression-level=level
Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted 
values are in the range 1-20 (inclusive) with a default of 3, while for gzip it is 1-9 (inclusive) and has a default of 5.

See: https://github.com/containers/skopeo/blob/main/docs/skopeo-copy.1.md

--dest-compress-format format

Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked.

--dest-compress-level format

Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive), while for gzip it is 1-9 (inclusive).

Then:

  • Build and push container images to container registries.
  • Inspect them to check that their format is zstd:chunked.
  • Pull the image back to the host.
  • Now make a small change to the image and push it back to the registry.
  • Remove the changed image leaving the original image in place.
  • Now pull the updated image again.
  • You should see a big change in download speed.
  • Now attempt to pull the same image with Docker/Moby to make sure the image works with those container engines.

User Experience

Users should experience a nice improvement in download speeds of container images that were previously created with zstd:chunked.

Dependencies

N/A

Contingency Plan

If we find issues with zstd:chunked while turned on in Rawhide, we can easily reverse direction and go back to the original default gzip format.

All the container images and tags pushed with the zstd:chunked format will have to be pushed again with the gzip compression.

Documentation

N/A (not a System Wide Change)

Release Notes

Containers are pushed to container registries using the zstd:chunked format.