m (Update) |
m (Update) |
||
Line 41: | Line 41: | ||
== Feedback == | == Feedback == | ||
This change was pushed by mistake to Fedora 41 and Fedora 40 with: | |||
* Fedora 41: https://bodhi.fedoraproject.org/updates/FEDORA-2024-99379626aa (to be confirmed) | |||
* Fedora 40: https://bodhi.fedoraproject.org/updates/FEDORA-2024-ab42dd0ffb | |||
This change has been reverted for Fedora 40 in: | |||
* https://bodhi.fedoraproject.org/updates/FEDORA-2024-ebe5c816a1 | |||
* https://github.com/coreos/fedora-coreos-tracker/issues/1748 | |||
To avoid further disruptions, this change has not been reverted in Fedora 41 yet. It will be reverted if it is rejected. | |||
This change also impacts all Bootable Container variants of Fedora. Initial (non-optimized) support for zstd:chunked images is available in rpm-ostree since [https://github.com/coreos/rpm-ostree/releases/tag/v2024.6 v2024.6]. | |||
== Benefit to Fedora == | == Benefit to Fedora == | ||
Line 67: | Line 77: | ||
Users would want to keep the existing format can enforce it with the following commands: | Users would want to keep the existing format can enforce it with the following commands: | ||
$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/ | |||
$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/ | $ echo 'compression_format = "gzip"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf | ||
$ echo 'compression_format = "gzip"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf | |||
== How To Test == | == How To Test == | ||
Line 76: | Line 84: | ||
To test this change, you can create the following config file: | To test this change, you can create the following config file: | ||
$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/ | |||
$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/ | $ echo 'compression_format = "zstd:chunked"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf | ||
$ echo 'compression_format = "zstd:chunked"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf | |||
See: https://github.com/containers/common/blob/main/docs/containers.conf.5.md | See: https://github.com/containers/common/blob/main/docs/containers.conf.5.md | ||
Line 87: | Line 93: | ||
See: https://docs.podman.io/en/latest/markdown/podman-push.1.html | See: https://docs.podman.io/en/latest/markdown/podman-push.1.html | ||
--compression-format=gzip | zstd | zstd:chunked | |||
--compression-format=gzip | zstd | zstd:chunked | Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. The default is gzip unless overridden in the containers.conf file. | ||
Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. The default is gzip unless overridden in the containers.conf file. | |||
--compression-level=level | |||
--compression-level=level | Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted | ||
Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive) with a default of 3, while for gzip it is 1-9 (inclusive) and has a default of 5. | values are in the range 1-20 (inclusive) with a default of 3, while for gzip it is 1-9 (inclusive) and has a default of 5. | ||
See: https://github.com/containers/skopeo/blob/main/docs/skopeo-copy.1.md | See: https://github.com/containers/skopeo/blob/main/docs/skopeo-copy.1.md | ||
--dest-compress-format format | |||
--dest-compress-format format | |||
Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. | |||
Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. | |||
--dest-compress-level format | |||
--dest-compress-level format | |||
Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive), while for gzip it is 1-9 (inclusive). | |||
Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive), while for gzip it is 1-9 (inclusive). | |||
Then: | Then: |
Revision as of 13:24, 4 July 2024
Default podman created images to zstd:chunked
Summary
Modify podman push
, buildah push
and skopeo copy
to default to creating images with zstd:chunked
. Users can modify containers.conf
to continue to use the legacy gzip
image format.
Owner
- Name: Daniel J Walsh
- Email: dwalsh@redhat.com
- Name: Giuseppe Scrivano
- Email: gscrivan@redhat.com
Current status
- Targeted release: Fedora Linux 41
- Last updated: 2024-07-04
- [<will be assigned by the Wrangler> devel thread]
- FESCo issue: <will be assigned by the Wrangler>
- Tracker bug: <will be assigned by the Wrangler>
- Release notes tracker: <will be assigned by the Wrangler>
Detailed Description
The goal is to speed up the pulling of container images, via using zstd:chunked
. zstd:chunked
format has shown itself to speed up the pulling of images by as much as 90%. This change has been waiting for change for over 3 years, waiting for Docker to support the format.
Feedback
This change was pushed by mistake to Fedora 41 and Fedora 40 with:
- Fedora 41: https://bodhi.fedoraproject.org/updates/FEDORA-2024-99379626aa (to be confirmed)
- Fedora 40: https://bodhi.fedoraproject.org/updates/FEDORA-2024-ab42dd0ffb
This change has been reverted for Fedora 40 in:
- https://bodhi.fedoraproject.org/updates/FEDORA-2024-ebe5c816a1
- https://github.com/coreos/fedora-coreos-tracker/issues/1748
To avoid further disruptions, this change has not been reverted in Fedora 41 yet. It will be reverted if it is rejected.
This change also impacts all Bootable Container variants of Fedora. Initial (non-optimized) support for zstd:chunked images is available in rpm-ostree since v2024.6.
Benefit to Fedora
Fedora would be the first distribution to support building zstd:chunked
format by default. Container engines (Podman, Buildah, Skopeo, CRI-O) which support the format natively would see a great speed up in pulling images. Other container engines including Docker and Containerd would see some speed up do the the compression algorithms of zstd, but not to the same degree as container engines with full support.
Scope
- Proposal owners: Dan Walsh, Giuseppe Scrivano
- Switch the default to
zstd:chunked
- Switch the default to
- Other developers:
- Test if their containers still work with the new format.
- Release engineering: #Releng issue number
- Verify that the Fedora infra can use
zstd:chunked
container images.
- Verify that the Fedora infra can use
- Policies and guidelines: N/A (not needed for this Change)
- Trademark approval: N/A (not needed for this Change)
- Alignment with Community Initiatives: N/A
Upgrade/compatibility impact
TODO: List here the first versions of podman, buildah, skopeo, moby with support for zstd:chunked.
This change is implemented by updating the /usr/share/containers/containers.conf
configuration file to set compression_format = "zstd:chunked"
by default.
Users would want to keep the existing format can enforce it with the following commands:
$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/ $ echo 'compression_format = "gzip"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf
How To Test
To test this change, you can create the following config file:
$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/ $ echo 'compression_format = "zstd:chunked"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf
See: https://github.com/containers/common/blob/main/docs/containers.conf.5.md
You can also use the --compression-format=zstd:chunked
option with podman push
or the --dest-compress-format=zstd:chunked
with skopeo.
See: https://docs.podman.io/en/latest/markdown/podman-push.1.html
--compression-format=gzip | zstd | zstd:chunked Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. The default is gzip unless overridden in the containers.conf file. --compression-level=level Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive) with a default of 3, while for gzip it is 1-9 (inclusive) and has a default of 5.
See: https://github.com/containers/skopeo/blob/main/docs/skopeo-copy.1.md
--dest-compress-format format Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. --dest-compress-level format Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive), while for gzip it is 1-9 (inclusive).
Then:
- Build and push container images to container registries.
- Inspect them to check that their format is
zstd:chunked
. - Pull the image back to the host.
- Now make a small change to the image and push it back to the registry.
- Remove the changed image leaving the original image in place.
- Now pull the updated image again.
- You should see a big change in download speed.
- Now attempt to pull the same image with Docker/Moby to make sure the image works with those container engines.
User Experience
Users should experience a nice improvement in download speeds of container images that were previously created with zstd:chunked
.
Dependencies
N/A
Contingency Plan
If we find issues with zstd:chunked
while turned on in Rawhide, we can easily reverse direction and go back to the original default gzip
format.
Documentation
N/A (not a System Wide Change)
Release Notes
Containers are pushed to container registries using the zstd:chunked
format.