Revision as of 13:19, 4 July 2024

Default podman created images to zstd:chunked

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

Modify podman push, buildah push and skopeo copy to default to creating images with zstd:chunked. Users can modify containers.conf to continue to use the legacy gzip image format.

Owner

Name: Daniel J Walsh
Email: dwalsh@redhat.com
Name: Giuseppe Scrivano
Email: gscrivan@redhat.com

Current status

Targeted release: Fedora Linux 41
Last updated: 2024-07-04
[<will be assigned by the Wrangler> devel thread]
FESCo issue: <will be assigned by the Wrangler>
Tracker bug: <will be assigned by the Wrangler>
Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

The goal is to speed up the pulling of container images, via using zstd:chunked. zstd:chunked format has shown itself to speed up the pulling of images by as much as 90%. This change has been waiting for change for over 3 years, waiting for Docker to support the format.

Feedback

Benefit to Fedora

Fedora would be the first distribution to support building zstd:chunked format by default. Container engines (Podman, Buildah, Skopeo, CRI-O) which support the format natively would see a great speed up in pulling images. Other container engines including Docker and Containerd would see some speed up do the the compression algorithms of zstd, but not to the same degree as container engines with full support.

Scope

Proposal owners: Dan Walsh, Giuseppe Scrivano
- Switch the default to zstd:chunked
Other developers:
- Test if their containers still work with the new format.
Release engineering: #Releng issue number
- Verify that the Fedora infra can use zstd:chunked container images.
Policies and guidelines: N/A (not needed for this Change)
Trademark approval: N/A (not needed for this Change)
Alignment with Community Initiatives: N/A

Upgrade/compatibility impact

TODO: List here the first versions of podman, buildah, skopeo, moby with support for zstd:chunked.

This change is implemented by updating the /usr/share/containers/containers.conf configuration file to set compression_format = "zstd:chunked" by default.

Users would want to keep the existing format can enforce it with the following commands:

` $ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/ $ echo 'compression_format = "gzip"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf `

How To Test

To test this change, you can create the following config file:

` $ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/ $ echo 'compression_format = "zstd:chunked"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf `

See: https://github.com/containers/common/blob/main/docs/containers.conf.5.md

You can also use the --compression-format=zstd:chunked option with podman push or the --dest-compress-format=zstd:chunked with skopeo.

See: https://docs.podman.io/en/latest/markdown/podman-push.1.html

` --compression-format=gzip | zstd | zstd:chunked Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. The default is gzip unless overridden in the containers.conf file.

--compression-level=level Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive) with a default of 3, while for gzip it is 1-9 (inclusive) and has a default of 5. `

See: https://github.com/containers/skopeo/blob/main/docs/skopeo-copy.1.md

` --dest-compress-format format

Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked.

--dest-compress-level format

Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive), while for gzip it is 1-9 (inclusive). `

Then:

Build and push container images to container registries.
Inspect them to check that their format is zstd:chunked.
Pull the image back to the host.
Now make a small change to the image and push it back to the registry.
Remove the changed image leaving the original image in place.
Now pull the updated image again.
You should see a big change in download speed.
Now attempt to pull the same image with Docker/Moby to make sure the image works with those container engines.

User Experience

Users should experience a nice improvement in download speeds of container images that were previously created with zstd:chunked.

Dependencies

N/A

Contingency Plan

If we find issues with zstd:chunked while turned on in Rawhide, we can easily reverse direction and go back to the original default gzip format.

Documentation

N/A (not a System Wide Change)

Release Notes

Containers are pushed to container registries using the zstd:chunked format.

@@ Line 1: / Line 1: @@
-<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
 = Default podman created images to zstd:chunked =
@@ Line 6: / Line 4: @@
 == Summary ==
-Modify podman push to default to creating images with zstd:chunked, allowing users to modify containers.conf to continue to use the legacy gzip image format.
+Modify `podman push`, `buildah push` and `skopeo copy` to default to creating images with `zstd:chunked`. Users can modify `containers.conf` to continue to use the legacy `gzip` image format.
 == Owner ==
-<!--
-For change proposals to qualify as self-contained, owners of all affected packages need to be included here. Alternatively, a SIG can be listed as an owner if it owns all affected packages.
-This should link to your home wiki page so we know who you are.
--->
 * Name: [[User:dwalsh| Daniel J Walsh]]
-<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
 * Email: dwalsh@redhat.com
-<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
-* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address>
--->
 * Name: [[User:giuseppe| Giuseppe Scrivano]]
 * Email: gscrivan@redhat.com
@@ Line 29: / Line 20: @@
 <!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete-->
-<!-- Select proper category, default is Self Contained Change -->
 [[Category:SelfContainedChange]]
-<!-- [[Category:SystemWideChange]] -->
 * Targeted release: [https://docs.fedoraproject.org/en-US/releases/f41/ Fedora Linux 41]
@@ Line 47: / Line 36: @@
 == Detailed Description ==
-<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
-The goal is to speed up the pulling of container images, via using zstd:chunked.  zstd:chunked format has shown itself to speed up the pulling of images by as much as 90%. This change has been waiting for change for over 3 years, waiting for Docker to support the format.
+The goal is to speed up the pulling of container images, via using `zstd:chunked`. `zstd:chunked` format has shown itself to speed up the pulling of images by as much as 90%. This change has been waiting for change for over 3 years, waiting for Docker to support the format.
 == Feedback ==
-<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->
 == Benefit to Fedora ==
-Fedora would be the first distribution to support building zstd:chunked format by default. Container engines (Podman, Buildah, Skopeo, CRI-O)  which support the format natively would see a great speed up in pulling images.  Other container engines including Docker and Containerd would see some speed up do the the compression algorythms of zstd, but not to the same degree as container engines with full support.
+Fedora would be the first distribution to support building `zstd:chunked` format by default. Container engines (Podman, Buildah, Skopeo, CRI-O)  which support the format natively would see a great speed up in pulling images.  Other container engines including Docker and Containerd would see some speed up do the the compression algorithms of zstd, but not to the same degree as container engines with full support.
 == Scope ==
 * Proposal owners: Dan Walsh, Giuseppe Scrivano
-<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
+** Switch the default to `zstd:chunked`
+* Other developers:
+** Test if their containers still work with the new format.
+* Release engineering: [https://pagure.io/releng/issues #Releng issue number]
+** Verify that the Fedora infra can use `zstd:chunked` container images.
+* Policies and guidelines: N/A (not needed for this Change)
+* Trademark approval: N/A (not needed for this Change)
+* Alignment with Community Initiatives: N/A
+== Upgrade/compatibility impact ==
+TODO: List here the first versions of podman, buildah, skopeo, moby with support for zstd:chunked.
+This change is implemented by updating the `/usr/share/containers/containers.conf` configuration file to set `compression_format = "zstd:chunked"` by default.
+Users would want to keep the existing format can enforce it with the following commands:
+```
+$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/
+$ echo 'compression_format = "gzip"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf
+```
+== How To Test ==
+To test this change, you can create the following config file:
+```
+$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/
+$ echo 'compression_format = "zstd:chunked"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf
+```
+See: https://github.com/containers/common/blob/main/docs/containers.conf.5.md
-* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+You can also use the `--compression-format=zstd:chunked` option with `podman push` or the `--dest-compress-format=zstd:chunked` with skopeo.
-<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
-* Release engineering: [https://pagure.io/releng/issues #Releng issue number] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+See: https://docs.podman.io/en/latest/markdown/podman-push.1.html
-<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.
-The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
-* Policies and guidelines: N/A (not needed for this Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+```
-<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. Please submit a pull request with the proposed changes before submitting your Change proposal. -->
+--compression-format=gzip | zstd | zstd:chunked
+Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. The default is gzip unless overridden in the containers.conf file.
-* Trademark approval: N/A (not needed for this Change)
+--compression-level=level
-<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://pagure.io/Fedora-Council/tickets/issues ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->
+Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive) with a default of 3, while for gzip it is 1-9 (inclusive) and has a default of 5.
+```
-* Alignment with Community Initiatives:
+See: https://github.com/containers/skopeo/blob/main/docs/skopeo-copy.1.md
-<!-- Does your proposal align with the current Fedora Community Initiatives: https://docs.fedoraproject.org/en-US/project/initiatives/ ? It's okay if it doesn't, but it's something to consider -->
-== Upgrade/compatibility impact ==
+```
-<!-- What happens to systems that have had a previous versions of Fedora installed and are updated to the version containing this change? Will anything require manual configuration or data migration? Will any existing functionality be no longer supported? -->
+--dest-compress-format format
-<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked.
+--dest-compress-level format
-== How To Test ==
+Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive), while for gzip it is 1-9 (inclusive).
-Build and push container images to container registries. Inspect them to check that their format is zstd:chunked.  Pull the image back to the host.  Now make a small change to the image and push it back to the registry
+```
-Remove the changed image leaving the original image in place.  Now pull the updated image again.  You should see a big change in download speed.
-Now attempt to pull the same image with Docker/Moby to make sure the image works with those container engines.
+Then:
+* Build and push container images to container registries.
+* Inspect them to check that their format is `zstd:chunked`.
+* Pull the image back to the host.
+* Now make a small change to the image and push it back to the registry.
+* Remove the changed image leaving the original image in place.
+* Now pull the updated image again.
+* You should see a big change in download speed.
+* Now attempt to pull the same image with Docker/Moby to make sure the image works with those container engines.
 == User Experience ==
-Users should experience a nice improvement in download speeds of container images that were previously created with zstd:chunked.
+Users should experience a nice improvement in download speeds of container images that were previously created with `zstd:chunked`.
 == Dependencies ==
-<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this change depends?  In other words, completion of another change owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel change)? -->
-<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+N/A
 == Contingency Plan ==
-If we find issues with zstd:chunked while turned on in Rawhide, we can easily reverse direction and go back to the original default gzip format.
+If we find issues with `zstd:chunked` while turned on in Rawhide, we can easily reverse direction and go back to the original default `gzip` format.
 == Documentation ==
-<!-- Is there upstream documentation on this change, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
-<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 N/A (not a System Wide Change)
 == Release Notes ==
-<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are at https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/ -->
-<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this change, indicate them here.  A link to upstream documentation will often satisfy this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release.
-Release Notes are not required for initial draft of the Change Proposal but has to be completed by the Change Freeze.
+Containers are pushed to container registries using the `zstd:chunked` format.
--->

Search

Changes/zstd:chunked: Difference between revisions