From Fedora Project Wiki
mNo edit summary
 
(18 intermediate revisions by 4 users not shown)
Line 1: Line 1:
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
= Default podman created images to zstd:chunked =


= Change Proposal Name <!-- Modify podman created images to zstd:chunked by default --> =
{{Change_Rejected_Banner}}


{{Change_Proposal_Banner}}
== Summary ==


== Summary ==
Modify `podman push`, `buildah push` and `skopeo copy` to default to creating container images compressed with `zstd:chunked`. Users can modify `containers.conf` to continue to use the legacy `gzip` image format.
Modify podman push to default to creating images with zstd:chunked, allowing users to modify containers.conf to continue to use the legacy gzip image format.


== Owner ==
== Owner ==
<!--
For change proposals to qualify as self-contained, owners of all affected packages need to be included here. Alternatively, a SIG can be listed as an owner if it owns all affected packages.
This should link to your home wiki page so we know who you are.
-->
* Name: [[User:dwalsh| Daniel J Walsh]]
* Name: [[User:dwalsh| Daniel J Walsh]]
<!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
* Email: dwalsh@redhat.com
* Email: dwalsh@redhat.com
<!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
* FESCo shepherd: [[User:FASAccountName| Shehperd name]] <email address>
-->
* Name: [[User:giuseppe| Giuseppe Scrivano]]
* Name: [[User:giuseppe| Giuseppe Scrivano]]
* Email: giuseppe@redhat.com
* Email: gscrivan@redhat.com


== Current status ==
== Current status ==
Line 29: Line 20:
<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete-->
<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete-->


<!-- Select proper category, default is Self Contained Change -->
[[Category:SelfContainedChange]]
[[Category:SelfContainedChange]]
<!-- [[Category:SystemWideChange]] -->


* Targeted release: [https://docs.fedoraproject.org/en-US/releases/f40/ Fedora Linux 40]
* Targeted release: [https://docs.fedoraproject.org/en-US/releases/fxx/ Fedora Linux xx]
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}  
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page  
Line 41: Line 30:
ON_QA -> change is fully code complete
ON_QA -> change is fully code complete
-->
-->
* [<will be assigned by the Wrangler> devel thread]
* [https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/ULPLKTES265ZZSBVWSFPKR6NZTOLULHR/ Announced]
* FESCo issue: <will be assigned by the Wrangler>
* [https://discussion.fedoraproject.org/t/f41-change-proposal-default-podman-created-images-to-zstd-chunked-self-contained/125540 discussion thread]
* FESCo issue: [https://pagure.io/fesco/issue/3252 #3252]
* Tracker bug: <will be assigned by the Wrangler>
* Tracker bug: <will be assigned by the Wrangler>
* Release notes tracker: <will be assigned by the Wrangler>
* Release notes tracker: <will be assigned by the Wrangler>


== Detailed Description ==
== Detailed Description ==
<!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
 
The goal is to speed up the pulling of container images, via using zstd:chunked. zstd:chunked format has shown itself to speed up the pulling of images by as much as 90%. This change has been waiting for change for over 3 years, waiting for Docker to support the format.
The goal is to speed up the pulling of container images, via using `zstd:chunked`. `zstd:chunked` format has shown itself to speed up the pulling of images by as much as 90%. This change has been waiting for change for over 3 years, waiting for Docker to support the format.
 
== Feedback ==
== Feedback ==
<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->
 
It is not possible to directly push container images with both compression formats. See:
 
* https://github.com/moby/moby/issues/28394#issuecomment-1822741770
* https://github.com/opencontainers/image-spec/issues/803#issuecomment-745335277
 
Instead, it should be possible to create a manifest list with containers in both formats. See the discussion in https://discussion.fedoraproject.org/t/switch-fedora-container-images-to-support-zstd-chunked-format-by-default/123712 for more details.
 
This change was pushed by mistake to Fedora 41 and Fedora 40 with:
* Fedora 41: https://bodhi.fedoraproject.org/updates/FEDORA-2024-4918af16a3
* Fedora 40: https://bodhi.fedoraproject.org/updates/FEDORA-2024-ab42dd0ffb
 
This change has been reverted for Fedora 40 in:
* https://bodhi.fedoraproject.org/updates/FEDORA-2024-ebe5c816a1
* https://github.com/coreos/fedora-coreos-tracker/issues/1748
 
To avoid further disruptions, this change has not been reverted in Fedora 41 yet. It will be reverted if it is rejected.
 
This change also impacts all Bootable Container variants of Fedora. Initial (non-optimized) support for `zstd:chunked` images is available in rpm-ostree since [https://github.com/coreos/rpm-ostree/releases/tag/v2024.6 v2024.6].


== Benefit to Fedora ==
== Benefit to Fedora ==
Fedora would be the first distribution to support building zstd:chunked format by default. Container engines (Podman, Buildah, Skopeo, CRI-O)  which support the format natively would see a great speed up in pulling images.  Other container engines including Docker and Containerd would see some speed up do the the compression algorythms of zstd, but not to the same degree as container engines with full support.
 
Fedora would be the first distribution to support building `zstd:chunked` format by default. Container engines (Podman, Buildah, Skopeo, CRI-O)  which support the format natively would see a great speed up in pulling images.  Other container engines including Docker and Containerd would see some speed up do the the compression algorithms of zstd, but not to the same degree as container engines with full support.


== Scope ==
== Scope ==
* Proposal owners:
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Proposal owners: Dan Walsh, Giuseppe Scrivano
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
** Switch the default to `zstd:chunked`
* Other developers:
** Test if their containers still work with the new format.
* Release engineering: [https://pagure.io/releng/issues #Releng issue number] (To Do)
** Verify that the Fedora infra can use `zstd:chunked` container images.
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Community Initiatives: N/A


* Release engineering: [https://pagure.io/releng/issues #Releng issue number] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
== Upgrade/compatibility impact ==
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->


* Policies and guidelines: N/A (not needed for this Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
The following versions of each package have support for `zstd:chunked`:
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. Please submit a pull request with the proposed changes before submitting your Change proposal. -->


* Trademark approval: N/A (not needed for this Change)
* podman 5.1
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://pagure.io/Fedora-Council/tickets/issues ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->
* buildah 1.36
* skopeo 1.15


* Alignment with Community Initiatives:
Moby doesn't support `zstd:chunked` but it handles the container image as a standard `zstd` layer since Moby 23.0. containerd supports `zstd` since 1.5.
<!-- Does your proposal align with the current Fedora Community Initiatives: https://docs.fedoraproject.org/en-US/project/initiatives/ ? It's okay if it doesn't, but it's something to consider -->


== Upgrade/compatibility impact ==
This change is implemented by updating the `/usr/share/containers/containers.conf` configuration file to set `compression_format = "zstd:chunked"` by default.
<!-- What happens to systems that have had a previous versions of Fedora installed and are updated to the version containing this change? Will anything require manual configuration or data migration? Will any existing functionality be no longer supported? -->


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
Users that would like to keep using the existing format can enforce it with the following commands:


$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/
$ echo 'compression_format = "gzip"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf


== How To Test ==
== How To Test ==
Build and push container images to container registries. Inspect them to check that their format is zstd:chunked.  Pull the image back to the host.  Now make a small change to the image and push it back to the registry
Remove the changed image leaving the original image in place.  Now pull the updated image again.  You should see a big change in download speed.


Now attempt to pull the same image with Docker/Moby to make sure the image works with those container engines.
To test this change, you can create the following config file:
 
$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/
$ echo 'compression_format = "zstd:chunked"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf
 
See: https://github.com/containers/common/blob/main/docs/containers.conf.5.md
 
You can also use the `--compression-format=zstd:chunked` option with `podman push` or the `--dest-compress-format=zstd:chunked` with skopeo.
 
See: https://docs.podman.io/en/latest/markdown/podman-push.1.html
 
--compression-format=gzip | zstd | zstd:chunked
Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. The default is gzip unless overridden in the containers.conf file.
--compression-level=level
Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted
values are in the range 1-20 (inclusive) with a default of 3, while for gzip it is 1-9 (inclusive) and has a default of 5.
 
See: https://github.com/containers/skopeo/blob/main/docs/skopeo-copy.1.md
 
--dest-compress-format format
Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked.
--dest-compress-level format
Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive), while for gzip it is 1-9 (inclusive).
 
Then:
* Build and push container images to container registries.
* Inspect them to check that their format is `zstd:chunked`.
* Pull the image back to the host.
* Now make a small change to the image and push it back to the registry.
* Remove the changed image leaving the original image in place.
* Now pull the updated image again.
* You should see a big change in download speed.
* Now attempt to pull the same image with Docker/Moby to make sure the image works with those container engines.


== User Experience ==
== User Experience ==
Users should experience a nice improvement in download speeds of container images that were previously created with zstd:chunked.
 
Users should experience a nice improvement in download speeds of container images that were previously created with `zstd:chunked`.


== Dependencies ==
== Dependencies ==
<!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this change depends?  In other words, completion of another change owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel change)? -->


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A


== Contingency Plan ==


== Contingency Plan ==
If we find issues with `zstd:chunked` while turned on in Rawhide, we can easily reverse direction and go back to the original default `gzip` format.


If we find issues with zstd:chunked while turned on in Rawhide, we can easily reverse direction and go back to the original default gzip format.
All the container images and tags pushed with the `zstd:chunked` format will have to be pushed again with the `gzip` compression.


== Documentation ==
== Documentation ==
<!-- Is there upstream documentation on this change, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)  
N/A (not a System Wide Change)  


== Release Notes ==
== Release Notes ==
<!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are at https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/ -->
<!-- The release notes also help users know how to deal with platform changes such as ABIs/APIs, configuration or data file formats, or upgrade concerns.  If there are any such changes involved in this change, indicate them here.  A link to upstream documentation will often satisfy this need.  This information forms the basis of the release notes edited by the documentation team and shipped with the release.


Release Notes are not required for initial draft of the Change Proposal but has to be completed by the Change Freeze.  
Containers are pushed to container registries using the `zstd:chunked` format.
-->

Latest revision as of 15:23, 17 October 2024

Default podman created images to zstd:chunked

This is a rejected Change proposal for Fedora Linux.
This document represents a proposed Change that has been rejected or withdrawn as part of the Changes process. This page is preserved for historical record.

Summary

Modify podman push, buildah push and skopeo copy to default to creating container images compressed with zstd:chunked. Users can modify containers.conf to continue to use the legacy gzip image format.

Owner

Current status

Detailed Description

The goal is to speed up the pulling of container images, via using zstd:chunked. zstd:chunked format has shown itself to speed up the pulling of images by as much as 90%. This change has been waiting for change for over 3 years, waiting for Docker to support the format.

Feedback

It is not possible to directly push container images with both compression formats. See:

Instead, it should be possible to create a manifest list with containers in both formats. See the discussion in https://discussion.fedoraproject.org/t/switch-fedora-container-images-to-support-zstd-chunked-format-by-default/123712 for more details.

This change was pushed by mistake to Fedora 41 and Fedora 40 with:

This change has been reverted for Fedora 40 in:

To avoid further disruptions, this change has not been reverted in Fedora 41 yet. It will be reverted if it is rejected.

This change also impacts all Bootable Container variants of Fedora. Initial (non-optimized) support for zstd:chunked images is available in rpm-ostree since v2024.6.

Benefit to Fedora

Fedora would be the first distribution to support building zstd:chunked format by default. Container engines (Podman, Buildah, Skopeo, CRI-O) which support the format natively would see a great speed up in pulling images. Other container engines including Docker and Containerd would see some speed up do the the compression algorithms of zstd, but not to the same degree as container engines with full support.

Scope

  • Proposal owners: Dan Walsh, Giuseppe Scrivano
    • Switch the default to zstd:chunked
  • Other developers:
    • Test if their containers still work with the new format.
  • Release engineering: #Releng issue number (To Do)
    • Verify that the Fedora infra can use zstd:chunked container images.
  • Policies and guidelines: N/A (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with Community Initiatives: N/A

Upgrade/compatibility impact

The following versions of each package have support for zstd:chunked:

  • podman 5.1
  • buildah 1.36
  • skopeo 1.15

Moby doesn't support zstd:chunked but it handles the container image as a standard zstd layer since Moby 23.0. containerd supports zstd since 1.5.

This change is implemented by updating the /usr/share/containers/containers.conf configuration file to set compression_format = "zstd:chunked" by default.

Users that would like to keep using the existing format can enforce it with the following commands:

$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/
$ echo 'compression_format = "gzip"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf

How To Test

To test this change, you can create the following config file:

$ sudo install -dm 0755 -o 0 -g 0 /etc/containers/containers.conf.d/
$ echo 'compression_format = "zstd:chunked"' | sudo tee /etc/containers/containers.conf.d/compress-zstd-chunked.conf

See: https://github.com/containers/common/blob/main/docs/containers.conf.5.md

You can also use the --compression-format=zstd:chunked option with podman push or the --dest-compress-format=zstd:chunked with skopeo.

See: https://docs.podman.io/en/latest/markdown/podman-push.1.html

--compression-format=gzip | zstd | zstd:chunked
Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked. The default is gzip unless overridden in the containers.conf file.

--compression-level=level
Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted 
values are in the range 1-20 (inclusive) with a default of 3, while for gzip it is 1-9 (inclusive) and has a default of 5.

See: https://github.com/containers/skopeo/blob/main/docs/skopeo-copy.1.md

--dest-compress-format format

Specifies the compression format to use. Supported values are: gzip, zstd and zstd:chunked.

--dest-compress-level format

Specifies the compression level to use. The value is specific to the compression algorithm used, e.g. for zstd the accepted values are in the range 1-20 (inclusive), while for gzip it is 1-9 (inclusive).

Then:

  • Build and push container images to container registries.
  • Inspect them to check that their format is zstd:chunked.
  • Pull the image back to the host.
  • Now make a small change to the image and push it back to the registry.
  • Remove the changed image leaving the original image in place.
  • Now pull the updated image again.
  • You should see a big change in download speed.
  • Now attempt to pull the same image with Docker/Moby to make sure the image works with those container engines.

User Experience

Users should experience a nice improvement in download speeds of container images that were previously created with zstd:chunked.

Dependencies

N/A

Contingency Plan

If we find issues with zstd:chunked while turned on in Rawhide, we can easily reverse direction and go back to the original default gzip format.

All the container images and tags pushed with the zstd:chunked format will have to be pushed again with the gzip compression.

Documentation

N/A (not a System Wide Change)

Release Notes

Containers are pushed to container registries using the zstd:chunked format.