Latest revision as of 19:57, 28 February 2024

Build Fedora Cloud Edition Images Using Kiwi in Koji

Summary

Fedora Cloud Edition images will be built with Kiwi, which will replace the unmaintained ImageFactory tooling that is currently being used to build the cloud base images.

We can already build Fedora Cloud Edition images outside of Koji using composite Kiwi definitions. However the integration with Koji must be enabled to fulfuill our goal of building official images within the Fedora infrastructure and fully replace the current usage of ImageFactory.

This transition is consistent with the direction of the Cloud Product Requirements Description (PRD). Kiwi provides the Cloud Working Group with a tool that preserve previous choices to build images using composable configurations and to provide a reproducible process for building images related to the cloud edition, including Fedora Cloud Base images for Vagrant, Azure, AWS, GCP, and generic images. This also opens up the ability to run container builds and WSL2 builds using the the composable image definitions to maintain a base image and then update the specifics needed for each specialized image using a smaller configuration file.

Owner

Name: David Duncan
Email: davdunc@amazon.com

Current status

Targeted release: Fedora Linux 40
Last updated: 2024-02-28
Announced
Discussion thread
FESCo issue: #3137
Tracker bug: #2260081
Release notes tracker: #1074

Detailed Description

While working on the production of cloud images for Fedora Linux 38 and Fedora Linux 39, the cloud-sig team did significant work to support transition from the current ImageFactory-based build tools that are outdated (but still functioning) to a tool that is supported by a broader community. The cloud team has successfully built and tested the creation of images with the kiwi application. Successful builds and tests of image builds supporting all of the previous change proposals and configuration changes to the Fedora Cloud base images has been validated and can be reproduced using the kiwi descriptions. The cloud edition WG finds that kiwi provides the most consistent experience with the least number of concerns over our current deliverables today. The cloud working group continues to focus on building support for specific requirements around specialized images that are planned parts of the cloud edition PRD included in section 2.3.

This work is also supportive of the Toolbx team to build and test containers and move their work to kiwi. There is full support for this with the container pack.

Feedback

We have evaluated a number of existing image build tools as part of this Change. Ultimately, the Fedora Cloud WG chose to adopt kiwi because it retains ideal qualities of our current tooling in a way that benefits the cloud-sig and the community at large. We have cultivated a strong relationship with the upstream project, who has been receptive to our needs and made improvements based on our requirements. Kiwi is not a disruption, but an opportunity to decrease the complexity necessary to produce current and additional use cases immediately and to ensure that builds are execute securely.

We are aware of Fedora Workstation's trial of osbuild (the upstream project for Red Hat Image Builder) for their live image, and have closely evaluated it as an option for Fedora Cloud as well. Discussions with members of the image builder team have been promising, but their mission doesn't directly align with the Cloud Working Group's goals immediately. Without that alignment, we are not prioritizing the same goals today. This is not a shortcoming of the cloud working group or the osbuild tools, it is a difference in timing of feature delivery. Fedora Workstation and Fedora Cloud are two different groups. We use different tools for building images today so their changes are typically independent of those we make. Currently, Fedora Workstation uses Lorax and Fedora Cloud uses ImageFactory and Oz. The cloud working group is working aggressively to eliminate our usage of ImageFactory because it is legacy code and not easily extended.

We also evaluated mkosi and decided not to pursue it due to the lack of flexibility to support all the image types we are aiming to offer. Its highly opinionated view of how images should be structured and limited framework for customization make it difficult to recommend as a framework for our builds. Additionally, it cannot support all of Fedora's architectures due to requiring GPT, nor can it fully support Fedora Cloud's preferred disk setup due to the aforementioned opinions of how images should be structured. Finally, when testing the generated images, the results did not line up with how we expected images to be laid out and it caused difficulties when dealing with certain classes of package upgrades (such as bootloader or kernel packages). There is also no Koji plugin at this time for running mkosi builds.

Benefit to Fedora

Most importantly, the kiwi builders eliminate a series of legacy build tools for Fedora Cloud Base images

Visible to advanced users:

Allows Fedora Images to be built on many different platforms and distributions without modification to the runners
Extends the composition strategies available to users
Leaves the base image configuration that can be managed to ensure that it meets standard requirements for local virt installations
Includes the ability to leverage user-defined scripting in the image definition.
Adds a koji builder and image definitions that are simple to update and modify
Provides increased time for prioritization of features in the Fedora Images according to user feedback and user requirements
Supports multiple build types, from ISO to raw disk images, and all the way to WSL2 and containers.

This also aligns with the Fedora Asahi Remix and its usage of kiwi to build its images, as this lays the groundwork for those images to eventually be built in Fedora infrastructure as support for Apple Silicon Macs gets upstreamed.

Scope

Proposal owners:
- Build and test kiwi definition files: COMPLETE
- Package kiwi: COMPLETE
- Add configuration to Fedora Pungi configuration
Other developers:
- Enable kiwi plugin in Koji: releng issue #11726
- Add support for KiwiBuild tasks to pungi: pungi issue #1710

Submit image build requirements as a kiwi descriptions

Release engineering: #11854

Policies and guidelines: Fedora Cloud Edition documentation should be updated to reflect the usage of the new tooling and how to use and contribute to it.

Trademark approval: N/A (not needed for this Change)

Alignment with Community Initiatives:

All software and requests are consistent with the decision process and similar exceptions across other groups in Fedora.

Upgrade/compatibility impact

The previous methodologies for using Fedora Quickstarts for Fedora Cloud Edition will be retired. The kiwi descriptions will support builds. We will use Toddler and Ansible to deliver images to the various public cloud targets (GCP, AWS, Azure, OCI, etc.)

How To Test

Test by working with the various images

Import the image into a test account for the associated cloud provider(s)
start an instance from that image
login to the instance successfully.

User Experience

this provides a simplified method for creating composable image definitions and overlays. Users will find that there are additional images supporting targeted workloads and build methods. They will find that those images are more readily available.

Dependencies

This Change depends on work in pungi to enable the use of the KiwiBuild Koji task as part of composes. It also depends on release engineering to enable the kiwi plugin in Koji.

Contingency Plan

Contingency mechanism: Revert back to ImageFactory and continue to support builds using the kickstart (.ks) files for image builds.
Contingency deadline: Beta freeze
Blocks release? Yes

Documentation

Documentation for kiwi is available from the upstream site. Once the Koji plugin is enabled, we will create accompanying documentation for SIG members on using the functionality.

Release Notes

Fedora Cloud Images are now built with the kiwi image build tool, using definitions from the fedora-kiwi-descriptions repository.

This has enabled Fedora Cloud to introduce 64-bit ARM cloud images for Azure and Google Cloud, as well as 64-bit ARM Vagrant images.

@@ Line 1: / Line 1: @@
 = Build Fedora Cloud Edition Images Using Kiwi in Koji =
-{{Change_Proposal_Banner}}
 == Summary ==
-The primary focus of this transition to kiwi is the elimination of the unmaintained [https://github.com/redhat-imaging/imagefactory ImageFactory] tool to produce cloud base images, consistent with the direction of the Cloud Product Requirements Description (PRD).
-Fedora Cloud Edition images can now be built using composite kiwi definitions from Kiwi outside of Koji. However the integration with Koji must be enabled for building official images in Fedora infrastructure.
+Fedora Cloud Edition images will be built with [https://osinside.github.io/kiwi/ Kiwi], which will replace the unmaintained [https://github.com/redhat-imaging/imagefactory ImageFactory] tooling that is currently being used to build the cloud base images.
-Kiwi provides the Cloud Working Group with a tool that preserve previous choices to build images using composable configurations and to provide a reproducible process for building images related to the cloud edition, including Fedora Cloud Base images for Vagrant, Azure, AWS, GCP, and generic images. This also opens up the ability to run container builds and WSL2 builds using the the composable image definitions to maintain a base image and then update the specifics needed for each specialized image using a smaller configuration file.
+We can already build Fedora Cloud Edition images outside of Koji using [https://pagure.io/fedora-kiwi-descriptions composite Kiwi definitions]. However the integration with Koji must be enabled to fulfuill our goal of building official images within the Fedora infrastructure and fully replace the current usage of ImageFactory.
+This transition is consistent with the direction of the [https://fedoraproject.org/wiki/Cloud/Cloud_PRD Cloud Product Requirements Description (PRD)]. Kiwi provides the Cloud Working Group with a tool that preserve previous choices to build images using composable configurations and to provide a reproducible process for building images related to the cloud edition, including Fedora Cloud Base images for Vagrant, Azure, AWS, GCP, and generic images. This also opens up the ability to run container builds and WSL2 builds using the the composable image definitions to maintain a base image and then update the specifics needed for each specialized image using a smaller configuration file.
 == Owner ==
@@ Line 20: / Line 20: @@
 == Current status ==
-[[Category:ChangePageIncomplete]]
+[[Category:ChangeAcceptedF40]]
 <!-- When your change proposal page is completed and ready for review and announcement -->
 <!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
@@ Line 36: / Line 36: @@
 ON_QA -> change is fully code complete
 -->
-* [<will be assigned by the Wrangler> devel thread]
+* [https://lists.fedoraproject.org/archives/list/devel-announce@lists.fedoraproject.org/thread/OOMOAR6FZSFQUTT5QEYLZFDKMKTVRYRI/ Announced]
-* FESCo issue: <will be assigned by the Wrangler>
+* [https://discussion.fedoraproject.org/t/f40-change-proposal-build-fedora-cloud-edition-images-using-kiwi-in-koji-system-wide/100078 Discussion thread]
-* Tracker bug: <will be assigned by the Wrangler>
+* FESCo issue: [https://pagure.io/fesco/issue/3137 #3137]
-* Release notes tracker: <will be assigned by the Wrangler>
+* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=2260081 #2260081]
+* Release notes tracker: [https://pagure.io/fedora-docs/release-notes/issue/1074 #1074]
 == Detailed Description ==
-The cloud team has successfully built and tested the creation of images with the <code>kiwi</code> application. Successful builds and tests of image builds supporting all of the previous change proposals and configuration changes to the Fedora Cloud base images has been validated and can be reproduced using the [https://pagure.io/fedora-kiwi-descriptions kiwi descriptions]. The cloud edition WG finds that kiwi provides the most consistent experience with the least number of concerns over our current deliverables today. The cloud working group continues to focus on building support for specific requirements around specialized images that are planned parts of the [https://fedoraproject.org/wiki/Cloud/Cloud_PRD cloud edition PRD] included in section 2.3.
+While working on the production of cloud images for Fedora Linux 38 and Fedora Linux 39, the cloud-sig team did significant work to support transition from the current ImageFactory-based build tools that are outdated (but still functioning) to a tool that is supported by a broader community. The cloud team has successfully built and tested the creation of images with the <code>kiwi</code> application. Successful builds and tests of image builds supporting all of the previous change proposals and configuration changes to the Fedora Cloud base images has been validated and can be reproduced using the [https://pagure.io/fedora-kiwi-descriptions kiwi descriptions]. The cloud edition WG finds that kiwi provides the most consistent experience with the least number of concerns over our current deliverables today. The cloud working group continues to focus on building support for specific requirements around specialized images that are planned parts of the [https://fedoraproject.org/wiki/Cloud/Cloud_PRD cloud edition PRD] included in section 2.3.
-While working on the production of cloud images for Fedora Linux 38 and Fedora Linux 39, the cloud-sig team did significant work to support transition from the current image build tools that are outdated (but still functioning) to a tool that is supported by a broader community. Discussions with members of the image builder team have been promising, but their mission doesn't directly align with the Cloud Working Groups goals immediately. Without that alignment, we are not prioritizing the same goals today. This is not a shortcoming of the cloud working group or the osbuild tools, it is a difference in timing of feature delivery. Since the WG has achieved significant success by using the kiwi and the kiwi definitions currently in a way that is consistent with the desired working group results, the group believes that it is beneficial to use that work product while working with the Image Builder team to achieve the goals outlined below.
-From the Fedora Cloud Edition WG perspective, osbuild still requires a few components necessary to take advantage of today that are in the [https://fedoraproject.org/wiki/Cloud/Cloud_PRD#Example_Use_Cases cloud PRD use cases]. Here are a few of the things that we have identifies architecturally that led us in this direction:
-* osbuild expects code written for each distribution it can build, and hard-codes content and content locations
-* osbuild requires writing code to teach it about every distribution it runs on
-* osbuild does not provide a supported method to run arbitrary logic in an image build
-* osbuild does not provide a supported method to overlay arbitrary files into the image rootfs
-The first two problems mean that any blueprints we make are not reusable for downstream consumers nor derivatives who may want to use our image descriptions to build their own. From the perspective of encouraging people to remix our images, that's not ideal. For additional users not able to fully define everything in blueprints means that it's a nightmare for image reproducibility, because people need exactly the same osbuild versions to get the same output, which was something we avoided by using kiwi.
-The last two bulleted issues are why cloud chose to use kiwi to do our own customizations. Without this support, it is difficult to support work that downstream consumers ask to use to customize and build their own custom images. They are disappointed that they require the use of multiple tools to customer their builds: 1) osbuild for the base image and then use something like complex cloud-init scripts, or Amazon's ec2-imagebuilder or tools like ansible or packer for the customization. We'd like to maintain a significantly simpler and more consistent process. The Cloud team is aware that this is on the roadmap for osbuild, but those requirements are code complete in kiwi today. The use of kiwi in koji builds does not require any shift in the prioritization of feature support in osbuild or significantly depleting the engineering effort available for the cloud edition off mission.
-[https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/EOJB6WUY2PKS5VOTVN6FG5PLN2SAKSNY/#INCIUFQLWO6PGXF3RHZMCW7OHXYNHCDH Stable Container Builds] are currently supported with kiwi while image builder composes are not.
+This work is also supportive of the Toolbx team to build and test containers and move their work to kiwi. There is full support for this with the container pack.
-[https://lists.fedoraproject.org/archives/search?mlist=devel%40lists.fedoraproject.org&q=WSL Fedora WSL2 Build] can be delivered for users as a part of Cloud edition builds, something the cloud working group intends to
 == Feedback ==
-<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->
-It is well-known that there is significant pressure to use osbuild, the upstream project tools that supports the open core SaaS, [https://console.redhat.com/insights/image-builder Red Hat Image Builder] tools to produce images and this is not a disqualification of that effort. Kiwi is not a disruption, but an opportunity to decrease the complexity necessary to produce current and additional use cases immediately and to ensure that builds are execute securely. Using kiwi today means that the WG can continue to work with the image builder team to complete plans for a later osbuild integration.
+We have evaluated a number of existing image build tools as part of this Change. Ultimately, the Fedora Cloud WG chose to adopt kiwi because it retains ideal qualities of our current tooling in a way that benefits the cloud-sig and the community at large. We have cultivated a strong relationship with the upstream project, who has been receptive to our needs and made improvements based on our requirements. Kiwi is not a disruption, but an opportunity to decrease the complexity necessary to produce current and additional use cases immediately and to ensure that builds are execute securely.
-Users of the Fedora Cloud Base can easily reproduce these build environments on all currently supported Fedora releases.
+We are aware of Fedora Workstation's trial of osbuild (the upstream project for [https://console.redhat.com/insights/image-builder Red Hat Image Builder]) for their live image, and have closely evaluated it as an option for Fedora Cloud as well. Discussions with members of the image builder team have been promising, but their mission doesn't directly align with the Cloud Working Group's goals immediately. Without that alignment, we are not prioritizing the same goals today. This is not a shortcoming of the cloud working group or the osbuild tools, it is a difference in timing of feature delivery. Fedora Workstation and Fedora Cloud are two different groups. We use different tools for building images today so their changes are typically independent of those we make. Currently, Fedora Workstation uses Lorax and Fedora Cloud uses ImageFactory and Oz. The cloud working group is working aggressively to eliminate our usage of ImageFactory because it is legacy code and not easily extended.
-This is also independent of Fedora Workstation's trial of osbuild for their live image. Fedora Workstation and Fedora Cloud are two different groups. We use different tools for building images today so their changes are typically independent of those we make. Currently, Fedora Workstation uses Lorax and Fedora Cloud uses ImageFactory and Oz. The cloud working group is working aggressively to eliminate our usage of ImageFactory because it is legacy code and not easily extended.
+We also evaluated {{package|mkosi}} and decided not to pursue it due to the lack of flexibility to support all the image types we are aiming to offer. Its highly opinionated view of how images should be structured and limited framework for customization make it difficult to recommend as a framework for our builds. Additionally, it cannot support all of Fedora's architectures due to requiring GPT, nor can it fully support Fedora Cloud's preferred disk setup due to the aforementioned opinions of how images should be structured. Finally, when testing the generated images, the results did not line up with how we expected images to be laid out and it caused difficulties when dealing with certain classes of package upgrades (such as bootloader or kernel packages). There is also no Koji plugin at this time for running mkosi builds.
 == Benefit to Fedora ==
@@ Line 89: / Line 75: @@
 ** Build and test [https://pagure.io/fedora-kiwi-descriptions kiwi definition files]: COMPLETE
 ** Package {{package|kiwi}}: COMPLETE
+** Add configuration to [https://pagure.io/pungi-fedora/ Fedora Pungi configuration]
-<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
 * Other developers:
+** Enable kiwi plugin in Koji: [https://pagure.io/releng/issue/11726 releng issue #11726]
+** Add support for <code>KiwiBuild</code> tasks to {{package|pungi}}: [https://pagure.io/pungi/issue/1710 pungi issue #1710]
 Submit image build requirements as a kiwi descriptions
-* Release engineering: [https://pagure.io/releng/issues #Releng issue number] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Release engineering: [https://pagure.io/releng/issue/11854 #11854]
-Completion of work on the koji builder in [https://pagure.io/releng/issue/11726 issue #11726]
+* Policies and guidelines: Fedora Cloud Edition documentation should be updated to reflect the usage of the new tooling and how to use and contribute to it.
-* Policies and guidelines: N/A (not needed for this Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
-<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. Please submit a pull request with the proposed changes before submitting your Change proposal. -->
-Fedora Cloud Edition documentation should be updated to reflect this build method
 * Trademark approval: N/A (not needed for this Change)
@@ Line 161: / Line 143: @@
 == Dependencies ==
 <!-- What other packages (RPMs) depend on this package?  Are there changes outside the developers' control on which completion of this change depends?  In other words, completion of another change owned by someone else and might cause you to not be able to finish on time or that you would need to coordinate?  Other upstream projects like the kernel (if this is not a kernel change)? -->
+This Change depends on work in {{package|pungi}} to enable the use of the <code>KiwiBuild</code> Koji task as part of composes. It also depends on release engineering to enable the kiwi plugin in Koji.
 <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 == Contingency Plan ==
-If this is not accepted, we will continue to use imagefactory (python2.7 based) tools.
+* Contingency mechanism: Revert back to ImageFactory and continue to support builds using the kickstart (.ks) files for image builds.
-* Contingency mechanism: We will continue to support builds using the kickstart (.ks) files for image builds.
+* Contingency deadline: Beta freeze
-* Contingency deadline: Tue 2024-02-20
+* Blocks release? Yes
-* Blocks release? No
 == Documentation ==
 <!-- Is there upstream documentation on this change, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
-Documentation for kiwi is available from the upstream site. Once the koji builder is enabled, we will create accompanying documentation.
+Documentation for kiwi is available from [https://osinside.github.io/kiwi the upstream site]. Once the Koji plugin is enabled, we will create accompanying documentation for SIG members on using the functionality.
-<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
-N/A (not a System Wide Change)
 == Release Notes ==
-Koji now supports composed build using kiwi and kiwi image definitions for building Fedora Cloud Images.
+Fedora Cloud Images are now built with the [https://osinside.github.io/kiwi kiwi] image build tool, using definitions from the [https://pagure.io/fedora-kiwi-descriptions fedora-kiwi-descriptions] repository.
-A WSL2 Fedora Image is available for use on Personal Computers, Servers, and Cloud Instances
-Newly available in F40, official aarch64 cloud images for Azure Compute and GCP
-Newly available in F40, official aarch64 Vagrant images
-Newly available in F40, a newly defined kiwi definition for building a single executable cloud image which can be booted directly from UEFI firmware or sourced by boot-loaders with little or no configuration.
+This has enabled Fedora Cloud to introduce 64-bit ARM cloud images for Azure and Google Cloud, as well as 64-bit ARM Vagrant images.
 <!-- The Fedora Release Notes inform end-users about what is new in the release.  Examples of past release notes are at https://docs.fedoraproject.org/en-US/fedora/latest/release-notes/ -->

Search

Changes/KiwiBuiltCloudImages: Difference between revisions