Changes/KiwiBuiltCloudImages

Build Fedora Cloud Edition Images Using Kiwi in Koji

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

The primary focus of this transition to kiwi is the elimination of the unmaintained ImageFactory tool to produce cloud base images, consistent with the direction of the Cloud Product Requirements Description (PRD).

Fedora Cloud Edition images can now be built using composite kiwi definitions from Kiwi outside of Koji. However the integration with Koji must be enabled for building official images in Fedora infrastructure.

Kiwi provides the Cloud Working Group with a tool that preserve previous choices to build images using composable configurations and to provide a reproducible process for building images related to the cloud edition, including Fedora Cloud Base images for Vagrant, Azure, AWS, GCP, and generic images. This also opens up the ability to run container builds and WSL2 builds using the the composable image definitions to maintain a base image and then update the specifics needed for each specialized image using a smaller configuration file.

Owner

• Name: David Duncan
• Email: davdunc@amazon.com

Current status

• Targeted release: Fedora Linux 40
• Last updated: 2023-12-21
• [<will be assigned by the Wrangler> devel thread]
• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

While working on the production of cloud images for Fedora Linux 38 and Fedora Linux 39, the cloud-sig team did significant work to support transition from the current image build tools that are outdated (but still functioning) to a tool that is supported by a broader community. The cloud team has successfully built and tested the creation of images with the kiwi application. Successful builds and tests of image builds supporting all of the previous change proposals and configuration changes to the Fedora Cloud base images has been validated and can be reproduced using the kiwi descriptions. The cloud edition WG finds that kiwi provides the most consistent experience with the least number of concerns over our current deliverables today. The cloud working group continues to focus on building support for specific requirements around specialized images that are planned parts of the cloud edition PRD included in section 2.3.

Feedback

It is well-known that there is significant pressure to use osbuild, the upstream project tools that supports the Red Hat Image Builder tools to produce images and this is not a disqualification of that effort. Discussions with members of the image builder team have been promising, but their mission doesn't directly align with the Cloud Working Groups goals immediately. Without that alignment, we are not prioritizing the same goals today. This is not a shortcoming of the cloud working group or the osbuild tools, it is a difference in timing of feature delivery.

We also evaluated mkosi and decided not to pursue it due to the lack of flexibility to support all the image types we are aiming to offer. Its highly opinionated view of how images should be structured and limited framework for customization make it difficult to recommend as a framework for our builds. Additionally, it cannot support all of Fedora's architectures due to requiring GPT, nor can it fully support Fedora Cloud's preferred disk setup due to the aforementioned opinions of how images should be structured. Finally, when testing the generated images, the results did not line up with how we expected images to be laid out and it caused difficulties when dealing with certain classes of package upgrades (such as bootloader or kernel packages). There is also no Koji plugin at this time for running mkosi builds.

Ultimately, the Fedora Cloud WG chose to adopt kiwi because it retains ideal qualities of our current tooling in a way that benefits the cloud-sig and the community at large. We have cultivated a strong relationship with the upstream project, who has been receptive to our needs and made improvements based on our requirements. Kiwi is not a disruption, but an opportunity to decrease the complexity necessary to produce current and additional use cases immediately and to ensure that builds are execute securely.

This is also independent of Fedora Workstation's trial of osbuild for their live image. Fedora Workstation and Fedora Cloud are two different groups. We use different tools for building images today so their changes are typically independent of those we make. Currently, Fedora Workstation uses Lorax and Fedora Cloud uses ImageFactory and Oz. The cloud working group is working aggressively to eliminate our usage of ImageFactory because it is legacy code and not easily extended.

Benefit to Fedora

Most importantly, the kiwi builders eliminate a series of legacy build tools for Fedora Cloud Base images

Visible to advanced users:

• Allows Fedora Images to be built on many different platforms and distributions without modification to the runners
• Extends the composition strategies available to users
• Leaves the base image configuration that can be managed to ensure that it meets standard requirements for local virt installations
• Includes the ability to leverage user-defined scripting in the image definition.
• Adds a koji builder and image definitions that are simple to update and modify
• Provides increased time for prioritization of features in the Fedora Images according to user feedback and user requirements
• Supports multiple build types, from ISO to raw disk images, and all the way to WSL2 and containers.

This also aligns with the Fedora Asahi Remix and its usage of kiwi to build its images, as this lays the groundwork for those images to eventually be built in Fedora infrastructure as support for Apple Silicon Macs gets upstreamed.

Scope

• Proposal owners:
  • Build and test kiwi definition files: COMPLETE
  • Package kiwi: COMPLETE

• Other developers:

Submit image build requirements as a kiwi descriptions

• Release engineering: #Releng issue number

Completion of work on the koji builder in issue #11726

• Policies and guidelines: N/A (not needed for this Change)

Fedora Cloud Edition documentation should be updated to reflect this build method

• Trademark approval: N/A (not needed for this Change)

• Alignment with Community Initiatives:

All software and requests are consistent with the decision process and similar exceptions across other groups in Fedora.

Upgrade/compatibility impact

The previous methodologies for using Fedora Quickstarts for Fedora Cloud Edition will be retired. The kiwi descriptions will support builds. We will use Toddler and Ansible to deliver images to the various public cloud targets (GCP, AWS, Azure, OCI, etc.)

How To Test

Test by working with the various images

1. Import the image into a test account for the associated cloud provider(s)
2. start an instance from that image
3. login to the instance successfully.

User Experience

this provides a simplified method for creating composable image definitions and overlays. Users will find that there are additional images supporting targeted workloads and build methods. They will find that those images are more readily available.

Dependencies

This Change depends on work in pungi to enable the use of the KiwiBuild Koji task as part of composes.

Contingency Plan

• Contingency mechanism: Revert back to ImageFactory and continue to support builds using the kickstart (.ks) files for image builds.
• Contingency deadline: Beta freeze
• Blocks release? Yes

Documentation

Documentation for kiwi is available from the upstream site. Once the Koji plugin is enabled, we will create accompanying documentation for SIG members on using the functionality.

Release Notes

Fedora Cloud Images are now built with the kiwi image build tool, using definitions from the fedora-kiwi-descriptions repository.

This has enabled Fedora Cloud to introduce 64-bit ARM cloud images for Azure and Google Cloud, as well as 64-bit ARM Vagrant images.