Use Kiwi koji builder for Cloud Edition
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
Summary
The primary focus of this transition to kiwi is the elimination of [imagefac](https://github.com/redhat-imaging/imagefactory) which uses legacy python support to produce cloud base images consistent with the direction of the Cloud Product Requirements Description (PRD). In order to use kiwi, the integration with koji must be in place. Fedora Cloud Edition images can now be built using composite kiwi definitions from Kiwi outside of koji. The Kiwi builder provides the Cloud Working Group with a tool that preserve previous choices to build images using composable configurations and to provide a reproducible process for building images related to the cloud edition, including Fedora Cloud Base images for Vagrant, Azure, AWS, GCP, and generic images. This also opens up the ability to run container builds and WSL2 builds using the the composable image definitions to maintain a base image and then update the specifics needed for each specialized image using a smaller configuration file.
Owner
- Name: David Duncan
- Email: davdunc@amazon.com
Current status
- Targeted release: Fedora Linux 40
- Last updated: 2023-12-17
- [<will be assigned by the Wrangler> devel thread]
- FESCo issue: <will be assigned by the Wrangler>
- Tracker bug: <will be assigned by the Wrangler>
- Release notes tracker: <will be assigned by the Wrangler>
Detailed Description
The cloud team has successfully built and tested the creation of images with the kiwi-ng application. The successful building and testing of image builds supporting all of the previous change proposals and configuration changes to the Fedora Cloud base images has been validated and can be reproduced using the [kiwi descriptions](https://pagure.io/fedora-kiwi-descriptions). Fedora Workstation and Fedora Cloud are two different groups. We use different tools for building images today so their changes are typically independent of those we make. Currently, Fedora Workstation uses Lorax and Fedora Cloud uses ImageFactory[2] and Oz. The cloud working group are working aggressively to eliminate our usage of ImageFactory because it is legacy code and not easily extended. The cloud edition WG finds that kiwi provides the most consistent experience with the least number of concerns over our current deliverables today. The cloud working group continues to focus on building support for specific requirements around specialized images that are planned parts of the [cloud edition PRD](https://fedoraproject.org/wiki/Cloud/Cloud_PRD) included in section 2.3.
While working on the the generation builds of Fedora 38 and Fedora 39, the cloud-sig team did significant work to support transition from the current image build tools that are outdated (but still functioning) to a tool that is supported by a broader community. Discussions with members of the image builder team have been promising, but their mission doesn't directly align with the Cloud Working Groups goals immediately. Without that alignment, we are not prioritizing the same goals today. This is not a shortcoming of the cloud working group or the osbuild tools, it is a difference in timing of feature delivery. Since the WG has achieved significant success by using the kiwi and the kiwi definitions currently in a way that is consistent with the desired working group results, the group believes that it is beneficial to use that work product while working with the Image Builder team to achieve the goals outlined below
From the Fedora Cloud Edition WG perspective, osbuild still requires a few components necessary to take advantage of today that are in the [cloud PRD use cases](https://fedoraproject.org/wiki/Cloud/Cloud_PRD#Example_Use_Cases). Here are a few of the things that we have identifies architecturally that led us in this direction:
- osbuild expects code written for each distribution it can build, and hard-codes content and content locations - osbuild requires writing code to teach it about every distribution it runs on - osbuild does not provide a supported method to run arbitrary logic in an image build - osbuild does not provide a supported method to overlay arbitrary files into the image rootfs
The first two problems mean that any blueprints we make are not reusable for downstream consumers nor derivatives who may want to use our image descriptions to build their own. From the perspective of encouraging people to remix our stuff, that's not ideal. For additional users not able to fully define everything in blueprints means that it's a nightmare for image reproducibility, because people need exactly the same osbuild versions to get the same output, which was something we avoided by using kiwi.
The last two bulleted issues are why cloud chose to use kiwi to do our own customizations. Without this support, it is difficult to support work that downstream consumers ask to use to customize and build their own custom images. They are disappointed that they require the use of multiple tools to customer their builds: 1) osbuild for the base image and then use something like complex cloud-init scripts, or Amazon's ec2-imagebuilder or tools like ansible or packer for the customization. We'd like to maintain a significantly more consistent process. The Cloud team is aware that this is on the roadmap for osbuild, but those requirements are code complete in kiwi today. The use of kiwi in koji builds does not require any shift in the prioritization of feature support in osbuild or significantly depleting the engineering effort available for the cloud edition off mission.
[Stable Container Builds](https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/EOJB6WUY2PKS5VOTVN6FG5PLN2SAKSNY/#INCIUFQLWO6PGXF3RHZMCW7OHXYNHCDH) are currently supported with kiwi while image builder composes are not. [Fedora WSL2 Build](https://lists.fedoraproject.org/archives/search?mlist=devel%40lists.fedoraproject.org&q=WSL) can be delivered for users as a part of Cloud edition builds, something the cloud working group intends to
Feedback
It is well-known that there is significant pressure to use osbuild, the upstream project tools that supports the open core SaaS, [Red Hat Image Builder](https://console.redhat.com/insights/image-builder) tools to produce images and this is not a disqualification of that effort. Kiwi is not a disruption, but an opportunity to decrease the complexity necessary to produce current and additional use cases immediately and to ensure that builds are execute securely. Using kiwi today means that the WG can continue to work with the image builder team to complete plans for a later osbuild transistion.
Users of the Fedora Cloud Base can easily reproduce these build environments on all currently supported Fedora releases.
> > We would like to add a new cloud image variant in Fedora 40,
> > which is planned to be uefi only and will use UKIs. See the fedora change > > page > > for details: > > > > https://fedoraproject.org/wiki/Changes/Unified_Kernel_Support_Phase_2 > > > > What does it take to get this done? > >
With this change, the first step would be to make a kiwi definition file that modifies specific entries in the fedora-cloud-base definition file and updates configuration to be uefi only. This is supported in kiwi today.
> > > > What is needed to add a new image to the image build process?
How about working with the cloud working group for these updates?
Benefit to Fedora
Most importantly, the kiwi builders eliminate a series of legacy build tools for Fedora Cloud Base images
Visible to advanced users:
- Allows Fedora Images to be built on many different platforms and distributions without modification to the runners
- Extends the composition strategies available to users
- Leaves the base image configuration that can be managed to ensure that it meets standard requirements for local virt installations
- Includes the ability to leverage user-defined scripting in the image definition.
- Adds a koji builder and image definitions that are simple to update and modify
- Provides increased time for prioritization of features in the osbuild tools according to user feedback and downstream requirements
- Provides increased time for prioritization of features in the Fedora Images according to user feedback and user requirements
- Supports multiple build types, from ISO to raw disk images, and all the way to WSL2 and containers.
Scope
- Proposal owners:
Build and test kiwi definition files: [COMPLETE](https://pagure.io/fedora-kiwi-descriptions) Package kiwi: [COMPLETE]()
- Other developers:
Submit image build requirements as a kiwi descriptions
- Release engineering: #Releng issue number
Completion of work on the koji builder in [issue #11726](https://pagure.io/releng/issue/11726)
- Policies and guidelines: N/A (not needed for this Change)
Fedora Cloud Edition documentation should be updated to reflect this build method
- Trademark approval: N/A (not needed for this Change)
- Alignment with Community Initiatives:
All software and requests are consistent with the decision process and similar exceptions across other groups in Fedora.
Upgrade/compatibility impact
The previous methodologies for using Fedora Quickstarts for Fedora Cloud Edition will be retired. The kiwi descriptions will support builds. We will use Toddler and Ansible to deliver images to the various public cloud targets (GCP, AWS, Azure, OCI, etc.)
How To Test
Test by working with the various images 1) Import the image into a test account for the associated cloud provider(s) 2) start an instance from that image 3) login to the instance successfully.
User Experience
this provides a simplified method for creating composable image definitions and overlays. Users will find that there are additional images supporting targeted workloads and build methods. They will find that those images are more readily available.
Dependencies
Contingency Plan
If this is not accepted, we will continue to use imagefactory (python2.7 based) tools. We will continue to support builds using the quickstart (.qs) files for image builds.
- Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
- Contingency deadline: N/A (not a System Wide Change)
- Blocks release? N/A (not a System Wide Change), Yes/No
Documentation
Documentation for kiwi is available from the upstream site. Once the koji builder is enabled, we will create accompanying documentation.
N/A (not a System Wide Change)
Release Notes
Koji now supports composed build using kiwi and kiwi image definitions for building Fedora Cloud Images.
A WSL2 Fedora Image is available for use on Personal Computers, Servers, and Cloud Instances
Newly available in F40, official aarch64 cloud images for Azure Compute and GCP
Newly available in F40, official aarch64 Vagrant images
Newly available in F40, a newly defined kiwi definition for building a single executable cloud image which can be booted directly from UEFI firmware or sourced by boot-loaders with little or no configuration.