Changes/ToolbxReleaseBlocker

Make Toolbx a release-blocking deliverable and have release-blocking test criteria

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

Up to date fedora-toolbox OCI images must be published on registry.fedoraproject.org as release-blocking deliverables, and there must be release-blocking test criteria to ensure usable toolbox RPMs.

Owner

• Name: Debarshi Ray, Sumantro Mukherjee
• Email: debarshir@redhat.com, sumukher@redhat.com

Current status

• Targeted release: Fedora Linux 39
• Last updated: 2023-05-16

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/54NW7GHUDLQCXLD754RNNZA6VVDWI75P/

• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

Currently, there is no formal requirement for Toolbx to be usable when a new Fedora is released. This is a problem for a tool that's so popular and provides something as fundamental as an interactive command line environment for software development and troubleshooting the host operating system. This Change aims to address this.

Toolbx has two parts — an OCI image, which defaults to fedora-toolbox on Fedora hosts, and the toolbox RPM. The OCI image is pulled by the RPM to set up a containerized interactive CLI environment.

First, we want to ensure that there are up to date fedora-toolbox OCI images published on registry.fedoraproject.org as release-blocking deliverables at critical points in the development schedule, just like the installation ISOs for the Editions from download.fedoraproject.org. This must at least happen when an upcoming Fedora release is branched from Rawhide, and for the Beta and Final release candidates. If possible, they should be updated more frequently as part of the nightly composes. We do not expect this to happen after a Fedora release has gone GA.

Note that making the fedora-toolbox OCI images release-blocking deliverables will implicitly make the base fedora images also release-blocking. Currently, that is not the case.

Second, we want to have release-blocking test criteria to ensure usable toolbox RPMs at critical points in the development schedule. This must be used to ensure that both changes in the Toolbx stack, and future Changes in other parts of the OS do not break Toolbx, and at least cover the Beta and Final release candidates.

Examples of changes in the Toolbx stack causing breakage can be FUSE preventing RPMs with file capabilities from being installed inside Toolbx containers, Toolbx bind mounts preventing RPMs with %attr() from being installed or causing systemd-tmpfiles(8) to throw errors, etc.. Examples of changes in other parts of the OS can be changes to Fedora's Kerberos stack causing Kerberos to stop working inside Toolbx containers, changes to the sysctl(8) configuration breaking ping(8), changes in Mutter breaking graphical applications, etc..

Note that having release-blocking test criteria for the toolbox RPM will also implicitly test the fedora-toolbox image.

Benefit to Fedora

This Change improves the quality of the containerized interactive command line Toolbx environments on Fedora by adding formal requirements to ensure that they are usable when a new Fedora is released. This will bring them closer to the reliability of similar environments running directly on the host operating system.

Toolbx is installed by default on CoreOS, Silverblue and Workstation. It is indispensable for software developers using Silverblue to bypass the difficulty of setting up a development environment in the usual way. It is widely used, even on Workstation, by those who don't want to pollute their host OS, or want to access a CLI environment that's different from the host's without installing a virtual machine, or want a pre-configured development environment.

It will be beneficial to consider the fedora-toolbox images as release-blocking deliverables because Fedora's OCI infrastructure is often broken. Here are two recent examples of fedpkg container-build not working. In the second case, it was preventing the images from being rebuilt to pull in an important bug-fix. The broken infrastructure prevents regular Fedora contributors from jumping in to rebuild and publish the images at critical points in the development schedule. Making them release-blocking deliverables would attract greater attention and scrutiny from release engineering and ensure that a Fedora development cycle does not proceed with broken or outdated or missing fedora-toolbox images.

At the moment, the branching of an upcoming Fedora release from Rawhide is a particularly chaotic time. Since the fedora-toolbox images for Fedora Branched and Rawhide are not rebuilt as part of the branching process, there is a lot of confusion for end-users until someone manually rebuilds the images and gets them published, which can take some time as described above. Having the images built and published by release engineering as part of the branching will avoid this.

If someone installs the Beta or the Final on their host, and creates a Toolbx container using the default fedora-toolbox image, then, barring exceptions, the host and the container will have the same RPM versions. Just like Workstation and Silverblue are released with the same versions. This will ensure greater consistency in terms of bug-fixes, features and pending updates.

Last but not the least, we cannot have release-blocking test criteria for the toolbox RPMs unless the images are also release-blocking deliverables, because the RPMs depend on the images. This brings us to the benefits of the second part of this Change.

It will be beneficial to have release-blocking test criteria for the toolbox RPMs because they interact with different parts of the OS to set up the containerized interactive CLI environments. These environments have seamless access to the user’s home directory, the Wayland and X11 sockets, networking (including Avahi), removable devices (like USB sticks), systemd journal, SSH agent, D-Bus, ulimits, /dev and the udev database, etc., and any unexpected change in another part of the OS can break the environment.

The release-blocking test criteria will be a way to co-ordinate several disparate groups of developers to ensure that the containerized interactive CLI Toolbx environments on Fedora are just as reliable as those running directly on the host OS.

Scope

• Proposal owners: Write down the release-blocking test criteria for the toolbox RPMs that will be used for testing and the blocker bugs process.

• Other developers: The release-blocking test criteria for Toolbx will require others to debug and fix bugs, possibly blockers, and be mindful of making changes to other parts of the operating system that break the Toolbx environment.

• Release engineering: #11399

• Policies and guidelines: N/A (not needed for this Change)

• Trademark approval: #449

• Alignment with Community Initiatives:

Upgrade/compatibility impact

This is only a change to the Fedora release processes. Therefore, systems with a previous version of Fedora won't need any manual intervention.

There should not be any user-visible change, other than, barring exceptions, the fedora-toolbox image having the same RPM versions as the host at critical points in the development schedule, fewer bugs and a more reliable Toolbx.

How To Test

Up to date fedora-toolbox images must be available from registry.fedoraproject.org when an upcoming Fedora release is branched from Rawhide, and for the Beta and Final release candidates. The following steps can be used to test this:

• Check the Toolbx containers and images present locally using toolbox list.
• Remove any Toolbx containers and images for the Fedora release under development using toolbox rm and toolbox rmi.
• Create a new Toolbx container with toolbox create and enter it with toolbox enter.
• Use sudo dnf update inside the Toolbx container. Barring exceptions, the container should have the same RPM versions as the host.

The toolbox RPMs must satisfy the test criteria for the Beta and Final release candidates. Writing the test criteria is part of this Change, so they don't exist at the moment, but likely examples can be:

• Graphical Apps running inside Toolbx container should be accessible outside
• Graphical Apps (text editors) should retain their state even when the virtual terminal is closed

User Experience

This Change improves the quality of the containerized interactive command line Toolbx environments on Fedora by ensuring that they are just as reliable as those running directly on the host operating system.

If someone installs the Beta or the Final on their host, and creates a Toolbx container using the default fedora-toolbox image, then, barring exceptions, the host and the container should have the same RPM versions. Just like Workstation and Silverblue are released with the same versions. This will ensure greater consistency in terms of bug-fixes, features and pending updates.

Dependencies

N/A (not needed for this Change)

Contingency Plan

• Contingency mechanism: If there are no up to date fedora-toolbox images published on registry.fedoraproject.org as release-blocking deliverables, then the release-blocking test criteria for the toolbox RPMs cannot be put into production. In that case this Change cannot be implemented and status quo will be maintained. If the images get published, but the test criteria is absent, then only the first half of the Change will be implemented, and users can still benefit from the more predictably updated images.
• Contingency deadline: We need this by the Change completion deadline or before Fedora 39 is branched from Rawhide, whichever is earlier. As per the schedule, both of those are currently set to happen on the 8th of August 2023.
• Blocks release? No

Documentation

• Toolbx basics: https://containertoolbx.org/install/
• Toolbx manuals: https://github.com/containers/toolbox/tree/main/doc