Shipping Updates to Modules
A Factory 2.0 Focus Document
What is a Focus Document?
The Factory 2.0 team produces a confusing number of documents. The first round was about the Problem Statements we were trying to solve. Let’s retroactively call them Problem Documents. The Focus Documents (like this one) focus on some system or some aspect of our solutions that cut across different problems. The content here doesn’t fit cleanly in one problem statement document, which is why we broke it out.
Introduction
For Fedora 26 (Boltron), we said from the start that we would only be doing a GA release and that we would release no updates, security fixes or otherwise. For Fedora 27 and onwards that cannot stand. We need a way to ship updates to the modular release.
Background on Bodhi
- Bodhi is the service in Fedora that manages updates to the traditional distro.
- Packagers submit their builds to Bodhi as updates to queue them for distribution.
- Community QA persons provide feedback on bodhi updates (+1 or -1).
- The output of automated QA tools is visualized in the Bodhi web UI.
- Release Engineers use Bodhi’s backend to make and distribute the updates repos. This is called the “bodhi masher”.
Multi-type Support in Bodhi
Beyond shipping updates to modules, there are requests out there for Bodhi to ship updates to lots of other non-rpm content: containers chief among them. There is a milestone in the upstream issue tracker listing the work needed in general to make Bodhi capable of handling non-rpm content types.
The common requirement on these content types is that their individual builds must be represented as a build in koji. The maxim from the Bodhi maintainers is: “if it can be tagged in Koji, it can be shipped by Bodhi”. There are a thousand different ways we could have designed this -- but this statement helps us narrow in on an approach.
RPMs, of course, meet this requirement. They are built in Koji, and therefore have a uniquely-identifying “build object” in Koji’s database. They can be tagged into Koji tags.
Containers, too, meet this requirement. They are built in a different system -- the OpenShift Build System (OSBS) -- but when OSBS is done, it imports them back into Koji via the Content Generator API. At this point, container builds have a corresponding “build object” in Koji’s database, and they can be tagged into Koji tags.
Modules, at the moment, do not meet this requirement. Module builds are orchestrated by the Module Build Service (MBS). It uses Koji to build all of the components of the module, but koji never knows about the module as a whole.
To solve this, we are going to add an additional step at the end of the MBS’ build process. The MBS will use the Content Generator API to import a “note” about the built modules back into Koji. This top-level build object serves as a marker that Bodhi can move through various koji tags as a part of the standard update workflow.
Other fun facts about multi-type/modular support
No multi-type updates. On one level, it might make sense to have a single update that contains both a module and a container built from that module. They either ship or don’t ship as a unit. For a variety of reasons, we decided to forbid this. Updates can only be of a single type: either rpm updates, modular updates, or container updates. Eventually, (after F27) Bodhi will grow the ability to link these updates so that they can ship as a group.
No specification of type when submitting new updates. Instead, the server infers the content_type from NVR given. This makes it so the same API with the same arguments can be used to submit an rpm update, a modular update or a container update. These means the API change for multi-type support is minimal. You can read the content-type of an update from the API, but you cannot write it. The Bodhi server decides this for you by asking Koji for details about the NVRs you gave.
Mashing Modules
This is the hard part.
On one level, there is a simple requirement: the mashed updates repositories produced from modules must now additionally contain the the concatenated modulemd file describing what modules are present.
There is a much deeper problem to solve of how to mash the repository in the first place.
Bodhi’s backend does this today by invoking a tool called mash. Mash expects to only deal with rpms that come from a common koji tag (like, say, f26-updates-pending).
To mash repos from modules, we have identified two possible approaches:
For a first, high-level approach we could teach the bodhi masher to mash modules directly. We could mandate that the module NSV (the “NVR” of the module as a whole) must appear in a traditional-style tag. We could then either:
- Have Bodhi tag the components from each module into a common tag, and then have mash operate on that common tag.
- Teach mash how to operate on multiple tags at once and to pull content from all of those tags together into a single mash.
In either case, inserting the concatenated modulemd file into the repo metadata will be necessary.
The second high-level approach is to replace the bodhi masher with pungi, since pungi already understands how to put modules together from our F26 work on the “Boltron” release. Bodhi’s backend would “simply” need to write out a pungi config file on the fly, and invoke some pungi functions. A major challenge here is that pungi currently only knows how to produce composes from scratch. To produces “updates” repos like Bodhi does, we would want pungi to be able to produce a new “updates” compose as a kind of tweak on a pre-existing compose. It currently has no notion of doing this, so we would need to teach it.
This has other indirect benefits. From Bodhi’s point of view, using pungi would give us the ability to produce other kinds of artifacts as part of the updates process (ostrees, base images, etc..). From the point of view of the GA compose, an incremental compose process could hypothetically cut down on the many-hours long compose process that we endure today.
Updating pungi to allow for incremental composes and rewiring bodhi to use that looks promising, but it is also a more invasive and risky change.
Further scoping and practical investigation is required before we figure out which way we’re going to go.