Module Stream Expansion
The “Problem”
Imagine I have an httpd module. To simplify things, let’s say that this module has only one stream: 2.4.
Today, in the modulemd for this module, I declare build and runtime dependencies like this:
dependencies:
buildrequires:
platform: f26
requires:
platform: f26
This works.
Now, imagine that Fedora 27 comes along.
I want the httpd module to be installable on f27, so I update those streams to point to f27. But now I can’t ship updates to f26 anymore.
Just use more streams?
We could just ask packagers to create more streams to deal with this. httpd wouldn’t have a 2.4 stream. It would need to have a f26-2.4 stream and a f27-2.4 stream.
This gets us exactly back into the place we were before we had arbitrary branching. You need to have as many branches for every component as you have releases of the distro (or, as you have “products”).
Solution: “Input” Modulemd Syntax Changes
We’re going to extend the modulemd syntax to allow specifying multiple dependencies in an “input” modulemd (the one that packagers modify). When submitted to the build system, the module-build-service (MBS) will expand these values and under the hood, and submit multiple module builds to itself based on the expansion.
Here are some examples of modulemd files (maintained by humans) that might be submitted to the MBS.
Build on all active streams of the platform module.
dependencies:
buildrequires:
platform: []
Build on only the f27 and f26 platform streams.
dependencies:
buildrequires:
platform: [ f27, f26 ]
Build on all active streams of platform except for the f26 stream.
dependencies:
buildrequires:
platform: [ -f26 ]
The following syntax which builds on only the f26 stream, will still be supported:
dependencies:
buildrequires:
platform: f26
Take the following example (described above):
dependencies:
buildrequires:
platform: [ f27, f26 ]
Submitting this modulemd to the MBS would in turn generate two module builds: one of our httpd module against the f27 stream and another against the f26 stream. Each module build would be associated with its own unique “flattened” output modulemd file that specifies exactly which platform stream it was built against.
New Problems
Having multiple module builds for a single dist-git commit of a modulemd file poses new implementation problems. NSV uniqueness Today, we uniquely identify a module build in a variety of systems with <name>-<stream>-<version> (NSV) where version is derived from the dist-git commit timestamp. Here we’ll have an httpd-2.4 module built on f26 and an http-2.4 module built on f27: two different module builds with the same name, stream, and version. How can we tell them apart?
We will introduce a new value called the context of a built module and include it in the modulemd metadata that gets carried with the built module.
- For user facing cases (dnf installation) it will generally be hidden. The old NSV value will still appear. If a user ever needs to surface the value, client tooling can find it in the modulemd metadata. The additional identifier will give users access to explicit pointers to the content that is installed:
- For cloning a machine.
- For comparing two installed hosts.
- For reporting bugs.
- Some build and compose time systems will have to be modified to use the context as part of a new unique identifier. NSVC will be the <name>-<stream>-<version>-<context>. The systems in question are PDC, koji/brew, pungi, and bodhi.
The context value will be a hash, generating as the first step in the build process (but after expansion). Consider what metadata needs to be hashed: we think that hashing the whole modulemd is problematic, because the modulemd can and will be modified after build time.
Therefore, the context_hash value will need to be derived from only the stuff that uniquely identifies the build and runtime context of the module -- name, stream, version and crucially, its dependencies
Buildtime/Runtime Dep Correlation
Another problem. We now have input modulemd files for a single stream that can be expand buildtime dependencies to to ‘f27’ and ‘f26’, but what about the runtime dependencies?
Consider the following yaml file:
dependencies:
buildrequires:
platform: [f28, f27, f26]
requires:
platform: [f28, f27, f26]
With our thinking caps on, this should obviously generate three module builds (one that buildrequires f28 and run requires f28, a second that buildrequires f27 and run requires f27, and a third for f26). The naive cross-product of all streams is not valuable; the MBS needs some logic to correlate the build time requirements with the run-time requirements. That logic will contain assumptions about how this will be used, and it needs to be well-defined.
Let’s try to define that. Consider a more complicated yaml file that depends on both multiple streams of platform and on multiple hypothetical streams of a “shared-userspace” module:
dependencies:
buildrequires:
platform: [f28, f27, f26]
shared-userspace: [fancy, nonfancy]
requires:
platform: [f28, f27, f26]
shared-userspace: [fancy, nonfancy]
Here are some rules the MBS should obey:
- If a build-time dep contains an expansion, that dep does not have to also appear as a run-time dep.
- If a build-time dep contains an expansion, and if that dep also appears as a runtime dep, the runtime expansion must match the buildtime expansion exactly.
Now that the streams from build-time and runtime can be mapped one-to-one, the cross-product of the deps is taken to produce a set of modules builds. In our example here, we get:
One build with platform f28 and shared-userspace fancy.
- One build with platform f27 and shared-userspace fancy.
- One build with platform f26 and shared-userspace fancy.
- One build with platform f28 and shared-userspace nonfancy.
- One build with platform f27 and shared-userspace nonfancy.
- One build with platform f26 and shared-userspace nonfancy.
We like this.
But, a new corner case emerges! What if the nonfancy branch of shared-userspace is only compatible with f27 and f28, but not with f26? Build #6 will always fail - or, if it doesn’t fail it will be unusable.
We thought through a number of ways that corner cases like these could be handled automatically, and are still working through them. At the moment, we are leaning towards suggesting that a failing corner requires a new stream of the module in question to handle the exclusion. I.e., no convenient syntax to say “all combinations except for this one”.
In this case, you would have two streams of the httpd-2.4 module: httpd-2.4 and httpd-2.4_f26. The first modulemd file in the 2.4 stream would be nice, like this:
dependencies:
buildrequires:
platform: [f28, f27]
shared-userspace: [fancy, nonfancy]
requires:
platform: [f28, f27]
shared-userspace: [fancy, nonfancy]
The second in the 2.4_f26 stream would exist only to support the desired corner case:
dependencies:
buildrequires:
platform: [f26]
shared-userspace: [fancy]
requires:
platform: [f26]
shared-userspace: [fancy