Changes/ModularRelease

Modular Release

Summary

The build, release, distribution, and update changes associated with and required for the [Changes/Modular_Server] and [Changes/Host_and_Platform] Changes.

Owner

• Name: Ralph Bean
• Email: rbean@redhat.com
• Release notes owner:
• Edition: Fedora Server
• Responsible WG: Modularity WG

Current status

• Targeted release: Fedora 27
• Last updated: 2017-07-25
• Tracker bug: #1474935

Detailed Description

Preamble

This change is intended to cover the workflow and technical tooling aspects of a “modular” release for F27.

There are other Changes that are not part of the scope of this Change, but which are related:

• Changes/Modular_Server is a proposal to replace the Fedora Server edition with a modular release for F27.
• Changes/Host_and_Platform deals with the content of what goes into the core of the modular release for Fedora Server (the Modular Server) in F27.

This Change is about how we’re going to get those two other changes through the infra/build/release tooling.

Client tooling is being worked on (I have seen some cool demos from the May-June timeframe. Ask about them!), but client tooling is not part of the scope of this Change proposal.

Note that there currently are no proposals to replace either Fedora Workstation or Fedora Atomic with modular revisions. This means that here we cannot move to replace existing workflows wholesale, but have to look at maintaining the two flows (traditional and modular) in tandem, for at very least the F27 release cycle.

(The Changes/ArbitraryBranching change is also related to this proposal, but not covered in any further detail here.)

Builds

Module builds for the F27 cycle will look much like they did for the F26 cycle. No major changes here.

See the F26 Change document for details here, Changes/ModuleBuildService

As soon as the FPC approves the Module:Guidelines, we can open up the MBS for use by the general packager group.

Dependencies:

• We need the Modularity team to take the module guidelines to the FPC, work through them, and get an agreed-upon version approved.
• We are indirectly dependent on release engineering to create some initial tags for bootstrapping the host and platform content. This should be referenced in that Change proposal.

Automation

Ok, I lied. One thing about builds will change: we’re going to automate rebuilds with Infrastructure/Factory2/Focus/Freshmaker.

Module Rebuilds

For F26, if a packager wanted to update an rpm in a module, they would:

• Patch the spec file of that rpm, commit, and push.
• Switch to a checkout of the module that included that rpm, and run mbs-build submit which would kick off the appropriate builds.
• If another module included that same rpm stream, and the packager forgot about it, then they’re just out of luck.

For F27, we’re working on a system to automate this. The workflow will be:

• Patch the spec file of an rpm, commit, and push.
• Watch the fireworks.

The freshmaker daemon will notice the commit, then look up all modules that depend on that rpm stream. It will submit requests to the module-build-service to build those modules.

We won’t have a nice UI for this for F27, but we will have a JSON API provided by freshmaker to query and find the list of module builds that were triggered as a result of your commit (or anyone else’s commits to any other packages).

There are some exceptions here. We will have a site-wide policy configured for freshmaker to not do automated rebuilds for a blacklisted set of modules. This blacklist set must include the bootstrap module. It includes many thousands of rpm streams and an update to any one of them would cause a mass-rebuild of (nearly) every other module. This is too much, so we’ll instead rely on the bootstrap maintainers and release engineering to only request these rebuilds via MBS by hand.

Container Rebuilds

We're approaching container rebuilds in two phases: for short hand, we're calling them the "slow" flow and the "fast" flow. We'll do the slow flow first for F27. The fast flow is a stretch goal for F27, but will more likely land in F28.

In the slow flow, we automatically kick off rebuilds of containers when rpms that previously went into those containers are shipped to the updates repo in Bodhi. The lag time here can be around a week to two weeks. Freshmaker will watch on fedmsg to find when those rpms make it to the master mirror and will kick off the appropriate builds. The container rebuild process should already be pulling from that repo; so we should be good to go.

In the fast flow, we automatically kick off rebuilds of containers when rpms that previously went into those containers are first added to an update in Bodhi. The lag time here can be quite fast. As soon as you make a specfile patch and the rpms get built, the update can be created which in turns triggers freshmaker to kick off container rebuilds.

fast flow container rebuilds require a yum repository with the rpms to be available before they are mashed into the updates repo. For this, we're building Infrastructure/Factory2/Focus/ODCS. This "on demand compose service" will be used by freshmaker to produce repos of rpm and module content for container rebuilds, as well as taskotron test runs.

No further details on ODCS here, but it does require non-standard write access to a subtree of /mnt/koji. We will need support from Fedora Release Engineering and Fedora Infrastructure in the request-for-resources ticket for this.

Dependencies:

• We’ll need Fedora Infrastructure to kindly support us through the request-for-resources process.
• We need Fedora Infrastructure to finish implement service-account support for OpenID Connect (in ipsilon).
• After that, we need to be granted an OIDC service account so that freshmaker can submit builds to the module-build service.
• The VM for ODCS will need non-standard write access to a subtree of /mnt/koji (we don't need to write to the whole thing, just a new /mnt/koji/compose/odcs/ directory where we can write out temporary composes).

Composes

We will continue to do general composes with the same tool - pungi.

• For rawhide, the compose configuration should list the master streams of all modules to be included in the rawhide compose. The latest builds in those streams will be included in each nightly compose: https://pagure.io/pungi-fedora/blob/master/f/variants-modular.xml
• For branched, the compose configuration should list f27 streams of all modules to be included in the F27 server compose. The latest builds in those streams will be included in each nightly compose.
• For the alpha and beta candidates, release engineering will need to pin the versions of the module streams in the variants-module.xml file in the f27 branch of the pungi-fedora repo. This will allow them to stabilize the release and allow for systematic QA and go/no-go decisions.

As mentioned in the automation section above, we're introducing a new tool called the ODCS to generate temporary, throwaway composes for automated testing and automated layered image creation.

Dependencies:

• The image building work from F26. For F26 "Boltron", we wanted to produce base images but ran into more issues than expected. Most all of those are resolved now. Full completion of that work (being able to produce base images in koji via pungi from modular content) is a hard requirement for F27, where it was a soft requirement for F26.
• The variants files for F27. The Factory2 team will produce these and submit them to release-engineering's pungi-fedora repo. Releng will need to review and eventually merge.
• As the beta cycle unfolds, Release Engineering will need to freeze the module versions in variants-modular.xml, adding in new versions only to address blocker bugs (like how we do today with the f26-compose tag).

Updates

Updates to the (modular) Fedora Server release will be managed by Bodhi.

Fun facts:

• The Bodhi database model now has multi-type (modular) support.
• The Bodhi API now has multi-type (modular) support.
• The Bodhi bindings and CLI and web UI now have multi-type (modular) support.

The only thing left to do is (take a deep breath) the masher.

See Infrastructure/Factory2/Focus/Bodhi for how to plan to handle the code changes to Bodhi.

As for configuration, we’re going to need to be able to mash updates for traditional style Fedora Workstation and Fedora Atomic while also mashing updates for (Modular) Fedora Server.

We will need:

• f27-updates, f27-updates-testing and all the normal associated -pending and -candidate tags for Workstation and Atomic. The F27 “release” in Bodhi’s database will point at this tags, as per usual.
• f27-modular-updates, f27-modular-updates-testing, and similar kinds of -pending and -candidate tags for Server. We will then need to additionally create a separate Modular F27 “release” in Bodhi’s database that points to those tags.

The masher will produce separate modular-updates and modular-updates-testing repos alongside the traditional updates and updates-testing repos.

Note - the work we've already done here paves the way for containers in Bodhi, but a workflow that has layered image updates passing through Bodhi is likely not in the cards for F27. Someone will have to take it up for F28.

Dependencies:

• We need the content generator flags enabled in koji for the MBS (see https://pagure.io/releng/issue/6799 ).
• We need the tags hierarchy created for modular updates.
• We need the release created in the bodhi DB (for the modular repos) at the same time that the traditional release is created.
• Release-engineering runs the push command. There will inevitably be problems with the first iteration of our modularized bodhi masher. Please work with the Factory2 team to solve them as we move forwards with the release.

Mirrors

We don’t expect that mirroring this content will require any changes to mirrormanager or mirrorlist.

We will be creating an extra compose and an extra set of updates repos, but there are no fundamental changes to the mirroring process. We will have an additive increase in load on mirrors, but no an exponential one.

There has been lots of worry here. Let me explain. Once upon a time, we thought that (with Modularity) we would distribute every version of every module ever built. Users could switch between any of them, at any time, since they would all be available on the mirrors. This would explode our storage ask of the mirror volunteers.

But, this is not how our modularized bodhi masher will work. We will distribute multiple streams of the same rpms, but we will not retain or mirror multiple versions of the same stream. Only the latest builds of a module stream will be included in the updates repo, not every version we ever built.

The F27 server compose should go exactly where it would normally go in the published tree.

The F27 modular-updates repos should sit right next to the traditional updates and updates-testing repos.

Benefit to Fedora

Enable Changes/Host_and_Platform and Changes/Modular_Server.

Scope

• Proposal owners: We have to finish and deploy freshmaker, our bodhi changes, including changes to pungi. We'll be actively working with the infrastructure and releng teams on deployment and configuration.

• Other developers: Is there any work required of other developers? No. They will still be able to use the "traditional" workflow to build and ship content for Fedora Workstation and Fedora Atomic. To participate in the development of Fedora Server, they'll need to learn how to create new modules (the module guidelines should cover this).

• Release engineering: ticket #6852

• • List of deliverables:

I don't think anything on this list changes. See Changes/Modular_Server.

• Policies and guidelines: See Changes/Modular_Server.

• Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

See Changes/Modular_Server.

How To Test

If the user can install the Fedora 27 Server and we can deliver updates, then it is a success.

User Experience

See Changes/Modular_Server.

Dependencies

See the dependencies bullets in the "Detailed Description" section above.

Contingency Plan

• Contingency mechanism: Since we're going to keep the traditional flow for Workstation and Atomic for F27, we should also have the chance to ditch all this stuff and produce Server in the same way at any point (provided we have enough time to stabilize the compose).
• Contingency deadline: 2017-08-29 (the Bodhi activation point).
• Blocks release? Yes
• Blocks product? Fedora Server

Documentation

See Infrastructure/Factory2 and the modularity docs.