Infrastructure/Factory2/Focus/MBS

From November 2016

What is a Focus Document?

The Factory 2.0 team produces a confusing number of documents. The first round was about the Problem Statements we were trying to solve. Let’s retroactively call them Problem Documents. The Focus Documents (like this one) focus on some system or some aspect of our solutions that cut across different problems. The content here doesn’t fit cleanly in one problem statement document, which is why we broke it out.

Background on the MBS

• MBS is a build service. Unsurprising!
• It has an HTTP frontend where you can POST new module builds and GET status.
• It does not have a visible web UI (for now, we may add one some day. KISS.)
• Originally written by the Modularity team, and is now developed by the Factory 2.0 team.
• It has had many names: “Řida” and “FM-Orchestrator”.
• We renamed it MBS to try and better describe it.
• It uses a variety of backends to do the heavy-lifting.
• It can be thought of as automation on top of those backends.

Links:

• Production instance
• Source code
• Fedora Change
• Discussion on the Fedora devel list.
• FESCo meeting minutes.

Where do modules even come from?

You know how when you build an rpm in Koji or Brew you do so from a .spec file that lives in dist-git? Well, we carved out namespaces in dist-git to allow us to store not just specfiles, but also docker files. We will be using that same namespace separation to store modulemd yaml files in dist-git. For an example, see the base-runtime definition or this testmodule definition for a simpler one.

We’re not there yet, but the Modularity Working Group is planning on working with the Fedora Packaging Committee to develop guidelines, policies, and processes for how modulemd yaml files are written and what kind of reviews they need to pass before they can be included in the distribution.

How do you submit a build?

In short, you do an HTTP POST to the endpoint where it receives new builds. Take a look at this script and the accompanying JSON file that we use for development on local instances of the MBS to get an idea for how it works. You tell the MBS the scmurl of a modulemd yaml file that you want it to build, and it takes it from there.

Of course, we don’t expect packagers/engineers to POST to this thing with curl! Karsten Hopp developed patches to rpkg (which is the library that undergirds the well-known fedpkg and rhpkg commands) which will allow us all to submit builds with fedpkg module-build in the future. Sound familiar?

Eventually, we plan to reach a place where new commits to dist-git (either to the modulemd definition of a module itself or to the spec files of a component that is included in that module) will automatically trigger new rebuilds of the module.

What happens when you submit a build?

Let’s walk through how a module build works when the MBS is configured to submit to Koji:

• The modulemd definition is validated and enriched by the frontend.
  • Your modulemd yaml file will include references to some components like glibc or python-requests. It will furthermore specify that the module wants a particular git ref of those components. The validation step here includes checking out those git refs to make sure they actually exist before we waste time submitting stuff to koji, clogging it up unnecessarily.
  • Your modulemd yaml file furthermore will not specify its own version string like you usually do in a .spec file. The MBS will enrich the submitted metadata by assigning it stream and version identifiers that make sense.
• Once the file has been validated and enriched, the MBS publishes a message to the message bus stating that it has recorded this new request in its database. It marks the module as being in the Wait state, and it returns HTTP 201.
• While the MBS is waiting, another service receives its message bus announcement and sets to work. The pdc-updater daemon receives the notice and makes a note in PDC. The primary responsibility here is to assign a new unused Koji tag for this module build, and to do so in a way that isn’t susceptible to the fragility of hard-coded strings in the future.
• Meanwhile, the MBS backend daemon wakes up and considers whether or not to start submitting work to Koji. Once the new unique Koji tag is available from PDC, it sets to work. The first step is to initialize the buildroot for this module build:
  • It creates koji “tags”. One build tag, and one resultant tag.
  • Link in dependency tags
  • Define build and srpm-build groups
  • Build a module-build-macros srpm
• After the buildroot is initialized, the MBS starts an iterative process of rebuilding all the components from source.
  • The components are grouped by a buildorder defined in the modulemd yaml file which groups the components into batches. The first batch is built first, then the repo is regenerated. Then the second batch is built, then the repo is regenerated, etc..
• If any component fails in any batch, then the module build fails as a whole.
• If all of the components successfully build, then the module is marked as being in the “done” state (which is not the final state).
• External to the MBS, tests are run (in taskotron and/or jenkins) and reported to resultsdb. When tests have passed sufficiently, MBS transitions the module to the final “ready” state, meaning that it is ready to be consumed by a pungi compose.

What about local builds?

We recently have implemented a local mock backend for the MBS. This means that you can clone the repo, run a local instance, and submit module builds to it all without touching koji or the real build system. This is indispensable for rapidly working on modules and for working on the MBS itself.

You can submit a local module build by running the following from the git repo:

$ python manage.py build_module_locally file:///path/to/module/repo

Nicer client tooling coming soon!

Writing additional backends.

Take a look in builder.py. You’ll see a GenericBuilder base class with some abstract methods. All you need is to provide a new class that extends GenericBuilder and implement the missing methods.