Koschei

Overview

Koschei (https://apps.fedoraproject.org/koschei) is a continuous integration service for RPM packages that tracks package dependency changes in Fedora Rawhide and rebuilds packages whose dependencies change too much. Packages are rebuilt in Koji from the latest available SRPM and the rebuilds are logged to display the dependency changes and current state to users.

Usage

The web interface displays a current buildability state of tracked packages. Packages can be in different states:

• OK - package builds fine
• Failing - package failed to build
• Unresolved - package's dependencies couldn't be resolved, Koschei won't even attempt to build it
• Blocked - package is blocked in Koji, Koschei won't attempt to build it
• Unknown - package was just added and the buildability state is not known yet

Clicking on a package name displays package detail page which shows latest package builds with links to Koji logs and shows dependency changes compared to previous build. Builds done by packagers in Koji are also logged by Koschei and displayed with light green background. Dependency changes show the dependency name, previous version and current version. If the previous version is missing, it means that the dependency was just introduced. If the current version is missing, the dependency was dropped. The dependency changes are ordered by the distance in the dependency graph, so the ones at the top should be the most relevant ones.

Adding packages

Any package in Fedora can be added to Koschei entering the package name in the Koschei web interface.

Priorities

Packages are rebuilt when they reach certain priority threshold and packages with higher priority are rebuilt before the ones with lower priority. The priority value is calculated from multiple sources:

• Updates of package dependencies, which are valued according to their distance in the dependency graph
• Elapsed time since last successful build (logarithmic formula)
• State of last two builds - packages which started to fail recently are prioritized to rule out temporary factors
• Static priority - set by Koschei admins to make some packages more important
• Manual priority - set by users in the manual priority field in package detail. By setting the priority manually, you can force a package to be rebuilt by Koschei sooner (or later by setting it negative). The advantage over submitting a scratch-build yourself is that it won't need to build a new SRPM from SCM and you'll see the dependency differences since the last build.

If there is no priority displayed, it means that the package is not viable for scheduling, because has a build already running or it's dependencies cannot be resolved. If there are packages with high priority, but builds are not being scheduled, it might mean that Koji load is too high.

Package groups

Packages can be categorized to groups, such as Java or Ruby. The groups are currently maintained by Koschei admins and it's not possible to edit them directly. Group manipulation can be done by contacting Koschei maintainers. Proper group editing support is planned for the next version of Koschei.

Your packages

When you log in, Koschei can sync package information from dist-git. Under the "My packages" tab, you can then see the state of all packages that you have any ACL on (including watch). You can also display packages of any other user by altering the URL which contains the FAS username.

Notifications

Koschei sends fedmsg notifications when the state of a tracked package changes (For example when it starts to fail). Notifications to users are not enabled by default, but you can subscribe to them in Fedora Notifications app to receive messages via IRC or email. There are two Koschei rules defined in FMN:

• Continuous integration state changes for a package (koschei) - Matches all package state changes.
• Particular Koschei package groups - Matches package state changes for packages in certain Koschei groups.

Those filters are intended to be combined with generic filters, such as "A particular user's packages (any acl)" to receive package state change messages only for particular user.

Example notification filter configuration

Reporting FTBFS bugs through Koschei

Package detail contains a link "File new FTBFS bug" that can be used to report a bug in Bugzilla. Clicking it redirects you to Red Hat's Bugzilla New bug page with prefilled component, summary and description, which you can then edit manually and submit. It doesn't submit anything by itself and it currently doesn't check for duplicates.

Setting arch override

Arch override is a way to specify on which architecture the package should be built. This also affects noarch packages which are otherwise built on any machine that is available. Arch override can be set in the package detail view. Valid values are architectures supported by Koji ('i386', 'x86_64', 'armhfp', 'aarch64', 'ppc64', 'ppc64le'), or any space-separated list of them (for example 'x86_64 ppc64le aarch64'). You need to be logged in to alter this setting. Please don't use this feature unless it's necessary.

Rationale

The problem

Given package at given moment can be successfully built, or not. This will be called package buildability in this document. Situation when package doesn't build will be called FTBFS (from "fails to build from source").

Build process often involves code compilation and/or checking dependencies, but not only. All packages are recommended to run unit or integration tests as part of their build process, be it in %check or %build sections of rpm spec file. Some kinds of bugs that are caught by unit tests should cause FTBFS too. This means that checking buildability is a way of running package tests.

Package buildability is one of means of measuring and controlling software quality. Many software components are shipped once they successfully build -- without any additional sanity tests.

Constantly growing number of packages makes it more difficult to deliver good quality software. One example is software collections, which once introduced can increase number of packages very quickly.

Package maintainers are often unaware of their packages FTBFS. They don't find out until next mass rebuild, or until an urgent bug must be fixed. In the former case fixing FTBFS problem must be done before fixing the other bug, which delays delivering the fix. Koschei tries to minimize time between package becoming FTBFS and maintainer finding out about this event.

Time elapse

Time elapse is a very negative factor when it comes to fixing FTBFS bugs, for several reasons.

One reason is that with time people tend to forget what they were working on. If some FTBFS bug was introduced due to some change, it is much easier to revert or rework the change immediately. After time the person working on it will simply forget most of the details and will have to spend more time and other resources to fix the bug.

Secondly, with time more bugs may appear. These bugs shadow each other, making them more difficult do discover and fix, compared to fixing several bugs separately. Additionally bugs in different packages often depend on each other and in order to fix them maintainer has to work on several packages at the same time, greatly reducing overall efficiency.

From the above it seems obvious that FTBFS bugs should be fixed as soon as they appear.

Design

The solution to the problem described above seems to be continuous integration of packages included in distribution. Package buildability should be monitored continuously. Interested parties should be able to obtain up-to-date information about current buildability status of individual packages or group of packages and they should be informed about each change in buildability of packages they care about.

It would be nice if continuous integration system also provided some additional information useful in diagnosing FTBFS problems.

One of questions that appear is how often packages should be rebuilt.

• Rebuilding all packages every week is a simple idea, but the delay of a few days is still too long for fixing FTBFS bugs quickly.
• More important packages could be rebuilt more often, for example every day. But this way only small number of packages can be rebuilt.
• Packages could be rebuilt after they reverse dependencies change. But this would require way more resources than available.

Another question that comes up is where to rebuild packages. There are several possibilities, like maintainers' own machines, Fedora Koji, Copr or Fedora cloud. Each of them has their advantages and disadvantages.

Authors of Koschei believe that the best place to rebuild packages as part of continuous integration effort is the official Fedora Koji infrastructure. It's an existing, working and stable platform. It has quite a lot of spare resources, which are not used except for short periods of time, such as during mass rebuilds. It is maintained by Fedora infrastructure team, so no additional manpower is needed. Data transfer over network is not a problem as the most up-to-date repositories are available in Koji LAN. Lastly, Koji defines canonical Fedora build environment, so using it minimizes probability of false positive and true negative ratios when detecting FTBFS.

Dependency resolution

Dependency resolution is done via hawkey library, which is also used by DNF. That will allow us to mimic the resolution happening in Koji as closely as possible. Resolution for build scheduling is done using the latest repo and submitted builds are resolved again as they complete. The reason for that is that we have no control over exact repo used to build the package and therefore we cannot safely assume it would be the same as the one we have as the latest.

Structure

Koschei has access to relational database where it keeps information about packages, package groups, dependencies and builds.

There are several components that access Koschei database concurrently. Each service can be ran on separate machine and communicate over network with database. Koschei scheduler monitors Koji load and submits scratch builds when there are enough resources available. Koschei watcher keeps track of running builds and updates package statuses when builds finish. It also updates package dependencies as they are updated in Koji.

Environment

Koschei was implemented with respect to current standards, especially Fedora Infrastructure best practices. It is implemented in Python. It uses PostgreSQL with SQLAlchemy for ORM and Alembic for database migration. It is built in modular way. Each service provides a systemd unit.

Koschei is proud to be free software and it uses only free dependencies. The code is licensed under GPL v2 or later. Source code and documentation is available on GitHub.

Links

• Production instance
• Mikolaj Izdebski, Michael Simacek - Koschei contact persons
• #fedora-admin^[?], #fedora-apps^[?] - IRC channels
• Staging instance (used for testing upcoming releases)
• Koschei source code repository at GitHub
• presentation about Koschei (source)