Koschei

Overview

Koschei is a continuous integration service for RPM packages that tracks package dependency changes in Fedora Rawhide and rebuilds packages whose dependencies change too much. Packages are rebuilt in Koji from the latest available SRPM and the rebuilds are logged to display the dependency changes and current state to users.

Usage

The web interface displays a current buildability state of tracked packages. Packages can be in different states:

• Ok (Green tick sign) - package builds fine
• Failed (Red minus sign) - package failed to build
• Unresolved (Red cross) - package's dependencies couldn't be resolved, Koschei won't even attempt to build it
• Unknown (Question mark) - package was just added and the buildability state is not known yet

Clicking on a package name displays package detail page which shows latest package builds with links to Koji logs and shows dependency changes compared to previous build. Builds done by packagers in Koji are also logged by Koschei and displayed with light green background. Dependency changes show the dependency name, previous version and current version. If the previous version is missing, it means that the dependency was just introduced. If the current version is missing, the dependency was dropped. The dependency changes are ordered by the distance in the dependency graph, so the ones at the top should be the most relevant ones.

Adding packages

Any package in Fedora can be added to Koschei in the "Add packages" tab. It accepts a list of whitespace separated package names, which will be checked for existence in Koji. You need to be logged in.

Priorities

Packages are rebuilt when they reach certain priority threshold and packages with higher priority are rebuilt before the ones with lower priority. The priority value is calculated from multiple sources:

• Updates of package dependencies, which are valued according to their distance in the dependency graph
• Elapsed time since last succesful build (logarithmic formula)
• State of last two builds - packages which started to fail recently are prioritized to rule out temporary factors
• Static priority - set by Koschei admins to make some packages more important
• Manual priority - set by users in the manual priority field in package detail. By setting the priority manually, you can force a package to be rebuilt by Koschei sooner (or later by setting it negative). The advantage over submitting a scratch-build yourself is that it won't need to build a new SRPM from SCM and you'll see the dependency differences since the last build.

If there is no priority displayed, it means that the package is not viable for scheduling, because has a build already running or it's dependencies cannot be resolved. If there are packages with high priority, but builds are not being scheduled, it might mean that Koji load is too high.

Package groups

Packages can be categorized to groups, such as Java or Ruby. The groups are currently maintained by Koschei admins and it's not possible to edit them directly. Group manipulation can be done by contacting Koschei maintainers. Proper group editing support is planned for the next version of Koschei.

Notifications

Koschei sends fedmsg notifications when the state of a tracked package changes (For example when it starts to fail). You can subscribe to those in Fedora Notifications app to receive messages via IRC or email. There are two Koschei rules defined in FMN:

• Continuous integration state changes for a package (koschei) - Matches all package state changes.
• Particular Koschei package groups - Matches package state changes for packages in certain Koschei groups.

Those filters are intended to be combined with generic filters, such as "A particular user's packages (any acl)" to receive package state change messages only for particular user.

Rationale

The problem

Given package at given moment can be successfully built, or not. This will be called package buildability in this document. Situation when package doesn't build will be called FTBFS (from "fails to build from source").

Build process often involves code compilation and/or checking dependencies, but not only. All packages are recommended to run unit or integration tests as part of their build process, be it in %check or %build sections of rpm spec file. Some kinds of bugs that are caught by unit tests should cause FTBFS too. This means that checking buildability is a way of running package tests.

Package buildability is one of means of measuring and controlling software quality. Many software components are shipped once they successfully build -- without any additional sanity tests.

Constantly growing number of packages makes it more difficult to deliver good quality software. One example is software collections, which once introduced can increase number of packages very quickly.

Package maintainers are often unaware of their packages FTBFS. They don't find out until next mass rebuild, or until an urgent bug must be fixed. In the former case fixing FTBFS problem must be done before fixing the other bug, which delays delivering the fix. Koschei tries to minimize time between package becoming FTBFS and maintainer finding out about this event.

Time elapse

Time elapse is a very negative factor when it comes to fixing FTBFS bugs, for several reasons.

One reason is that with time people tend to forget what they were working on. If some FTBFS bug was introduced due to some change, it is much easier to revert or rework the change immediately. After time the person working on it will simply forget most of the details and will have to spend more time and other resources to fix the bug.

Secondly, with time more bugs may appear. These bugs shadow each other, making them more difficult do discover and fix, compared to fixing several bugs separately. Additionally bugs in different packages often depend on each other and in order to fix them maintainer has to work on several packages at the same time, greatly reducing overall efficiency.

From the above it seems obvious that FTBFS bugs should be fixed as soon as they appear.

Design

The solution to the problem described above seems to be continuous integration of packages included in distribution. Package buildability should be monitored continuously. Interested parties should be able to obtain up-to-date information about current buildability status of individual packages or group of packages and they should be informed about each change in buildability of packages they care about.

It would be nice if continuous integration system also provided some additional information useful in diagnosing FTBFS problems.

One of questions that appear is how often packages should be rebuilt.

• Rebuilding all packages every week is a simple idea, but the delay of a few days is still too long for fixing FTBFS bugs quickly.
• More important packages could be rebuilt more often, for example every day. But this way only small number of packages can be rebuilt.
• Packages could be rebuilt after they reverse dependencies change. But this would require way more resources than available.

Another question that comes up is where to rebuild packages. There are several possibilities, like maintainers' own machines, Fedora Koji, Copr or Fedora cloud. Each of them has their advantages and disadvantages.

Authors of Koschei believe that the best place to rebuild packages as part of continuous integration effort is the official Fedora Koji infrastructure. It's an existing, working and stable platform. It has quite a lot of spare resources, which are not used except for short periods of time, such as during mass rebuilds. It is maintained by Fedora infrastructure team, so no additional manpower is needed. Data transfer over network is not a problem as the most up-to-date repositories are available in Koji LAN. Lastly, Koji defines canonical Fedora build environment, so using it minimizes probability of false positive and true negative ratios when detecting FTBFS.

Priority

Packages tracked by Koschei have priories according to which Koschei schedules them for rebuild. Priority changes with time and is reset after given package is rebuilt. Priority depends on several factors:

Time since last rebuild. Package priority increases with time. The more time elapsed since last rebuild the more likely the package is to be rebuilt by Koschei.

Dependency changes. Package priority is increased when one of build dependencies changes (usually when it is updated to newer version, but also downgraded). Transitive build dependencies are also considered, but priority bump depends on distance in dependency graph to given dependency. Updates of distant indirect dependencies have less effect on package priority than updates of direct dependencies. Additionally each package in minimal buildroot (@build group in Koji comps) is considered as build dependency of every package.

Previous state. Packages that are known to be FTBFS are scheduled for rebuild more likely than packages which are in non-FTBFS status. The reason for this is attempt to save packager time. People usually check only build failures and do nothing when rebuild is successful. Rebuilding package by Koschei is assumed to cost less than having an engineer look at the failure. By eliminating temporary build failures more quickly, human resources can be saved.

Package importance. Each package has some base priority. Some packages are more important than others and should be rebuilt more often, while some other packages require a lot of resources to build and therefore should be rebuilt less often to save resources for other packages. Package importance allows for priority adjustments depending on specifics of individual packages.

Manual trigger. Sometimes it is useful to force rebuild of particular package. For this purpose priority can be adjusted by one-time manual boost. A boost high enough will result in immediate package rebuild.

Plugins. Koschei supports external plugins which can affect package priorities. Interested parties can write their own plugins which would implement priority computation specific to some types of packages.

Structure

Koschei has access to relational database where it keeps information about packages, package groups, dependencies and builds.

There are several components that access Koschei database concurrently. Each service can be ran on separate machine and communicate over network with database. Koschei scheduler monitors Koji load and submits scratch builds when there are enough resources available. Koschei watcher keeps track of running builds and updates package statuses when builds finish. It also updates package dependencies as they are updated in Koji. Koschei reporter generates HTML report which contains information about package statuses. Koschei RPC interface allows Koschei admins to control many aspects of the service. Diagram shows the overall Koschei architecture.

Environment

Koschei was implemented with respect to current standards, especially Fedora Infrastructure best practices. It is implemented in Python. It uses PostgreSQL with SQLAlchemy for ORM and Alembic for database migration. It is built in modular way. Each service provides a systemd unit.

Koschei is proud to be free software and it uses only free dependencies. The code is licensed under GPL v2 or later. Source code and documentation is available on Github.

Current status

Koschei is packaged as RPM. It is available in official Fedora repositories.

Current instance of Koschei is being hosted at Fedora cloud, but there are plans to make it an official Fedora service.

Links

• Koschei instance at Fedora cloud
• Koschei source code repository at Github
• presentation about Koschei (source)