Abstract
Koschei is a service for scratch-rebuilding RPM packages in Fedora Koji instance when their build-dependencies change or after some time elapse.
This document describes the problem Koschei is trying to solve, design decisions, system structure and current status.
The problem
Given package at given moment can be successfully built, or not. This will be called package buildability in this document. Situation when package doesn't build will be called FTBFS (from "fails to build from source").
Build process often involves code compilation and/or checking
dependencies, but not only. All packages are recommended to run unit
or integration tests as part of their build process, be it in %check
or %build sections of rpm spec file. Some kinds of bugs that are
caught by unit tests should cause FTBFS too. This means that checking
buildability is a way of running package tests.
Package buildability is one of means of measuring and controlling software quality. Many software components are shipped once they successfully build -- without any additional sanity tests.
Constantly growing number of packages makes it more difficult to deliver good quality software. One example is software collections, which once introduced can increase number of packages very quickly.
Package maintainers are often unaware of their packages FTBFS. They don't find out until next mass rebuild, or until an urgent bug must be fixed. In the former case fixing FTBFS problem must be done before fixing the other bug, which delays delivering the fix. Koschei tries to minimize time between package becoming FTBFS and maintainer finding out about this event.
Time elapse
Time elapse is a very negative factor when it comes to fixing FTBFS bugs, for several reasons.
One reason is that with time people tend to forget what they were working on. If some FTBFS bug was introduced due to some change, it is much easier to revert or rework the change immediately. After time the person working on it will simply forget most of the details and will have to spend more time and other resources to fix the bug.
Secondly, with time more bugs may appear. These bugs shadow each other, making them more difficult do discover and fix, compared to fixing several bugs separately. Additionally bugs in different packages often depend on each other and in order to fix them maintainer has to work on several packages at the same time, greatly reducing overall efficiency.
From the above it seems obvious that FTBFS bugs should be fixed as soon as they appear.
Usage
Notifications
Koschei sends fedmsg notifications when the state of a tracked package changes (For example when it starts to fail). You can subscribe to those in Fedora Notifications app to receive messages via IRC or email. There are two Koschei rules defined in FMN:
- Continuous integration state changes for a package (koschei) - Matches all package state changes.
- Particular Koschei package groups - Matches package state changes for packages in certain Koschei groups.
Those filters are intended to be combined with generic filters, such as "A particular user's packages (any acl)" to receive package state change messages only for particular user.
Design
The solution to the problem described above seems to be continuous integration of packages included in distribution. Package buildability should be monitored continuously. Interested parties should be able to obtain up-to-date information about current buildability status of individual packages or group of packages and they should be informed about each change in buildability of packages they care about.
It would be nice if continuous integration system also provided some additional information useful in diagnosing FTBFS problems.
One of questions that appear is how often packages should be rebuilt.
- Rebuilding all packages every week is a simple idea, but the delay of a few days is still too long for fixing FTBFS bugs quickly.
- More important packages could be rebuilt more often, for example every day. But this way only small number of packages can be rebuilt.
- Packages could be rebuilt after they reverse dependencies change. But this would require way more resources than available.
Another question that comes up is where to rebuild packages. There are several possibilities, like maintainers' own machines, Fedora Koji, Copr or Fedora cloud. Each of them has their advantages and disadvantages.
Authors of Koschei believe that the best place to rebuild packages as part of continuous integration effort is the official Fedora Koji infrastructure. It's an existing, working and stable platform. It has quite a lot of spare resources, which are not used except for short periods of time, such as during mass rebuilds. It is maintained by Fedora infrastructure team, so no additional manpower is needed. Data transfer over network is not a problem as the most up-to-date repositories are available in Koji LAN. Lastly, Koji defines canonical Fedora build environment, so using it minimizes probability of false positive and true negative ratios when detecting FTBFS.
Implementation
Koschei is a tool for continuously scratch-rebuilding packages using Fedora build infrastructure -- Koji.
Koschei tracks set of packages, which can be any subset of Fedora packages. It reports package buildability to interested parties via web interface, RPC and fedmsg. Packages are rebuilt on Koji according to internal priorities, as available resources allow.
Priority
Packages tracked by Koschei have priories according to which Koschei schedules them for rebuild. Priority changes with time and is reset after given package is rebuilt. Priority depends on several factors:
Time since last rebuild. Package priority increases with time. The more time elapsed since last rebuild the more likely the package is to be rebuilt by Koschei.
Dependency changes. Package priority is increased when one of build dependencies changes (usually when it is updated to newer version, but also downgraded). Transitive build dependencies are also considered, but priority bump depends on distance in dependency graph to given dependency. Updates of distant indirect dependencies have less effect on package priority than updates of direct dependencies. Additionally each package in minimal buildroot (@build group in Koji comps) is considered as build dependency of every package.
Previous state. Packages that are known to be FTBFS are scheduled for rebuild more likely than packages which are in non-FTBFS status. The reason for this is attempt to save packager time. People usually check only build failures and do nothing when rebuild is successful. Rebuilding package by Koschei is assumed to cost less than having an engineer look at the failure. By eliminating temporary build failures more quickly, human resources can be saved.
Package importance. Each package has some base priority. Some packages are more important than others and should be rebuilt more often, while some other packages require a lot of resources to build and therefore should be rebuilt less often to save resources for other packages. Package importance allows for priority adjustments depending on specifics of individual packages.
Manual trigger. Sometimes it is useful to force rebuild of particular package. For this purpose priority can be adjusted by one-time manual boost. A boost high enough will result in immediate package rebuild.
Plugins. Koschei supports external plugins which can affect package priorities. Interested parties can write their own plugins which would implement priority computation specific to some types of packages.
Structure
Koschei has access to relational database where it keeps information about packages, package groups, dependencies and builds.
There are several components that access Koschei database concurrently. Each service can be ran on separate machine and communicate over network with database. Koschei scheduler monitors Koji load and submits scratch builds when there are enough resources available. Koschei watcher keeps track of running builds and updates package statuses when builds finish. It also updates package dependencies as they are updated in Koji. Koschei reporter generates HTML report which contains information about package statuses. Koschei RPC interface allows Koschei admins to control many aspects of the service. Diagram shows the overall Koschei architecture.
Environment
Koschei was implemented with respect to current standards, especially Fedora Infrastructure best practices. It is implemented in Python. It uses PostgreSQL with SQLAlchemy for ORM and Alembic for database migration. It is built in modular way. Each service provides a systemd unit.
Koschei is proud to be free software and it uses only free dependencies. The code is licensed under GPL v2 or later. Source code and documentation is available on Github.
Current status
Koschei is packaged as RPM. It is available in official Fedora repositories.
Current instance of Koschei is being hosted at Fedora cloud, but there are plans to make it an official Fedora service.
Links
- Koschei instance at Fedora cloud
- Koschei source code repository at Github
- presentation about Koschei (source)