On Demand Compose Service (a.k.a. “ODCS”)
What is a Focus Document?
The Factory 2.0 team produces a confusing number of documents. The first round was about the Problem Statements we were trying to solve. Let’s retroactively call them Problem Documents. The Focus Documents (like this one) focus on some system or some aspect of our solutions that cut across different problems. The content here doesn’t fit cleanly in one problem statement document, which is why we broke it out.
Motivation
Currently, there is no API which can be used by other services or even people to generate temporary composes with limited content. There is increasing need for such composes from various reasons:
- To rebuild Docker images automatically with updated packages (for example when there is OpenSSL security update), we need a repository containing the updated packages, so we can point Koji to take the packages from these repositories without waiting for the packages to appear in the official public repository.
- To test modules right after the build, QA team needs a compose containing the built module together with all the modules this module depends on, so QA team is able to install the module and run the tests.
Furthermore:
- In the mid-/long-term, Fedora releng would like to generate the main compose from the smaller composes generated by the ODCS, so the composing would be faster.
- Current composes are also not event-based. For example in Fedora, the composes are built even when nothing changed on input side of a compose.
On Demand Compose Service
In the first phase of deployment, the ODCS should have the following features:
- The service should have thin REST API providing following functionality:
- Create new temporary compose defined by list of Koji tags + specific list of packages from these tags or modules. The compose will be built by Pungi using an automatically generated configuration file. The compose will be removed after certain time which can be overwritten in this REST call. Returns request_id to track progress. When compose is generated, emits fedmsg. If possible, the service should reuse the already created compose when the same one is requested by another service/user.
- Get a status of a compose by the request_id. Returns “wait”, “done”, “removed” states. For “done” composes, returns the link to compose top-level directory and expiration time..
- Get list of all composes in “wait” or “done” states.
- Delete the temporary compose/repository defined by request_id even before the expiration time.
- Authentication and authorization. POST requests that trigger new composes should be limited to a whitelist of users and/or FAS/ldap groups.
- These temporary composes will be available over HTTP, stored in /mnt/koji. Since the service composes only based on the Koji builds, binary RPMs will be hardlinked in the compose and therefore the storage needed is quite low - only for the repository metadata and Pungi compose specific data.
In the second phase of deployment, the ODCS should have the following features:
- The service should listen on fedmsg/UMB for events on which it can generate the compose automatically
- This should be configurable, eventually use Greenwave (Policyengine) to find out if it should create the compose.
- The service should allow recreating older expired composes with exactly the same RPMs from Koji.