ELN Buildroot and Compose

Summary

The goal of the ELN project is to continuously build Fedora Rawhide packages and composes in the way which resembles the CentOS and RHEL build process and to provide a feedback loop for Fedora maintainers on the status of those builds.

Owner

Name: Aleksandra Fedorova
Email: alpha@bookwar.info
Name: Stephen Gallagher
Email: sgallagh@redhat.com

Current status

Targeted release: Fedora 33
Last updated: 2020-03-26
FESCo issue: <will be assigned by the Wrangler>
Tracker bug: <will be assigned by the Wrangler>
Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

This Change supersedes the previously-approved Change to enable an additional buildroot. During development, its scope has expanded to include the entire process of how the Fedora sources are built and composed into shippable artifacts.

ELN is an evolution of the request for an alternate buildroot for newer x86_64 processors. The reasoning behind that new buildroot was that we expected that the next major release of RHEL would likely drop support for older hardware and therefore could take advantage of enhancements and processor extensions available for newer hardware. As plans for this proceeded, they expanded into a desire to do more than just test out the processor architecture. Instead, we want to have a complete alternative compose of Fedora Rawhide that resembles the way that Red Hat and CentOS builds their packages. The idea being that Fedora developers and third-party vendors who rely on Red Hat Enterprise Linux have a place where they can directly contribute to what will eventually become the next RHEL.

ELN (Enterprise Linux Next) is going to be that place. What we want to do is establish a set of RPM conditionals that can be used for the set of packages that may need to be built differently for a RHEL-like target as compared to a Fedora one. (For example, we may want to skip regenerating documentation during package-build and instead use pre-built docs from the tarball or SRPM.)

This includes:

buildroot configuration, rpm macro and compile flags,
comps files and the compose content,
compose itself and the pipeline which builds it.

Under this umbrella we are going to have:

RPM variables for conditionals to allow tweaking rpm spec files
A new Koji tag which allows control of the buildroot,
A fork of the pungi configuration which describes the compose configuration,
A pipeline which builds a periodic compose based on that configuration,
Storage for such composes and additional pipelines which will verify the quality of the compose and test additional scenarios on top of it.

The RPM variables:

%{dist} will return .eln (no numerical suffix)
%{fedora} will be unset in ELN.
%{rhel} will be set and will return a number one higher than the most recent major release of Red Hat Enterprise Linux (at present, that will be 9).
%{eln} will be set and will expand to %{rhel}

The reasoning behind this approach is so that we break as little as possible when we implement this new build and compose. Existing packages that are using conditionals such as if 0%{?rhel} > 7 will transition over to building "like RHEL" automatically. Any packages that do not have a shared Fedora/RHEL specfile will continue to build as normal. There will be some unknown number of packages using conditionals like if 0%{?rhel} that will need to be updated to specify if 0%{?rhel} <= 8.

The %{eln} variable will be set primarily to allow for the possibility of needing conditionals that apply to ELN only and should not carry over to RHEL, but we expect its use to be exceptionally rare.

Benefit to Fedora

Who benefits from the implementation of this feature:

Fedora Infrastructure
- As we are going to try and test new, more easily maintainable infrastructure for composes.
Fedora Minimization
- As we are going to modify the compose content according to minimization goals and provide feedback to Fedora maintainers on the results
CentOS Stream, EPEL and RHEL
- We are going to build Fedora packages into a compose similar to the multi-repo structure of CentOS and RHEL.
Fedora Community
- The feedback pipelines built for the project will allow downstream developers to open up their work and bring it closer to Fedora. In particular, it will enable projects like OpenShift to do their work in Fedora.
- Making Fedora friendlier to its downstream will justify bringing more resources and more contributions to Fedora Rawhide. So that downstream not only tries to catch up with Fedora going forward, but actively participates in Fedora development and testing.
- The tooling developed for this project will allow further development of changes in Fedora Infrastructure without blocking the regular Fedora packaging and release process.

Scope

Proposal owners:
- Setup build environment for a new disttag (eln).
- Define and publish the Content Set for ELN, which will be a subset of Fedora Rawhide content that will be useful for the upcoming RHEL major release (presently 9)
- Setup automation to trigger new ELN build every time there is a new update submitted to Fedora Rawhide.
- Post build result to Fedora Messaging, so that it appears in Bodhi and can be used as a gating test.
- Setup automation to run periodic partial composes (via ODCS) without installation media to generate repositories with these packages.
- Update packaging documentation to mention new disttag and how it can be used.
- Setup ELN SIG. The SIG will
  - own the scope of the ELN component set;
  - own the scope of the eln disttag;
  - own the configuration of the compose;
  - maintain infrastructure required to build packages and composes;
  - track the status of package builds and composes;
  - work with other package maintainers to resolve the issues;
  - keep track of the eln-conditionals in RPM spec files.

Other developers:
Anyone who wants to produce different content for the ELN compose will need to implement the new macros in their specfile. The overwhelming majority of packages will require no modification.

Release engineering: #9154

Policies and guidelines: #955

Trademark approval: not needed. There is no new shippable artifact.

Upgrade/compatibility impact

N/A

Though it is a System Wide change it doesn’t impact Fedora upgrades, as it is a development environment tracking Fedora Rawhide.

How To Test

There is going to be a development snapshot of the state of the ELN repositories.

It is under discussion whether this snapshot will have its own installation media. For now the preferred way to test ELN composes would be to use standard Fedora Rawhide images and then include ELN as an additional repository.

User Experience

There is no user-facing part in this change. No ELN artifacts are going to be shipped to the end-user.

Dependencies

There is no dependency on other changes, though some of the infrastructure services like ODCS composes and Fedora CI will be involved.

Contingency Plan

Contingency mechanism: If this effort doesn’t land in Fedora 33 we review it and decide whether it needs to be moved further to Fedora 34 or cancelled.
Contingency deadline: N/A as it is a continuous process in Fedora Rawhide and it is not pinned to a specific release.
Blocks release? No
Blocks product? No

Additional risk: Infrastructure resources taken by ELN builds can create load on Fedora build system

Mitigation: In case of such load we cancel or reschedule ELN tasks (package and compose builds and tests) on request from the Fedora Infrastructure team. At least initially, we will also set ELN tasks to low-priority, similar to how COPR does it.

Documentation

https://fedoraproject.org/wiki/ELN

Release Notes

N/A

Though it is a System-Wide Change it has no user-facing component. We may announce it through other channels.

Search

Sgallagh/Changes/ELN Buildroot and Compose

Contents