Sgallagh/Changes/ELN Buildroot and Compose

ELN Buildroot and Compose

Summary

ELN is a new buildroot and compose process for Fedora that will take Fedora Rawhide dist-git sources and emulate a Red Hat Enterprise Linux compose. Feedback from this build, compose and integration testing will be provided to Fedora packagers so that they can see the potential impact of their changes on RHEL development.

Owner

• Name: Aleksandra Fedorova
• Email: alpha@bookwar.info
• Name: Stephen Gallagher
• Email: sgallagh@redhat.com

Current status

• Targeted release: Fedora 33
• Last updated: 2020-03-27
• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

This Change supersedes the previously-approved Change to enable an additional buildroot. During development, its scope has expanded to include the entire process of how the Fedora sources are built and composed into shippable artifacts.

ELN is an evolution of the request for an alternate buildroot for newer x86_64 processors. The reasoning behind that new buildroot was that we expected that the next major release of RHEL would likely drop support for older hardware and therefore could take advantage of enhancements and processor extensions available for newer hardware. As plans for this proceeded, they expanded into a desire to do more than just test out the processor architecture. Instead, we want to have an alternative compose of a portion of Fedora Rawhide that will resemble the way that Red Hat and CentOS builds their packages. The idea is that Fedora developers and third-party vendors who rely on Red Hat Enterprise Linux will have a place where they can directly contribute to what will eventually become the next RHEL.

ELN (Enterprise Linux Next) is going to be that place. What we want to do is establish a set of RPM conditionals that can be used for the set of packages that may need to be built differently for a RHEL-like target as compared to a Fedora one. (For example, Fedora often ships with experimental or less-commonly-used features enabled for packages that Red Hat would not want to support for Enterprise Linux customers).

The main goal of ELN is to rebuild Fedora Rawhide as it it were RHEL. See what works, what breaks, what changed. We will be doing this by creating a new set of Koji tags and targets to represent this new buildroot, adjusting some variables in RPM that can be used in conditionals and creating a new compose configuration to produce testable trees. We anticipate only building a relatively small subset of the Fedora Package Collection in ELN: mostly those that will be useful to CentOS and RHEL downstream.

As a sister effort to this, the OSCI team will also be implementing automated tests that will run against ELN composes and provide non-blocking feedback to the package maintainers about potential issues in the Enterprise Linux configuration.

The RPM variables:

• %{dist} will return .eln (no numerical suffix)
• %{fedora} will be unset in ELN. (And thus return nothing)
• %{rhel} will be set and will return a number one higher than the most recent major release of Red Hat Enterprise Linux (at present, that will be 9).
• %{eln} will be set and will expand to %{rhel}

The reasoning behind this approach is so that we break as little as possible when we implement this new build and compose. Existing packages that are using conditionals such as if 0%{?rhel} > 7 will transition over to building "like RHEL" automatically. Any packages that do not have a shared Fedora/RHEL specfile will continue to build as normal. There will be some unknown number of packages whose existing conditionals will need to be updated to account for ELN. At present, we are estimating that there will be around 200 packages that need such attention. The ELN SIG will be providing guidance and pull-requests to assist with this.

The %{eln} variable will be set primarily to allow for the possibility of needing conditionals that apply to ELN only and should not carry over to RHEL, but we expect its use to be exceptionally rare.

Currently, we expect the Koji tagging structure to look like:

eln-candidate: empty tag          # packages get tagged 'eln-candidate' when built
eln: empty tag                    # packages are moved to the 'eln' tag from the gate tag

eln-build:                        # buildroot tag which inherits from Fedora Rawhide
-> eln
-> f33-build

and the target:
Target: eln
Build: eln-build
Dest: eln-candidate

Please note that these tags/targets may change at the recommendation of Release Engineering.

What if packages don't build on ELN?

The vast majority of failed ELN builds will be because of bad %if's, specifically those that are %if's on %{fedora} and/or %{rhel}. There might be a handful that aren't because of bad %if's, but until we start doing ELN builds, we don't really know what they are.

If a package fails to build on ELN, what will happen?

The failure will be looked at and we will evaluate how it can be fixed with the least change and/or impact. Usually this is by changing something like

 %if 0%{?fedora}

to

 %if 0%{?fedora} || 0%{?rhel} >= 9

Or something that is less intuitive

 %if 0%{?fedora} < 32

to

 %if 0%{?fedora} < 32 && 0%{?rhel} <= 8

The fix will be done via a pull request that states a time limit. We want the regular maintainers to see / comment / commit, but we also don't want things to stall for months and get forgotten about.

What if there is a feature RHEL wants in a package, how will that go in?

That is not in the scope of ELN.

What if I do not want to have %if's in my spec files?

As long as your package builds in ELN then just maintain your package like normal. If there is a build failure, the ELN SIG may provide a PR as described above or will discuss alternative approaches on an individual basis with you.

What if I do not want to have %if's in my spec files, but I want to see how changes show up in RHEL?

That is not in the scope of ELN.

Benefit to Fedora

Classically, Fedora sees a flurry of activity from Red Hat developers in the run-up to a new Red Hat Enterprise Linux release. These Red Hat teams frantically attempt to push a few years' downstream effort up into Fedora before it is branched for the next RHEL development. It is not uncommon for these teams to disappear downstream again once this has happened. This harms both Fedora and Red Hat with the lack of upstream involvement. With ELN, we hope to make Fedora Rawhide a more appealing (and canonical) place for those developers to work. In turn, this will make Fedora a more valuable resource for Red Hat, which can help to ensure funding and support.

This new buildroot will allow us to experiment with new features without having a disruptive impact on the existing Fedora user-base. One example of such would be exploring setting a new processor baseline for performance improvements in server use-cases. Another could be the usage of link-time-optimization.

Scope

• Proposal owners:
  • Setup build environment for a new disttag (eln).
  • Define and publish the Content Set for ELN, which will be a subset of Fedora Rawhide content that will be useful for the upcoming RHEL major release (presently 9)
  • Setup automation to trigger new ELN build every time there is a new update submitted to Fedora Rawhide.
  • Post build result to Fedora Messaging, so that it appears in Bodhi and can be used as a gating test.
  • Setup automation to run periodic partial composes (via ODCS) without installation media to generate repositories with these packages.
  • Update packaging documentation to mention new disttag and how it can be used.
  • Setup ELN SIG. The SIG will
    • own the scope of the ELN component set;
    • own the scope of the eln disttag;
    • own the configuration of the compose;
    • maintain infrastructure required to build packages and composes;
    • track the status of package builds and composes;
    • work with other package maintainers to resolve the issues;
    • keep track of the eln-conditionals in RPM spec files.

• Other developers:

  Anyone who wants to produce different content for the ELN compose will need to implement the new macros in their specfile. The overwhelming majority of packages will require no modification.

• Release engineering: #9154

• Policies and guidelines: #955

• Trademark approval: not needed. There is no new shippable artifact.

Upgrade/compatibility impact

N/A

Though it is a System Wide change it doesn’t impact Fedora upgrades, as it is a development environment tracking Fedora Rawhide.

How To Test

There is going to be a development snapshot of the state of the ELN repositories.

It is under discussion whether this snapshot will have its own installation media. For now the preferred way to test ELN composes would be to use standard Fedora Rawhide images and then include ELN as an additional repository.

User Experience

ELN artifacts will be made available for testing and development purposes, but we will not be shipping any content produced from ELN directly to the general public (such as on the standard mirror network or via getfedora.org).

Dependencies

There is no dependency on other changes, though some of the infrastructure services like ODCS composes and Fedora CI will be involved.

Contingency Plan

• Contingency mechanism: If this effort doesn’t land in Fedora 33 we review it and decide whether it needs to be moved further to Fedora 34 or be cancelled.
• Contingency deadline: N/A as it is a continuous process in Fedora Rawhide and it is not pinned to a specific release.
• Blocks release? No
• Blocks product? No

Additional risk: Infrastructure resources taken by ELN builds can create load on Fedora build system

Mitigation: In case of such load we cancel or reschedule ELN tasks (package and compose builds and tests) on request from the Fedora Infrastructure team. At least initially, we will also set ELN tasks to low-priority, similar to how COPR does it.

Documentation

https://fedoraproject.org/wiki/ELN

See also previous discussions:

• "Alternative buildroot as a development tool" @ devel

Release Notes

N/A

Though it is a System-Wide Change it has no user-facing component. We may announce it through other channels.