Changes/ELN Buildroot and Compose

ELN Buildroot and Compose

Summary

ELN is a new buildroot and compose process for Fedora that will take Fedora Rawhide dist-git sources and emulate a Red Hat Enterprise Linux compose. Feedback from this build, compose and integration testing will be provided to Fedora packagers so that they can see the potential impact of their changes on RHEL development.

Owner

• Name: Aleksandra Fedorova
• Email: alpha@bookwar.info
• Name: Stephen Gallagher
• Email: sgallagh@redhat.com

Current status

• Targeted release: Fedora 33
• Last updated: 2020-03-31
• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

This Change supersedes the previously-approved Change to enable an additional buildroot. During development, its scope has expanded to include the entire process of how the Fedora sources are built and composed into shippable artifacts.

ELN is an evolution of the request for an alternate buildroot for newer x86_64 processors. The reasoning behind that new buildroot was that we expected that the next major release of RHEL would likely drop support for older hardware and therefore could take advantage of enhancements and processor extensions available for newer hardware. As plans for this proceeded, they expanded into a desire to do more than just test out the processor architecture. Instead, we want to have an alternative compose of a portion of Fedora Rawhide that will resemble the way that Red Hat and CentOS builds their packages. The idea is that Fedora developers and third-party vendors who rely on Red Hat Enterprise Linux will have a place where they can directly contribute to what will eventually become the next RHEL.

ELN (Enterprise Linux Next) is going to be that place. What we want to do is establish a set of RPM conditionals that can be used for the set of packages that may need to be built differently for a RHEL-like target as compared to a Fedora one. (For example, Fedora often ships with experimental or less-commonly-used features enabled for packages that Red Hat would not want to support for Enterprise Linux customers).

The main goal of ELN is to rebuild Fedora Rawhide as if it were RHEL. See what works, what breaks, what changed. We will be doing this by creating a new set of Koji tags and targets to represent this new buildroot, adjusting some variables in RPM that can be used in conditionals and creating a new compose configuration to produce testable trees. We anticipate only building a relatively small subset of the Fedora Package Collection in ELN: mostly those that will be useful to CentOS and RHEL downstream.

ELN builds are not RHEL builds, to be clear. They are builds done in Fedora with a RHEL-like build configuration. Successful builds of ELN will be synced to the RHEL internal build system. At this sync point, we can apply additional changes that may have been unacceptable for Fedora (such as trademarks). We would like to minimize the divergence between ELN and RHEL, but we understand that this may not always be possible.

As a sister effort to this, the OSCI team will also be implementing automated tests that will run against ELN composes and provide non-blocking feedback to the package maintainers about potential issues in the Enterprise Linux configuration.

The RPM variables:

• %{dist} will return .eln (no numerical suffix)
• %{fedora} will be unset in ELN. (And thus return nothing)
• %{rhel} will be set and will return a number one higher than the most recent major release of Red Hat Enterprise Linux (at present, that will be 9).
• %{eln} will be set and will expand to %{rhel}

The reasoning behind this approach is so that we break as little as possible when we implement this new build and compose. Existing packages that are using conditionals such as if 0%{?rhel} > 7 will transition over to building "like RHEL" automatically. Any packages that do not have a shared Fedora/RHEL specfile will continue to build as normal. There will be some unknown number of packages whose existing conditionals will need to be updated to account for ELN. At present, we are estimating that there will be around 200 packages that need such attention. The ELN SIG will be providing guidance and pull-requests to assist with this.

The %{eln} variable will be set primarily to allow for the possibility of needing conditionals that apply to ELN only and should not carry over to RHEL, but we expect its use to be exceptionally rare.

Currently, we expect the Koji tagging structure to look like:

eln-candidate: empty tag          # packages get tagged 'eln-candidate' when built
eln: empty tag                    # packages are moved to the 'eln' tag from the gate tag

eln-build:                        # buildroot tag which inherits from Fedora Rawhide
-> eln
-> f33-build

and the target:
Target: eln
Build: eln-build
Dest: eln-candidate

Please note that these tags/targets may change at the recommendation of Release Engineering.

What if packages don't build on ELN?

The vast majority of failed ELN builds will be because of bad %if's, specifically those that are %if's on %{fedora} and/or %{rhel}. There might be a handful that aren't because of bad %if's, but until we start doing ELN builds, we don't really know what they are.

Will you be adding new %{rhel} or %{eln} conditionals to packages?

Regarding %{eln}, we intend not to use this variable if at all possible (an exception might be the kernel).

For packages that build successfully in ELN as-is, we won't be proposing new conditionals initially (with limited exceptions for packages such as kernel after discussion with the maintainers).

If a package fails to build on ELN, what will happen?

The failure will be looked at and we will evaluate how it can be fixed with the least change and/or impact.

Often this will be by changing an existing conditional such as

 %if 0%{?fedora}

to

 %if 0%{?fedora} || 0%{?rhel} >= 9

Or something that is less intuitive

 %if 0%{?fedora} < 32

to

 %if 0%{?fedora} < 32 && 0%{?rhel} <= 8

Such fixes will be done via a pull request that states a time limit. We want the regular maintainers to see / comment / commit, but we also don't want things to stall for months and get forgotten about.

For less trivial fixes, we may provide a pull-request or use other methods to communicate with the maintainers to determine a path forwards that is acceptable to them.

What if there is a new upstream feature RHEL wants in a package, how will that go in?

That is not in the scope of ELN.

What if I do not want to have %if's in my spec files?

As long as your package builds in ELN then just maintain your package like normal. If there is a build failure, the ELN SIG may provide a PR as described above or will discuss alternative approaches on an individual basis with you. The ELN SIG will not dictate how you must proceed, but will try to find a solution that you are comfortable with.

Can we do this in a branch instead of in master?

This adds no value to the current approach where Red Hat maintainers would manually merge their changes into the internal build infrastructure. There's no way to automate the sync from the master branch to the eln branch that wouldn't break and require maintainer involvement. Attempting to branch only individual packages would introduce significant complexity in the build process as well, leading to far more opportunity for bugs. Lastly, even the most diligent of maintainers can forget to sync every change to a new branch, thus leaving us in a situation where the eln branch has fallen behind and is no longer providing an accurate view of whether the package is still building or functioning in that environment.

What if I do not want to have %if's in my spec files, but I want to see how changes show up in RHEL?

If your package builds fine in ELN, then you can do that. If it would require conditionals to build in ELN and you do not wish to put them there, but you want to see how changes appear in RHEL, you've set up a mutually-exclusive set of requirements. You will need to find your own personal way to test those changes without taking advantage of ELN.

Benefit to Fedora

Classically, Fedora sees a flurry of activity from Red Hat developers in the run-up to a new Red Hat Enterprise Linux release. These Red Hat teams frantically attempt to push a few years' downstream effort up into Fedora before it is branched for the next RHEL development. It is not uncommon for these teams to disappear downstream again once this has happened. This harms both Fedora and Red Hat with the lack of upstream involvement. With ELN, we hope to make Fedora Rawhide a more appealing (and canonical) place for those developers to work. This should increase the amount of ongoing developer involvement in Fedora, and also make Fedora a more valuable resource for Red Hat, which can help to ensure funding and support.

This new buildroot will allow us to experiment with new features without having a disruptive impact on the existing Fedora user-base. One example of such would be exploring setting a new processor baseline for performance improvements in server use-cases. By doing this in ELN, we gain a place to test whether the proposed changes to using AVX2 as a minimum has real performance impact which we could then look at bringing back to standard Fedora (as a new variant, perhaps). Or, if it proves that the performance gains really aren't worthwhile (or are nonexistent), we have learned that without interfering with other Fedora users.

Another similar example could be the usage of link-time-optimization.

Scope

• Proposal owners:
  • Setup build environment for a new disttag (eln).
  • Define and publish the Content Set for ELN, which will be a subset of Fedora Rawhide content that will be useful for the upcoming RHEL major release (presently 9)
  • Setup automation to trigger new ELN build every time there is a new update submitted to Fedora Rawhide.
  • Post build result to Fedora Messaging, so that it appears in Bodhi and can be used as a gating test.
  • Setup automation to run periodic partial composes (via ODCS) without installation media to generate repositories with these packages.
  • Create documentation for ELN compose and ELN project
  • Update packaging documentation to mention new disttag and how it can be used.
  • Setup ELN SIG. The SIG will
    • own the scope of the ELN component set;
    • own the scope of the eln disttag;
    • own the configuration of the compose;
    • maintain infrastructure required to build packages and composes;
    • track the status of package builds and composes;
    • work with other package maintainers to resolve the issues;
    • keep track of the eln-conditionals in RPM spec files.

• Other developers:

  Anyone who wants to produce different content for the ELN compose will need to implement the new macros in their specfile. The overwhelming majority of packages will require no modification.

• Release engineering: #9154

• Policies and guidelines: #955

• Trademark approval: not needed. There is no new shippable artifact.

Upgrade/compatibility impact

N/A

Though it is a System Wide change it doesn’t impact Fedora upgrades, as it is a development environment tracking Fedora Rawhide.

How To Test

There is going to be a development snapshot of the state of the ELN repositories.

It is under discussion whether this snapshot will have its own installation media. It depends on availability of resources for ODCS infrastructure, but also on ability of the ELN SIG to verify the installation media.

For now the preferred way to test ELN composes would be to use standard Fedora Rawhide images and then include ELN as an additional repository.

User Experience

ELN artifacts will be made available for testing and development purposes, but we will not be shipping any content produced from ELN directly to the general public (such as on the standard mirror network or via getfedora.org). This does not necessarily mean that they will be shipped directly from Koji, merely that they won't come from the standard locations.

Documentation on how to access ELN project repositories, on how to contribute to ELN configurations and on how to join ELN SIG will be available.

Dependencies

There is no dependency on other changes, though some of the infrastructure services like ODCS composes and Fedora CI will be involved.

Contingency Plan

• Contingency mechanism: If this effort doesn’t land in Fedora 33 we review it and decide whether it needs to be moved further to Fedora 34 or be cancelled.
• Contingency deadline: N/A as it is a continuous process in Fedora Rawhide and it is not pinned to a specific release.
• Blocks release? No
• Blocks product? No

Additional risk: Infrastructure resources taken by ELN builds can create load on Fedora build system

Mitigation: In case of such load we cancel or reschedule ELN tasks (package and compose builds and tests) on request from the Fedora Infrastructure team. At least initially, we will also set ELN tasks to low-priority, similar to how COPR does it.

Documentation

https://fedoraproject.org/wiki/ELN

See also previous discussions:

• "Alternative buildroot as a development tool" @ devel

Release Notes

N/A

Though it is a System-Wide Change it has no impact on Fedora 33.