Infrastructure 2020/EPEL-8

This is a shorthand list of items we are needing to cover for building EPEL-8 in Fedora Infrastructure for the Fiscal Year 2020.

Document Details

• Status:WORKING DRAFT:
• Point of contact: Stephen Smoogen
• Implementation Schedule: 2019-07-01 for beta
• Release Date: 2019-10-01

Goals

• Make a version of EPEL available by end of March for the beta of RHEL-8.
  • Version will work out tools needed to make builds possible using modules
  • Version will test new compose rules for EPEL trees using a Stuff and Updates tree.
  • Version will test out new compose paths
• Make a larger version of EPEL-8 available soon after the General Availability of RHEL-8
  • Assume that for 8.0 that builds of modules may not be available but tested in alternate tree
• Make EPEL-8 fully able to build modular sets by release of RHEL-8.1
  • Test scripts to archive old releases and compose new ones at this time.

Background

With Red Hat Enterprise Linux (RHEL) 8 Beta released, it is time to evaluating making a version of EPEL for it. As with any major RHEL release, there are major changes which affect what can and can not be done in a release. Where in RHEL-7 there were changes in the init program moving from Upstart to systemctl, this release has added modular rpm package sets which change how packages can be built.

Modules are a tool which allows one to have parallel availability of different package sets. One can set up a system and choose to have php-7.2 OR php-7.4 installed with appropriate sub-modules as needed. These are done via the dnf command which has the needed logic to work out needed dependencies and conflicts. In order for the koji system to know this, there is also the Module Build System (MBS) which interprets and keeps track of this higher level repo data. For Fedora packages and modules which are completely built within koji this is all put together, however EPEL packages are built from artifacts which koji has no idea about. These are the Red Hat Enterprise Linux packages which koji and the build systems see as outside repositories. Since these packages were not built in the Fedora Build Infrastructure, neither koji nor MBS know enough about the packages which would cause build failures at times.

In order to work around this additional tools need to be built which can ferret out the information from the RHEL metadata in order to allow modules to be both pulled into various build roots and also have additional modules built on top of those pre-existing modules.

Strategic Fit

EPEL is used by a very large number of systems around the world and is part of the lifecycle of packages moving from Fedora Linux into the Red Hat Enterprise Linux ecosystem. Having Fedora packages and tools available in EPEL is seen as useful for both Fedora and Red Hat in maintenance of their userbase.

People involved

This project impacts a number of application or other projects. Here is a quick list of the different projects involved and their point of contact.

• Coordinator: Stephen Smoogen
• Impacted: Matthew Miller
• Release Engineering:
• Additional Tool Development:
• CentOS Coordination: Brian Stinson

Assumptions

The following assumptions are in place:

1. Fedora will be able to use the following RHEL-8 channels to build EPEL Packages
   1. Base
   2. Appstream
   3. Code Ready Linux Builder (CRLB)
2. Data for determining modules can be easily determined so that createrepo can be used
   1. This data can be used to build non-modular and modular packages
   2. This data will consumable
3. That EPEL will not need to rebuild additional packages which are made from src.rpms for the packages in Base, Appstream, CRLB
   1. EPEL will look at using CentOS packages when they become available to fill those gaps.
4. EPEL users are slow to migrate to new versions. This allows us some time to experiment.
5. EPEL users are vocal about problems. This means that while emotional taxing, we will get feedback quickly.

Additional Links

• Old Website for EPEL: https://fedoraproject.org/wiki/EPEL
• Place where EPEL will move to: https://pagure.io/epel/
• Red Hat Enterprise Linux Packages: ftp://ftp.redhat.com/redhat/rhel/rhel-8-beta
• Red Hat Code Ready Linux Builder: https://developers.redhat.com/blog/2018/11/15/introducing-codeready-linux-builder/
• Fedora Modularity Documentation: https://docs.fedoraproject.org/en-US/modularity/
• Fedora Modularity Upstream: https://docs.pagure.org/modularity/

Task Breakdown

1. Determine the buildroot requirements. The buildroot is the bare minimum packages that are needed to build the majority of packages. While it can be based on the Fedora 29 minimum buildroot, the EPEL-8 one has some major caveats.
   1. While we are able to use the packages from RHEL-8, we are not able to share them with the world. This means that koji can not import the packages but only knows about them via external repositories.
   2. There are specific rules and limitations in Koji rules for el-8 hidden external build roots
   3. In addition to not knowing enough about the packages in RHEL-8, the buildsystem has no knowledge about the modules in them. This will require gathering data, and then importing fake modules from RHEL-8 into MBS.
   4. After these are done, we can start to determine what default packages/modules are and what will be in a minimal buildroot.
   5. Write further tools which will take 8.x RHN channel and creates a 'composed' buildroot for koji to work on. It would be useful for the tool to report when packages are being over-written by ones koji does know about so that we can stop packages getting replaced by koji built ones.
   6. Both modularity and koji developers feel there will be further problems that will need solutions but what they are needs the above to be put in place.
2. Determine new directory layout for compose/ship. A proposal has been sent to the EPEL mailing lists for feedback which would change directory structure for EPEL-7 and possibly 6 as the tools to compose builds should be uniform for least maintenance.
   • /pub/epel/releases/8.x/Stuff
   • /pub/epel/releases/8.x/Modular
   • /pub/epel/updates/8.x/Stuff
   • /pub/epel/updates/8.x/Modular
3. Need to set up appropriate branches and tool changes for EPEL-8.
   1. Release engineering tools ( bodhi, koji, pdc, pkgs and many other tools) will need to know about the new branches and targets.
   2. Each branch for EPEL has caused problems with parts of the Fedora build system in the past. Various tools and cleanup have to be discovered to work around it.
   3. Fedpkg will need code changes and updates in various existing source code places
4. Determine architectures to build against
   1. Easy to build
      • x86_64
      • ppc64le
      • aarch64
   2. Hard to build
      • s390x
      • i386
      • arm32
5. Currently the Code Ready Linux Builder is only available to accounts with RHN and it only has a subset of packages which were used to build RHEL-8 with. There may be packages which EPEL will need which need to be added to CRLB or we need to work out a method of communicating our 'over-rides' with.
6. Need to consult Bugzilla maintainers for changes needed
   1. Package name
   2. Business rules
7. Establish policies for which packages are branched into EPEL-8
   • A problem with current builds in EPEL-6 and EPEL-7 are when a package is in x86_64 but is not in other architecture channels. In the past, EPEL has allowed for the rebuild of the required packages for those architectures. However this causes problems because now koji KNOWS about this rebuild and will use that one even if there is a newer NVR in the RHEL-8 channels.
8. Establish policies for how modules are branched into EPEL-8
   • EPEL and RHEL may have the same modules
   • EPEL and RHEL may not have module stream collisions (aka EPEL streams must be named clearly they are from EPEL).
9. Setup policies for package 7->8 bootstrapping (w/o review). Some packages going into EPEL need a review and other times they do not. The policies here need to be revamped as it has slowed down getting bootstrapping in the past.
10. Establish tool to block packages in 8.0
11. See what we can move back to RHEL-7

Future Plans

Phase 0

1. Meet with consumers and producers of the current EPEL tools
   • Get current problems they want fixed
   • Get expectations of how consumers want to get packages
   • Get expectations of how producers want to put packages in EPEL
   • Talk with MBS and koji developers on what is needed for any successful builds with existing infrastructure
2. Put together proposals for changes to EPEL-6/7 trees to meet problems that affect EPEL-8
3. Put together talk for 2019 Devconf.cz with Kevin Fenzi https://smooge.fedorapeople.org/EPEL-Next-201901/

Phase 1 (2019-03-15)

1. Work with Patrick Uiterwijk on standalone tools to build modular packages using RHEL-8 outside of Red Hat build system
2. Write up plans and post to mailing lists and wiki
3. Publish tools and gather feedback
4. Find out gotcha's and other problems

Phase 2 (2019-04-15)

1. Put tools into Fedora Build System
2. Prepare composer and tools to build packages in /pub/alt/epel versus /pub/epel
3. Run tests of building select set of F29 packages and modules with RHEL-8 beta
4. Publish results and gather developer and consumer feedback
5. Look to having 80% of current packages in EPEL-7 rebuilt with beta tools.

Phase 3 (2019-??-??)

1. Work on issues found in Phase 2 that affect either modular/non-modular packages
2. Wait for RHEL-8 General Availability
3. Import packages for RHEL-8.0 into specific trees
4. Setup changes in Fedora Build System
5. Notify packagers that branches to EPEL-8 are possible.
6. Build first sets of packages and compose EPEL-8.0 release tree.
7. Gather feedback and put in projects for required projects.

Phase 4 (20??-??-??)

1. When next minor RHEL-8 version is released begin process for next rollover and compose changes.
2. Roll out fixes from Phase 4.

End of Project

This project is considered finished when RHEL-8.1 has been released and EPEL-8.1 has been successfully composed. At this point future changes will be in other projects.