Changes/GatingRawhidePackages

Gating Rawhide Packages

Summary

We want to gate packages on test results before they can land in rawhide. This will reduce the amount of broken dependency, uninstallable packages and broken composes leading to a more stable rawhide as well as less work on the infrastructure and rel-eng teams to keep composes working.

Owner

• Name: Pierre-Yves Chibon
• Email: pingou - pingoured.fr

People who are/will be involved in this:

• Coordinator: Pierre-Yves Chibon
• Bodhi: Randy Barlow, Aurélien Bompard and Clément Verna
• koji: Mikolaj Izdebski and Mike McLean
• fedpkg/rpkg: Chenxiong and Lubomír Sedlář
• Fedora CI: Dominik Perpeet
• Release notes owner:

Current status

• Targeted release: N/A
• Targeted time period: <To be updated week 10 2019>
• Last updated: 2019-04-10
• Tracker bug: #1698501

Detailed Description

Rawhide is an unique place in the Fedora eco-system. It is the only place where large changes and rebuilds can be done. In stable Fedora releases, the rebuilds of large sets of packages is discouraged by the packaging guidelines, so the ratio of number of updates in bodhi which a single build vs multiple builds (95% to 5%) is not reflective of the reality of the rawhide ecosystem.

The basic principal of this proposal is to provide an environment in which packages can be built and tested without affecting other packages.

When considering gating rawhide package updates on test results, we need to consider two workflows: single build updates, multi-builds updates. For single build, the easiest solution is to provide a dedicated koji tag in which these packages are built and where they will wait for their tests to pass before they can enter the buildroot. For multi-builds, the solution will be to rely on side-tags in koji. These side-tags are basically tags created by the packager and available to them to do their work, in this case rebuild all the packages desired. The side-tag can then sent to the test system as being one unit of change.

To use a, now well-known, analogy, a single-build update is like sending a commit to a mailing list, it waits there to be reviewed and tested before being merged into the main repository. Multi-build updates are like pull-requests, they can contain one or more builds and which are reviewed and tested all together as one change before being merged.

Within this proposal, we aim at building the infrastructure allowing to gate packages in rawhide. The goal is for packages to go through bodhi, be tested and if the tests pass, land in the rawhide buildroot as they do today. In the simplest case, the packager workflow will not be affected by this proposal, less simple situation will require adjustements to the packager workflow that we would like to try to be as minimal as possible.

We do not aim at making any tests mandatory as part of these proposals. Once packages gating is in place it will be up to the community and likely FESCo to decide if they would like to enforce some rules on all packages and which ones.

Note that this proposal completes previous initiative such as Changes/NoMoreAlpha.

Workflows

• 

  Single package update with gating

• 

  Multi-packages update with gating

Benefit to Fedora

• As more packagers opt-into gating and add tests to their packages, rawhide becomes more stable
• More stable rawhide will lead to easier composes and a smoother release process
• Ability to use chain-build in rawhide and stable releases

Scope

The scope of this work is adding a mechanism to gate builds/updates before they enter the rawhide buildroot (and thus become accessible to others) so broken packages are kept out and cannot affect other packages or the compose until they are fixed by their maintainers.

The CI system, the tests and the decision on which tests are used to gate upon are out of scope for the present document.

• Proposal owners: pingou will be coordinating the work among the different stack holders

• Other developers:
  • Bodhi developers: Implement the needed changes
  • koji developers: Implement the needed changes
  • fedpkg/rpkg developers: Implement the needed changes
  • infrastructure: deploy the new version of bodhi and koji

• Release engineering: #8183

• Policies and guidelines: Once the entire system is in place, FESCo may want to set a policy on distribution-wide gating (ie: tests that would be enforced for all packages in the distributions). This is however out of scope of this proposal.

• Trademark approval: N/A (not needed for this Change)

Application impacted

Bodhi

• A message-bus listener will automatically create a bodhi update for each package built in the rawhide-gated tag.
• A message-bus listener will automatically push bodhi updates created for rawhide that have passed CI (the decision being announced by Greenwave).
• A message-bus listener will automatically comment on bodhi update about update in greenwave's decision
  • This will notify users about the test results and the corresponding gating status.
• bodhi will support creating a new update from a side-tag instead of a list of builds.

Koji

• A koji plugin will allow packagers to create new side-tags
• How side-tags are created will be optimized so they do not take hours to be available to the user requesting them

fedpkg/rpkg

• The command: fedpkg side-tag create will be added to create a new side-tag in koji for this user using the new plugin
• The command: fedpkg side-tag merge will be added to create a bodhi update corresponding from this side-tag
  • Alternatively: Expand the existing fedpkg update command to add support for creating a new update from a side-tag

Greenwave / WaiverDB

Nothing should change for these tools but we will have to confirm this and test in staging that they behave as expected.

CI system

Nothing should change for the CI system but we will have to confirm this and test in staging that they behave as expected.

Upgrade/compatibility impact

N/A

How To Test

Single build

With opting in into gating

1. Add tests to a package you maintain in Fedora using the ci/quick-start-guide documentation (see the full CI documentation for more information).
2. Make your package gated on some tests using package-specific policies (in a gating.yaml)
3. Update your package in rawhide
4. Ensure that the bodhi update is automatically created
5. Check that the tests have properly ran (see the Tests tab in the bodhi update)
6. Ensure that the bodhi update went through to rawhide once the tests passed

If one or more tests failed and you wish to waive them

1. Waive the failing tests using bodhi updates waive <id> see bodhi updates waive --help for more information

Without opting in into gating

1. Ensure your package does not have a gating.yaml in its dist-git repository
2. Build your package in rawhide as you do today
3. Ensure that the bodhi update is automatically created
4. Ensure that the bodhi update goes through to rawhide automatically

Multi-builds

1. Add tests to a package you maintain in Fedora using the ci/quick-start-guide documentation (see the full CI documentation for more information).
2. Make your package gated on some tests using package-specific policies (in a gating.yaml)
3. Request a new side-tag in koji for your rebuilds
4. Update your packages in this side-tag
5. Request for this side-tag to be merged either using fedpkg or bodhi's UI (create a new update from a side-tag)
6. Check that the tests have properly ran (see the Tests tab in the bodhi update)
7. Ensure that the bodhi update went through to rawhide once the tests passed
8. Check that the side-tag was correctly cleaned afterward

Without opting in into gating

1. Ensure all your packages do not have a gating.yaml in its dist-git repository
2. Build your packages in rawhide as you do today
3. Ensure that the bodhi updates are automatically created
4. Ensure that the bodhi updates go through to rawhide automatically

User Experience

Musts

Here below are the list of requirements for this proposal:

• Packagers must be notified when the update corresponding to their build(s) passes or fails CI
• Packagers must be able to see what is blocking an update (ie missing requirements/test results)
• Packagers must be able to re-trigger a test run (via either an UI or a CLI)
• Packagers must be able to waive false-negative (via either an UI or a CLI)

Nice to have

Here below is a list of ideas that would make the user experience more friendly but will not be part of the initial release:

• Packagers would like to follow the progress of the tests
  • This could be implemented by notifying the packagers when a test run starts and provide an URL where they could see and follow them
• Packagers would like to be able to re-trigger a test run from an UI and a CLI
• Packagers would like to be able to waive false-negative from an UI and a CLI
• Packagers would like the ability to transform a bodhi update automatically created for a single build into a side-tag-specifi bodhi update
  • This would allow the workflow where a packager makes a build, which fails CI because another package needs to be rebuilt as well. The packager can then convert the bodhi update into a side-tag based update, this would create the corresponding side-tag and move the existing build into it, allowing the packager to do the missing rebuilds before telling bodhi to process the update. This is entirely doable by hand, which is why this feature while convenient is considered a nice to have
• Packagers would like fedpkg chain-build to create the corresponding side-tag automatically

Single build update

   fedpkg clone rpms/foobar
   cd foobar/
   vim foobar.spec
   git commit -am "Git commit message"
   fedpkg build
   

If the CI tests pass, the package will land in rawhide, if they fail the user will be able to go to bodhi to see what failed and why as well as waiving the failed tests if needed.

Multi builds with gating

   fedpkg clone rpms/foobar
   cd foobar/
   fedpkg side-tag create foo1.2
   vim foobar.spec
   git commit -am "Update of foobar to update foo to 1.2"
   fedpkg push
   fedpkg build --target=user_foo1.2
   cd ..
   fedpkg clone rpms/foobaz
   cd foobaz
   vim foobaz.spec
   git commit -am "Update of foobaz to update foo to 1.2"
   fedpkg push
   fedpkg build --target=user_foo1.2
   cd ..
   fedpkg clone rpms/foo
   cd foo
   vim foo.spec
   git commit -am "Update foo to 1.2"
   fedpkg push
   fedpkg build --target=user_foo1.2
   fedpkg side-tag merge foo1.2

Dependencies

• Bodhi
• koji
• fedpkg

Contingency Plan

We can keep doing rawhide as we are doing it today.

Documentation

To be written/updated

List of documentation pages to update:

• https://fedoraproject.org/wiki/Updates_Policy
• https://fedoraproject.org/wiki/Package_maintenance_guide
• https://fedoraproject.org/wiki/Package_update_HOWTO
• https://docs.fedoraproject.org/en-US/packaging-guidelines/