Infrastructure 2020/Rawhide Gating

Document Details

• Status:WORKING DRAFT:
• Point of contact: Pingou
• Implementation Schedule: TBD
• Release Date: TBD

Goals

We want to gate packages on test results before they can land in rawhide. This will reduce the amount of broken dependency, uninstallable packages and broken composes leading to a more stable rawhide as well as less work on the infrastructure and rel-eng teams to keep composes working. The benefits to Fedora for implementing this:

• As more packagers opt-into gating and add tests to their packages, rawhide becomes more stable
• More stable rawhide will lead to easier composes and a smoother release process
• Ability to use chain-build in rawhide and stable releases

Background and Strategic Fit

Currently composing rawhide is often broken leading to a poor user-experience as well as impacting the release engineering team when trying to run a compose on rawhide.

By gating how rawhide packages land into rawhide and its buildroot we will be able to offer a more stable rawhide and reduce the workload when we branch and are getting ready to release.

Packages having tests are already being tested for every commit made to their repository. These results are then made available in bodhi on the update's page. In addition, if a pull-request is opened on https://src.fedoraproject.org against one of these packages, the tests are ran and their outcome indicated as a flag on the pull-request page.

The principle of this proposal is to set the infrastructure allowing to gate packages, how the packages are gated (ie: on which tests) is out of scope and will be a policy (FESCo) decision once the tooling and workflow is put in place.

People involved

This project impacts a number of application or other projects. Here is a quick list of the different projects involved and their point of contact.

• Coordinator: Pierre-Yves Chibon
• Bodhi: Randy Barlow and Aurélien Bompard
• koji: Mikolaj Izdebski and Mike McLean
• fedpkg/rpkg: Chenxiong and Lubomír Sedlář
• Packager notifications: ?
• Fedora CI: Dominik Perpeet

Assumptions and Questions

Are there any assumptions for completing this work?

We are using Bodhi and Koji which already exist, both will need some work which is part of the scope of change Which CI system are we using? <-- out of scope Which tests will be used? <-- out of scope

Agreement that the MVP is the most critical thing here so out of scope items will not be considered initially.

Additional Links

Change proposal: https://fedoraproject.org/wiki/Changes/GatingRawhidePackages

Initial Stories

A minimum user story list should be prepared here to facilitate Task Breakdown. The user story describes the type of user, what they want and why. A user story helps to create a simplified description of a requirement. Format should be:

As a < type of user >, I want < some goal > so that < some reason >.

User Stories elicited on the 7th of February 2019 call

Personas:

• Rawhide User: A user of the Rawhide system
• Fedora Community: Gen pop
• Release Engineering User: Members of the Release Engineering group
• Packager: Someone who packages Fedora packages

Story List

• As a Rawhide User, I want a more stable Rawhide environment, so that my system is more stable
  • Everybody expects that Rawhide has bugs :)
  • If we can get rid of the bugs and issues that CI picks up it makes it easier
  • For an existing Rawhide User this work is not a major benefit Vs a new User experience

• As a new Rawhide User, I want a better experience, so that I am encouraged to participate

• As a Fedora Community, I want to stablise rawhide, so that we can increase the usability of Rawhide
  • smaller bugs are caught
  • larger test base
  • more use cases

• As a Fedora Community, I want to see bugs squashed sooner, so that I can get packages out that are cleaner
  • Acceptance Criteria here is that the item is in the repo -- therefore it's sanitised
  • Potential to have packages available earlier than the 6 month cycle for stable

• As a Fedora Community User, I want to know the set of common tests, so that I can be aware of how to get into Rawhide
  • Note that is out of scope for Phase 1 as this is a policy decision

• As a Release Engineering User, I want a more stable Rawhide, to have confidence in cutting a release

• As a Fedora IT User, I want a reason why my package failed to get through the Gate, so that users can be informed why

• As a Packager, I want to have the same workflow with reasonable speed, so that I don't have to deal with a new process

Task breakdown

This is a shorthand list of items we are needing to cover for building Rawhide Gating in Fedora Infrastructure for the Fiscal Year 2020.

Bodhi

• Single package update
  • We will need to write a message-bus listener that will automatically create a bodhi update for each package built in the rawhide-gated tag.
  • We will need to write a message-bus listener to automatically push bodhi updates created for rawhide that have passed CI (the decision being announced by Greenwave).

• Multi package update
  • We will need to allow bodhi to create a new update from a specified side-tag instead of a list of builds.
  • We will need to write a message-bus listener that, upon greenwave notifications about an update created from a side-tag passing CI, will move all of its builds into the rawhide-candidate tag and clean the side-tag created for this update.
  • We will need to write a script that regularly clean old side-tags left around

General Acceptance Criteria:

• I am able to build the package automatically
• Want to be able to run the tests / gating criteria (TBD)
• Provide test results in the UI
  • Provide indication of success / fail in particular
• Automatically deploy into Rawhide if the test passed or stay in Bodhi if it failed

Open Question: Should a response {CLI, mail, fed-message} be part of the MVP? If I want to do my builds in an automated manner and not manually check the Bodhi UI. This may be a key packager User Story and needs to be clarified.

Koji

• We will need a koji plugin allowing packagers to create new side-tags

General Acceptance Criteria:

• Able to build the package in Rawhide buildroot without impacting other packaging (i.e. in isolation)

Open Question:

Site tag creation will be discussed next week which may elicit new requirements & acceptance criteria.

fedpkg/rpkg

• We will need to add support for the command: fedpkg side-tag create which will create a new side-tag in koji for this user using the new plugin
• We will need to add support for the command: fedpkg side-tag merge which will call bodhi to ask it to create an update corresponding to this side-tag
  • Alternatively: Expand the existing fedpkg update command to add support for creating a new update from a side-tag

General Acceptance Criteria:

• Able to create a side tag
• Able to merge a side tag

Future Plans

Phase 0

Deploy changes to koji and bodhi so basic gating can be tested.

Phase 1

Deploy changes to fedpkg/rpkg and deploy the changes to send notifications to packagers about gating status changes (more precisely informing the packager when their builds are gated/allowed through).

At this point, gating can be offered to all packagers in an opt-in basis.

Phase 2

Convince more packagers to opt-in to gating packages on tests.

Phase 3

Let FESCo decide which tests should be enforced for the entire distribution (if any).