Infrastructure 2020/Rawhide Gating

Document Details

• Status:WORKING DRAFT:
• Point of contact: Pingou
• Implementation Schedule: TBD
• Release Date: TBD

Goals

We want to gate packages on test results before they can land in rawhide. This will reduce the amount of broken dependency, uninstallable packages and broken composes leading to a more stable rawhide as well as less work on the infrastructure and rel-eng teams to keep composes working. The benefits to Fedora for implementing this:

• As more packagers opt-into gating and add tests to their packages, rawhide becomes more stable
• More stable rawhide will lead to easier composes and a smoother release process
• Ability to use chain-build in rawhide and stable releases

Background and Strategic Fit

Currently composing rawhide is often broken leading to a poor user-experience as well as impacting the release engineering team when trying to run a compose on rawhide.

By gating how rawhide packages land into rawhide and its buildroot we will be able to offer a more stable rawhide and reduce the workload when we branch and are getting ready to release.

Packages having tests are already being tested for every commit made to their repository. These results are then made available in bodhi on the update's page. In addition, if a pull-request is opened on https://src.fedoraproject.org against one of these packages, the tests are ran and their outcome indicated as a flag on the pull-request page.

The principle of this proposal is to set the infrastructure allowing to gate packages, how the packages are gated (ie: on which tests) is out of scope and will be a policy (FESCo) decision once the tooling and workflow is put in place.

People involved

This project impacts a number of application or other projects. Here is a quick list of the different projects involved and their point of contact.

• Coordinator: Pierre-Yves Chibon
• Bodhi: Randy Barlow and Aurélien Bompard
• koji: Mikolaj Izdebski and Mike McLean
• fedpkg/rpkg: Chenxiong and Lubomír Sedlář
• Fedora CI: Dominik Perpeet
• Fedora Messaging: Aurélien Bompard

Assumptions and Questions

Are there any assumptions for completing this work?

• The CI system is able to test/act on a Bodhi update
• We are using Bodhi and Koji which already exist, both will need some work which is part of the scope of change
• Which CI system are we using? <-- out of scope
• Which tests will be used? <-- out of scope
• We need to migrate the apps to use the fedora_messaging library instead of fedmsg, otherwise messages sent between applications will not be reliable, and may get lost (messages already get lost today). Reliability being important for this project, we can't avoid migrating but other tasks can move forward in parallel. It'll of course save work to migrate before adding new message emitters or listeners, but it's not a very significant amount of work.

Agreement that the MVP is the most critical thing here so out of scope items will not be considered initially.

Additional Links

Change proposal: https://fedoraproject.org/wiki/Changes/GatingRawhidePackages

Initial Stories

A minimum user story list should be prepared here to facilitate Task Breakdown. The user story describes the type of user, what they want and why. A user story helps to create a simplified description of a requirement. Format should be:

As a < type of user >, I want < some goal > so that < some reason >.

User Stories elicited on the 7th of February 2019 call

Personas:

• Rawhide User: A user of the Rawhide system
• Fedora Community: Gen pop
• Release Engineering User: Members of the Release Engineering group
• Packager: Someone who packages Fedora packages

Story List

• As a Rawhide User, I want a more stable Rawhide environment, so that my system is more stable
  • Everybody expects that Rawhide has bugs :)
  • If we can get rid of the bugs and issues that CI picks up it makes it easier
  • For an existing Rawhide User this work is not a major benefit Vs a new User experience

• As a new Rawhide User, I want a better experience, so that I am encouraged to participate

• As a Fedora Community, I want to stablise rawhide, so that we can increase the usability of Rawhide
  • smaller bugs are caught
  • larger test base
  • more use cases

• As a Fedora Community, I want to see bugs squashed sooner, so that I can get packages out that are cleaner
  • Acceptance Criteria here is that the item is in the repo -- therefore it's sanitised
  • Potential to have packages available earlier than the 6 month cycle for stable

• As a Fedora Community User, I want to know the set of common tests, so that I can be aware of how to get into Rawhide
  • Note that is out of scope for Phase 1 as this is a policy decision

• As a Release Engineering User, I want a more stable Rawhide, to have confidence in cutting a release

• As a Fedora IT User, I want a reason why my package failed to get through the Gate, so that users can be informed why

• As a Packager, I want to have the same workflow with reasonable speed, so that I don't have to deal with a new process

Task breakdown

This is a shorthand list of items we are needing to cover for building Rawhide Gating in Fedora Infrastructure for the Fiscal Year 2020.

Bodhi

The Bodhi tasks can be tracked on Bodhi's CI Gating kanban board.

• Single package update
  • We will need to write a message-bus listener that will automatically create a bodhi update for each package built in the rawhide-gated tag.
  • We will need to write a message-bus listener to automatically push bodhi updates created for rawhide that have passed CI (the decision being announced by Greenwave).

• Multi package update
  • We will need to allow bodhi to create a new update from a specified side-tag instead of a list of builds.
  • We will need to write a message-bus listener that, upon greenwave notifications about an update created from a side-tag passing CI, will move all of its builds into the rawhide-candidate tag and clean the side-tag created for this update.

• Bodhi will comment on the update when the gating status is known or changes so the packagers know about it

General Acceptance Criteria:

• I am able to build the package and have a bodhi update automatically created for it (for single package update)
• The CI system runs the package's tests and the gating criteria are applied
• Provide test results in the UI (bowlofeggs: We already have this, right? -- pingou: yes)
  • Provide indication of success / fail in particular
• Automatically deploy into Rawhide if the test passed or stay in Bodhi if it failed
• Packagers should be able to waive test results.

Open Question: Should a response {CLI, mail, fed-message} be part of the MVP? If I want to have updates created in an automated manner and not manually check the Bodhi UI. This may be a key packager User Story and needs to be clarified. (bowlofeggs: I think this should be part of the MVP. This is part of the long list of reasons the previous effort was a failure - packagers did not know when things went wrong and were surprised when their updates got blocked. Fortunately, it's easy - we will just have Bodhi comment on updates when the tests fail. See ticket #2210.)

Koji

• We will need a koji plugin allowing packagers to create new side-tags
• We need to optimize side-tag creations in koji so it doesn't take hours to be usable to the packagers requesting them

General Acceptance Criteria:

• All packagers are able to create side-tags on-demand
• Able to build the package in Rawhide buildroot without impacting other packaging (i.e. in isolation)

fedpkg/rpkg

• We will need to add support for the command: fedpkg side-tag create which will create a new side-tag in koji for this user using the new plugin
• We will need to add support for the command: fedpkg side-tag merge which will call bodhi to ask it to create an update corresponding to this side-tag
  • Alternatively: Expand the existing fedpkg update command to add support for creating a new update from a side-tag

General Acceptance Criteria:

• Able to create a side tag
• Able to merge a side tag

Phased release

Phase1

Adjust bodhi to support single package update gating in rawhide.

[Changes/GatingRawhideSinglePackageUpdates|FESCo proposal] for phase 1

Phase 2

Support multi-packages updates gating in rawhide.

[Changes/GatingRawhideMultiPackagesUpdates|FESCo draft proposal] for phase 2

Phase 2.0

Deploy changes to Bodhi and koji in staging so the workflow can be tested.

Phase 2.1

Deploy changes to Bodhi and koji in production, release a new version of fedpkg/rpkg allowing packager to create sidetags and ask for them to be merged.

At this point, gating can be offered to all packagers in an opt-in basis.

Phase 3

Convince more packagers to opt-in to gating packages on tests.

Phase 4

Let FESCo decide which tests should be enforced for the entire distribution (if any).