No Frozen Rawhide Proposal

Below is the original proposal submitted to and approved by FESCo. The No Frozen Rawhide Implementation page tracks the actual implementation.

Overview

Always keep rawhide moving forward, never stopping the train. At branch point, publish the release to be in a different path to allow rawhide to race forward toward the next release.

Problem Space

Currently we spend a lot of time blocking the progress of rawhide. We do this to try to get people to test the packages we're targeting for a release, instead of the packages in the ever rolling development tree that is rawhide. We get a lot of anger and naval gazing particularly near the end of a release cycle where things are building up in the next rawhide tag and in the updates tag for the release we're about to make. We then have a flood of pent up changes that hit nearly all at once, breaking god knows what.

We also tell folks that a development cycle of Fedora starts from the branch point of the release to be and goes all the way to the release of the next release, eg F12 development cycle started at F11 Beta (where we allowed pre-branching) and will last until F12 Feature Freeze. However by not publishing the builds done for F12 during the F11 Beta/PR/RC phases, little meaningful testing and development could be done for Fedora 12.

Shortly before a release, we try to move "rawhide" to be the next release as we've already finished the "pending" release. This will make it so that people wanting to stay on the pending release lose the ability to access the "everything" repo for their release until the mirrors unlock.

Proposed Solution

To keep rawhide (nearly) always moving, we'll have to change how we do freezes, particularly the long final development freeze. Without getting too much into detail, when we reach feature freeze, and offer pre-branch capability, we turn that into a full branch event. devel/ continues to publish to pub/fedora/linux/development and the train moves on. Somewhere else on the mirror we start dumping the builds from the release branch, say F-12 branch.

For a period of time the builds can go directly live each night until we freeze again. At the freeze point, bodhi would be used to manage freeze break requests. Builds would go into a -candidate tag, bodhi requests would push them out to a -testing dir for public consumption. Karma alone or karma + releng/qa approval would promote things from testing into the directory yet unnamed that is the "rawhide" of the pending release, until we reach gold/RC/whatever and things that make it out of -testing go to -updates as 0-day items.

Repo configs would be setup so that those installing after the Feature Freeze would stay on the pending release and never lose access to their "everything" repo.

Scope

• compose tools
• Wiki Pages
• Bodhi
• Mirror Layout
• package management repo configs
• bugzilla

Compose Tools

Compose tools will need to get a lot faster in order to handle composing rawhide, the pending release to be, potential updates to the pending release to be, and updates to our existing releases, all within the same day.

Bodhi

Bodhi will have to open early to allow maintainers to request freeze breaks for the pending release.

Bodhi will also have to be modified so that it can do tagging only "pushes" and allow the nightly compose scripts to compose from the koji tags that bodhi populates, however it would be useful to have bodhi generate the updateinfo.xml as well as manage bug states and send announcements. Perhaps bodhi can grow a two stage push method where first tagging action happens, then later via cron a mash request goes though which does all the mashing and generation of extra metadata.

Bodhi will have to distinguish between a freeze break to go into the release vs an update request which may go out as a 0-day update.

Mirror Layout

A new set of directories will need to be made to hold the pending release content, as well as candidate freeze breaks for the pending release.

A proposed layout would be that once branched, the pending release bits go to /pub/fedora/linux/releases/#/Everything/ and would be updated each night from the pending release tag (like dist-f12), and would contain install images. Potential updates to the pending release would be published nightly as well to /pub/fedora/linux/updates/testing/#/. At this point /pub/fedora/linux/updates/# would be an empty set of repos.

At release time, /pub/fedora/linux/releases/#/Fedora would be created and hidden during mirror staging. Revealing the Fedora/ path would equate to making the release. The Everything/ path would remain accessible throughout. At release time we would also scrub the install images from the Everything tree, leaving only the images in the Fedora/ tree.

package management repo configs

Repo configs will need to be changed to reflect the new mirror paths.

At Alpha, fedora-release would have repo configs that enable the fedora repo (Everything tree) as well as fedora-updates-testing. At or around Beta the updates-testing repo would be turned off in favour of the updates repo.

Wiki Pages

Many wiki pages will have to be changed. This should be an ongoing list of wiki pages that reference the development cycle.

Bugzilla

Bugzilla will need to grow a version earlier in the development cycle so that bugs can be filed for the pending release vs rawhide.

Release

How we do the release will change slightly. After we've branched, but before we've reached the Release Candidate phase of the final release, when bodhi pushes an update to "stable" the update will be tagged for the pending release, such as dist-f12. Then we will re-create the Everything/ tree nightly from the dist-f12 tag. Then once we've reached RC phase, the stable updates will instead go to the normal updates/ directory and be 0-day updates for the release.

Discussion Points

• What do we call the pending release tree?
• When do we branch? Feature Freeze?
• How are buildroot overrides handled, freeze break needs vs 0-day update needs
• Increased number of "double" commits (things committed to rawhide, and duplicated on the release branch)
• Relies on updates-testing to drive feedback for freeze breaks, and updates-testing is not as well used as it could be.
• Division of our testers between rawhide and the pending release
• Are we now Debian (Unstable, testing, stable)
• Potential to break chain-builds even more depending on when we force things through bodhi
• Do we always make install images for rawhide, or only make images for pending release tree?
• packages going "straight to stable" via bodhi during a freeze
• When and how does signing happen? How will we know if something isn't signed in time to fix it?

Comments?

To leave a comment, use the Talk page for this proposal.

Owner

Jesse Keating