Fedora 11 Mass Rebuild

Goal

The goal is to rebuild every single Fedora package, regardless of content, before the Fedora 11 Feature Freeze (March 3rd).

Driving Features

• Architecture Support

This requires a change in redhat-rpm-config for the i586 flags, as well as a koji config change to build for i586 rather than i386 by default. These two changes can happen at any time, and will likely happen this week to shake out any bugs. Dennis Gilmore and Jon Masters are responsible for getting these changes in place.

• GCC 4.4

This requires gcc 4.4.0-0.19 in the buildroots before the mass rebuild begins. Jakub is responsible for getting this built this week.

• Stronger Hashes

This requires redhat-rpm-config changes for default hash, as well as koji code changes to handle the hash. Mike Bonnet and Miloslav Trmač are responsible for getting the koji changes done upstream. Miloslav Trmač and Jon Masters are responsible for getting the redhat-rpm-config changes in place, which depend on the koji changes being live first. A koji outage is scheduled for this weekend to deploy a new version of koji, which will include changes for this feature. An updated redhat-rpm-config will be deployed shortly thereafter to ensure hash changes are in effect.

Schedule

Given the changes required for the above features and the given schedules, Release Engineering will be ready to start scripted rebuilds on Monday Feb. 23rd. All rebuilds should be finished prior to the Feature Freeze, which is March 3rd.

Timeline

• Over the weekend of Feb 20-22 there was a koji outage. This outage was to upgrade the koji packages and rpm packages in the buildsystem for the new features.
• At 2009-02-23 18:31:07 UTC the repodata used in dist-f11 buildroots contained an updated redhat-rpm-config which instructed rpm to use the larger checksums. Any build done after this point is considered acceptable for the mass rebuild.
• Around 2009-02-23 20:00:00 UTC the automated script was started.

Opting Out

Release Engineering has given maintainers the ability to opt out of the scripted rebuild in order to do the build on their own. Note that every single package needs to be rebuilt, regardless of the contents.

In order to opt out of the scripted rebuild, a maintainer will need to check a file into their package's devel/ branch, named noautobuild. This file should contain a short rationale of why you wish to do the build yourself.

If the build initiation script encounters such a file, it will skip that package. However, given the tight deadlines (Feature freeze being the 3rd of March), a very short grace period will be given for completing the builds on your own. If your build has not been attempted by the 26th of Feb, the script will no longer honor the noautobuild request and will attempt to rebuild your package. If, however, a rebuild has been attempted, the script will bypass your module. This should prevent multiple attempts to build something that fails for one reason or another.

Scripts

Release Engineering has created two scripts. One is to initiate the builds, and the other is to query for items still needing to be built.

Build Initiation

The rebuild script works in the following way:

 Generate a list of all packages in dist-f11
 Loop through each package:
   Query if a build has already been attempted/completed since koji changes went live.
   If so, move to next package
   If not, check out CVS
   Check for a noautobuild file
   If so, move to next package
   If not, rpmdev-bumpspec
   cvs commit;tag
   make (background) build
   Move on to next package

Each step will have some error catching and logging so that people can clean up the various failures for whatever reasons. The builds will be background builds, which will allow other builds done to take higher priority when a builder has a free slot. However maintainers should take care when doing this so that the background build won't take 'latest' precedent when it finishes. Asking rel-eng to kill the scripted build will typically be enough.

Build Tagging

Once the rebuild script has finished, another script will run to tag the builds into dist-f11. This script will check that a newer build hasn't been done or started in dist-f11 since the scripted rebuild was submitted. This will protect against the scripted rebuild overriding an even newer build done while the scripted rebuild was waiting in the background.

Status Query

Release Engineering has developed a script to query dist-f11 and report on packages that have yet to be rebuilt. Results of these queries will be delivered to various places, yet to be determined, broken down by maintainer for easier discovery of work needed.

Maintainer Actions

• Maintainers that wish to do their own builds should read the Opting Out section.
• Maintainers can help this effort by ensuring rpmdev-bumpspec correctly bumps your package's spec files.
• Maintainers should ensure that their packages currently build from source.
• Maintainers should ensure that there are no unwanted changes committed to CVS but not built yet.

Frequently Asked Questions

How do I opt out of the scripted rebuild?

See the Opting Out section.

When will my own build "count" for the rebuild?

After the koji outage necessary for Stronger Hashes, and the updated redhat-rpm-config is put into place, a build done by a maintainer will "count" toward the rebuild. The exact timing of this is not determined, but should likely happen after the outage and before 1800 UTC Monday the 23rd.

If I've already built my package for gcc-4.4, am I done?

gcc-4.4 is only one of the reasons why we need a rebuild. See the previous question.

Will the rebuilds be ordered?

Currently there is no plan to logically order the rebuilds based on deps. The ordering will be alphanumerical based on package name. There are no planned ABI/soname changes that would require logical build ordering. Further, all the builds will be kept in a special tag (dist-f11-rebuild) until the whole run is done, and then they will be tagged into dist-f11 in one shot to minimize the shuffle of buildroot contents during the rebuilds.

Feedback

Questions/comments/concerns should be directed to fedora-devel-list, the discussion page, or #fedora-devel on freenode IRC.