Features/Python 2.7/MassRebuild

Goal

The goal is to rebuild every Fedora Python package before the Fedora 14 Feature Freeze (July 27th).

Driving Features

• Python 2.7

From an RPM metadata perspective, the relevant packages contain:

• various files below /usr/lib{64}/python-2.6, which need to move to /usr/lib{64}/python-2.7
• a Requires: python(abi) = 2.6, which must become a Requires: python(abi) = 2.7

From an implementation perspective, the bytecode format for .pyc and .pyo files has changed, and the C ABI for extension modules has likely changed as well, necessitating a recompile of all of these

Schedule

All automated rebuilds should be finished prior to the Feature Freeze, which is July 27th. Any clean-up manual rebuilds should be finished prior to the Alpha Freeze, which is August 3rd.

Note that deltarpms will be disabled for the duration of the mass rebuild.

Note: dmalcolm intends to babysit and fix packages that don't rebuild. He is a proven packager.

He plans to start the scripted rebuild on Wednesday 21st July at noon EDT.

For the sake of simplicity, he does not plan to provide an "opt out" system for maintainers to do their own rebuilds.

TODO: need to email the devel list to let people know what's going on.

Scripts

Release Engineering has created two scripts for doing full mass-rebuilds. One is to initiate the builds, and the other is to query for items still needing to be built.

dmalcolm will use a modified version of the scripts which accepts a text file listing the names of the source packages to be rebuilt (on separate lines). It will also change a hardcoded reference to jkeating in the options passed to CVS (adding a command-line setting, perhaps)

TODO: generate this text file.

Target and Tags

rel-eng will create the Koji tags and target.

The plan is to segregate the rebuilt packages into their own tag and buildroot, so that we gradually build all packages using buildroots drawn from the "dist-f14-py27-rebuild" tag, building the results into the "dist-f14-py27-rebuild" tag.

koji add-tag --arches i686,x86_64 --parent dist-f14-build dist-f14-py27-rebuild
koji add-target dist-f14-py27-rebuild dist-f14-py27-rebuild

Build Initiation

TODO: currently a bit handwavy about ordering in the BuildRequires; need to better set up build ordering

TODO: do we need a compat-python-2.6 temporarily to resolve loops in the dep graph?

dmalcolm will first manually build these packages in this order:

• python
• python-setuptools
• python-nose
• numpy

in order to prepare 2.7 versions of the most important packages (from a BuildRequires perspective) into the tag serving the buildroot.

Assuming these build to completion, dmalcolm will then run the rebuild script.

The rebuild script works in the following way:

 Read the text file
 Loop through each package:
   Query if a build has already been attempted/completed since koji changes went live.
   If so, move to next package
   If not, check out CVS
   Check for a noautobuild file
   If so, move to next package
   If not, rpmdev-bumpspec
   cvs commit;tag
   make (background) build
   Move on to next package

Each step will have some error catching and logging so that people can clean up the various failures for whatever reasons. The builds will be background builds, which will allow other builds done to take higher priority when a builder has a free slot. However maintainers should take care when doing this so that the background build won't take 'latest' precedent when it finishes. Asking rel-eng to kill the scripted build will typically be enough.

TODO: verify this script

TODO: ensure that dmalcolm can get a shell on the CVS server (to speed up this script): infrastructure ticket opened

Build Tagging

Once the rebuild script has finished, another script will run to tag the builds into dist-f14. This script will check that a newer build hasn't been done or started in dist-f14 since the scripted rebuild was submitted. This will protect against the scripted rebuild overriding an even newer build done while the scripted rebuild was waiting in the background.

TODO: verify this script

Status Query

Release Engineering has developed a script to query dist-f14 and report on packages that have yet to be rebuilt. Results of these queries will be delivered to various places, yet to be determined, broken down by maintainer for easier discovery of work needed.

TODO: verify this script

Maintainer Actions

Hopefully none: dmalcolm will do the rebuild and will attempt to fix anything that breaks. If larger issues arise, he may file bugs

• Maintainers can help this effort by ensuring rpmdev-bumpspec correctly bumps your package's spec files.
• Maintainers should ensure that their packages currently build from source.
• Maintainers should ensure that there are no unwanted changes committed to CVS but not built yet.

Frequently Asked Questions

FIXME

Feedback

Questions/comments/concerns should be directed to fedora-devel-list, the discussion page, or #fedora-devel on freenode IRC.