Changes/Python setuptools 0.7

python-setuptools update to 0.7.x

Summary

Update to a new upstream release of python-setuptools that is not completely compatible with previous releases.

Owner

• Name: Toshio Kuratomi with help from Bohuslav Kabrda, Nick Coghlin, and the Python SIG
• Email: toshio@fedoraproject.org
• Release notes owner:

Current status

• Targeted release: Fedora 20
• Last updated: June 10 2013
• Tracker bug: <will be assigned by the Wrangler>

Detailed Description

python-setuptools is used as a buildtime dependency in many packages. It is also a runtime dependency for some packages where it provides a way to load plugins, load data files, and specify what versions of dependent packages are needed. In Fedora, we also make use of its multiversioning features to install some backwards compatibility packages.

The upstream for the python setuptools package has been in a maintenance mode for many years. During that time we have been shipping a fork, python-distribute, which has added bugfixes and python3 compatibility but not significantly altered the API. Just recently, python-setuptools upstream has become active and released a new version that merges the two forks and is not completely API compatible. As the older versions won't see bug fixes anymore, we need to move to this new version but there could be some usages that are broken by the new version.

Benefit to Fedora

The upstream activity is due to two competing projects merging: python-setuptools and python-distribute. The merger is a good thing for the python community as it unites the bugfixes and minor differences that have arisen in these two forks over the years. This makes it easier for upstreams to maintain their software. For us, it updates us to the latest version of the code which means that we'll be able to continue submitting patches as we find bugs in the code rather than having to maintain our own divergencies.

Scope

Note: From reading the upstream changelogs, I don't anticipate that we'll have much trouble with this (with the possible exception of the python package installers noted below). However, this is a change to the code used to build a large number of packages so it's best to both write this as a Change and get started on it early.

Proposal owners

• Update the python-setuptools package from the current code-base (python-distribute-0.6.x) to python-setuptools-0.7.x
• Rebuild packages with the new setuptools (Only necessary for testing)
• Areas that may need special attention. Note that these are areas that we will test to understand the scope of changes required. It is not a promise to thoroughly test what is broken due to the update or to fix the breakage (although we will help)
  • Test python package installers with the new setuptools (pip, buildout)
  • Test python packages that build with setuptools for related ftbfs (we could do a mass rebuild of everything requiring python at runtime or refine our list to the packages that BuildRequire: python-setuptools)
  • Test packages where we are using setuptools to build multiversion (example: python-cherrypy2)
  • Test python packages that have a runtime requirement on setuptools [Note: The owners of this Change are going to leave most of this to individual package maintainers due to this being highly dependent on what the packages actually do with setuptools at runtime]
• See if our python packaging guidelines need update (particularly the multi-version/python-setuptools/egg guidelines)
• Check that the python-setuptools managed metadata has not changed incompatibly

Other developers

Owners of packages which BuildRequire or Require python-setuptools will need to:

• Test for buildtime and runtime breakage from new setuptools (as applicable to each package)
• Fix FTBFS breakage from new setuptools
• Fix runtime breakage from new setuptools

Release engineering

• Rebuild packages which BuildRequire: python-setuptools with the new setuptools
  • Probably not strictly necessary (will confirm when we test whether the package metadata format has changed) but it is good for testing since setuptools is a buildtime dep in many python packages.

Policies and guidelines

• Unknown at this time. This will become apparent when the Change Owners test building with the new setuptools and if multi-version compat packages need any additional changes in order to install and run.

Upgrade/compatibility impact

• No data will need migrating but user installed python modules that rely on the old version of python-setuptools may no longer work.
• Probably no problems for packages installed with the older python-setuptools but change owners need to check that there are no incompatible changes to the package metadata managed by python-setuptools.

How To Test

First install the updated python-setuptools package.

# yum update python-setuptools
# rpm -q python-setuptools |grep 0.7
[should display a python-setuptools 0.7 package]

Check that packages that need setuptools to build still build

Testing with an upstream tarball/checkout:

$ grep setuptools setup.py
$ python setup.py install --root `pwd`/tmp/
[This should work with no errors]

Testing with an rpm package:

$ fedpkg srpm python-$MODULE
$ fedpkg scratch-build --srpm python-$MODULE*src.rpm

Test multiversion python packages

Building should be the same test as above.

Runtime testing:

[ install both versions of the package]
$ yum install python-cherrypy python-cherrypy2
[test that both versions of the package work]
[Install a package that uses the multiversion package as a dependency]
$ yum install bodhi
[test that the package still works (loads the correct version of the multi-version package)]

Check that packages that need setuptools to run still run

$ yum install $PACKAGE
$ rpm -q --requires $PACKAGE|grep python-setuptools
[Displays that $PACKAGE needs python-setuptools]
[Run the program and make use of plugins or other features that need python-setuptools]

Test python package installers with the new setuptools

$ yum install python-pip
$ pip install --user $OTHER_PACKAGE
[Other python-pip commands]

Other packages in this category include python-zc-buildout (python-virtualenv if it finally unbundles pip and setuptools).

User Experience

• Since this update includes API changes, developers who use setuptools as a library may notice changes.

See the following pages for details:

• http://pythonhosted.org/setuptools/merge.html
• https://pypi.python.org/pypi/setuptools#changes

Note that our current pyhton-setuptools package is the distribute fork so the merge changes are relative to that.

Dependencies

To see the dependent packages run these repoquery commands:

[Runtime]
$ repoquery -q --whatrequires python-setuptools
[103 packages]
[Buildtime]
$ repoquery --enablerepo rawhide-source --archlist=src --whatrequires python-setuptools > tmp.lst
$ repoquery --enablerepo rawhide-source --archlist=src --whatrequires python-setuptools-devel >> tmp.lst
$ sort tmp.lst | uniq > setuptools-br.lst
[918 packages]

Contingency Plan

Contingency mechanism: (What to do? Who will do it?)

Three alternate contingency mechanisms exist:

1. Someone who has a package that cannot be ported to the new python-setuptools will need to submit a backwards compat package for the older version of python-setuptools. The Change Owners can commit to reviewing the package but not to being the package owner for the new package.
2. The packages that have not been ported can be dropped from the distribution.
3. The new setuptools package can be reverted for F20.

Deciding which of these to perform depends on the results of testing (which will show how many packages suffer from issues, whether those packages are easy to port, whether the unported packages are essential to the distribution or partially unmaintained, and whether the changes to port other packages to the new setuptools would be hard to revert.)

Contingency deadline

For option 1, adding a backwards compatible package. We would want to add that early enough to be able to make sure that the package is usable by the unported packages. Soon after alpha would be ideal.

For option 2, beta freeze -- but we'd need to evaluate that the packages to be dropped are non-essential to the distribution much earlier. Probable at alpha freeze.

For option 3, it depends on how difficult it would be to back out changes for all packages. At this time, I don't think there will be a widespread need for patching or spec file changes, so this could be done at beta freeze.

Blocks release?

Probably not. Neither anaconda nor yum use this directly and it's not a critpath package. However, we might want to reevaluate this if we find that it blocks building of something essential to the distribution (as it could prevent quick rebuilds of the packages needed for the distribution).

Documentation

Upstream documentation:

• http://pythonhosted.org/setuptools/merge.html
• https://pypi.python.org/pypi/setuptools#changes

Notes from our testing will be added here or a separate wiki page if we encounter incompatibilities in updating.

Release Notes

• If the update does not cause any porting difficulties I would not release note this -- it will likely be just as invisible to end users as it is to us.

• If the update shows that the incompatibilities affect a lot of code then we should add something like this:

The version of the python-setuptools package has been updated to the 0.7.x series. This release series merges the setuptools and distribute upstream projects which has introduced a variety of changes to the API and behaviour. Please see https://fedoraproject.org/wikiChanges/Python_setuptools_0.7#Documentation for more details if this affects code you are writing or deploying.