Changes/Python setuptools 0.7

python-setuptools update to 0.7.x and greater

Summary

Update to a new upstream release of python-setuptools that is not completely compatible with previous releases. The python-setuptools "major" version may be higher than 0.7.x as there's some other structural changes happening that should not affect API.

• 0.8.x is supposed to be needed for pip-1.4, virtualenv-1.10, and is only supposed to switch to a single code base for python2 and python3 (instead of using 2to3 to translate the python2 sources to python3).
• 0.9.x added support for other hashes than md5 on packages hosted on the python package index (pypi).

Owner

• Name: Toshio Kuratomi with help from Bohuslav Kabrda, Nick Coghlan, and the Python SIG
• Email: toshio@fedoraproject.org
• Release notes owner:

Current status

• Targeted release: Fedora 20
• Last updated: July 17 2013
• Tracker bug: #998574

Detailed Description

python-setuptools is used as a buildtime dependency in many packages. It is also a runtime dependency for some packages where it provides a way to load plugins, load data files, and specify what versions of dependent packages are needed. In Fedora, we also make use of its multiversioning features to install some backwards compatibility packages.

The upstream for the python setuptools package has been in a maintenance mode for many years. During that time we have been shipping a fork, python-distribute, which has added bugfixes and python3 compatibility but not significantly altered the API. Just recently, python-setuptools upstream has become active and released a new version that merges the two forks and is not completely API compatible. As the older versions won't see bug fixes anymore, we need to move to this new version but there could be some usages that are broken by the new version.

Benefit to Fedora

The upstream activity is due to two competing projects merging: python-setuptools and python-distribute. The merger is a good thing for the python community as it unites the bugfixes and minor differences that have arisen in these two forks over the years. This makes it easier for upstreams to maintain their software. For us, it updates us to the latest version of the code which means that we'll be able to continue submitting patches as we find bugs in the code rather than having to maintain our own divergencies.

Scope

Note: From reading the upstream changelogs, I don't anticipate that we'll have much trouble with this (with the possible exception of the python package installers noted below). However, this is a change to the code used to build a large number of packages so it's best to both write this as a Change and get started on it early.

Proposal owners

• Update the python-setuptools package from the current code-base (python-distribute-0.6.x) to python-setuptools-0.7.x
  • Done.
  • Updated to python-setuptools-0.8.x
  • Updated to python-setuptools-0.9.x due to a fix for a security issue.
• Rebuild packages with the new setuptools (Only necessary for testing)
• Areas that may need special attention. Note that these are areas that we will test to understand the scope of changes required. It is not a promise to thoroughly test what is broken due to the update or to fix the breakage (although we will help)
  • Test python package installers with the new setuptools (pip, buildout)
    • Status: pip builds
    • python-zc-buildout updated to a 2.2.0 that builds with the new setuptools.
  • Test python packages that build with setuptools for related ftbfs (we could do a mass rebuild of everything requiring python at runtime or refine our list to the packages that BuildRequire: python-setuptools)
    • Status: Built a few packages. So far, things Just Work(TM)
    • http://copr-fe.cloud.fedoraproject.org/coprs/detail/toshio/setuptools-0.7/builds/
    • Listed python packages on the mass rebuild tracker ticket
  • Test packages where we are using setuptools to build multiversion (example: python-cherrypy2)
  • Test python packages that have a runtime requirement on setuptools [Note: The owners of this Change are going to leave most of this to individual package maintainers due to this being highly dependent on what the packages actually do with setuptools at runtime]
• See if our python packaging guidelines need update (particularly the multi-version/python-setuptools/egg guidelines)
  • Status: Doesn't look like it needs an update
• Check that the python-setuptools managed metadata has not changed incompatibly
  • Status: Doesn't look like this has changed

Other developers

Owners of packages which BuildRequire or Require python-setuptools will need to:

• Test for buildtime and runtime breakage from new setuptools (as applicable to each package)
• Fix FTBFS breakage from new setuptools
• Fix runtime breakage from new setuptools

Release engineering

• Rebuild packages which BuildRequire: python-setuptools with the new setuptools
  • Probably not strictly necessary (will confirm when we test whether the package metadata format has changed) but it is good for testing since setuptools is a buildtime dep in many python packages.

Policies and guidelines

Testing so far has revealed no differences between setuptools-0.6 and 0.7+ that requires guideline changes.

Upgrade/compatibility impact

• No data will need migrating but user installed python modules that rely on the old version of python-setuptools may no longer work.
  • Testing thus far does not show any incompatibilities that would make user installed python modules fail.
• Probably no problems for packages installed with the older python-setuptools but change owners need to check that there are no incompatible changes to the package metadata managed by python-setuptools.
  • Testing thus far has not shown any incompatibilities in metadata.
• [EDIT] Found that easy_install's default changed to install zipped eggs instead of unzipped eggs by default. This caused a small change in packaging guidelines to use a command line flag to restore the unzipped eggs behaviour for a few packages (packages which are installing a compatibility version of a python module via easy_install).

How To Test

First install the updated python-setuptools package.

# yum update python-setuptools
# rpm -q python-setuptools |grep 0.7
[should display a python-setuptools 0.7 package]

Check that packages that need setuptools to build still build

Testing with an upstream tarball/checkout:

$ grep setuptools setup.py
$ python setup.py install --root `pwd`/tmp/
[This should work with no errors]

Testing with an rpm package:

$ fedpkg srpm python-$MODULE
$ fedpkg scratch-build --srpm python-$MODULE*src.rpm

Test multiversion python packages

Building should be the same test as above.

Runtime testing:

[ install both versions of the package]
$ yum install python-cherrypy python-cherrypy2
[test that both versions of the package work]
[Install a package that uses the multiversion package as a dependency]
$ yum install bodhi
[test that the package still works (loads the correct version of the multi-version package)]

Check that packages that need setuptools to run still run

$ yum install $PACKAGE
$ rpm -q --requires $PACKAGE|grep python-setuptools
[Displays that $PACKAGE needs python-setuptools]
[Run the program and make use of plugins or other features that need python-setuptools]

Test python package installers with the new setuptools

$ yum install python-pip
$ pip install --user $OTHER_PACKAGE
[Other python-pip commands]

Other packages in this category include python-zc-buildout (python-virtualenv if it finally unbundles pip and setuptools).

User Experience

• Since this update includes API changes, developers who use setuptools as a library may notice changes.

See the following pages for details:

• http://pythonhosted.org/setuptools/merge.html
• https://pypi.python.org/pypi/setuptools#changes

Note that our current pyhton-setuptools package is the distribute fork so the merge changes are relative to that.

Dependencies

To see the dependent packages run these repoquery commands:

[Runtime]
$ repoquery -q --whatrequires python-setuptools
[103 packages]
[Buildtime]
$ repoquery --enablerepo rawhide-source --archlist=src --whatrequires python-setuptools > tmp.lst
$ repoquery --enablerepo rawhide-source --archlist=src --whatrequires python-setuptools-devel >> tmp.lst
$ sort tmp.lst | uniq > setuptools-br.lst
[918 packages]

Contingency Plan

Contingency mechanism: (What to do? Who will do it?)

Three alternate contingency mechanisms exist:

1. Someone who has a package that cannot be ported to the new python-setuptools will need to submit a backwards compat package for the older version of python-setuptools. The Change Owners can commit to reviewing the package but not to being the package owner for the new package.
2. The packages that have not been ported can be dropped from the distribution.
3. The new setuptools package can be reverted for F20.

Deciding which of these to perform depends on the results of testing (which will show how many packages suffer from issues, whether those packages are easy to port, whether the unported packages are essential to the distribution or partially unmaintained, and whether the changes to port other packages to the new setuptools would be hard to revert.)

Contingency deadline

For option 1, adding a backwards compatible package. We would want to add that early enough to be able to make sure that the package is usable by the unported packages. Soon after alpha would be ideal.

For option 2, beta freeze -- but we'd need to evaluate that the packages to be dropped are non-essential to the distribution much earlier. Probable at alpha freeze.

For option 3, it depends on how difficult it would be to back out changes for all packages. At this time, I don't think there will be a widespread need for patching or spec file changes, so this could be done at beta freeze.

Blocks release?

Probably not. Neither anaconda nor yum use this directly and it's not a critpath package. However, we might want to reevaluate this if we find that it blocks building of something essential to the distribution (as it could prevent quick rebuilds of the packages needed for the distribution).

Documentation

Upstream documentation:

• http://pythonhosted.org/setuptools/merge.html
• https://pypi.python.org/pypi/setuptools#changes

Notes from our testing will be added here or a separate wiki page if we encounter incompatibilities in updating.

Release Notes

• If the update does not cause any porting difficulties I would not release note this -- it will likely be just as invisible to end users as it is to us.
  • there hasn't been any difficulties with the current porting. I think we should forgo release notes for this change.

• If the update shows that the incompatibilities affect a lot of code then we should add something like this:

The version of the python-setuptools package has been updated to the 0.9.x series. This release series merges the setuptools and distribute upstream projects which has introduced a variety of changes to the API and behaviour. Please see https://fedoraproject.org/wiki/Changes/Python_setuptools_0.7#Documentation for more details if this affects code you are writing or deploying.