SIGs/Python/UpgradingPython

Upgrading Python from 3.N-1 to 3.N

Here are the notes from upgrading Python from 3.6 to 3.7. They are created to save us time next time when upgrading to 3.8. It is not a comprehensive guide, but if it once gets to it, it'd be great.

For Python 3.9, see section Ideas for Python 3.9 at the bottom.

Create a Change Proposal

Before anything can happen, we need to create a Change Proposal. Follow the normal procedure, get inspired by previous changes, but make sure the process hasn't changed (copypasta is good, but check if the template hasn't been updated).

Create python3N package

This can happen as soon as the first alpha is released. If you have enough resources, try to do it even before. The python3X packages are designed to have flat/nonflat switches since python36, so you will use this package once you want to update python3.

Make a copr repo, bootstrap "important stuff" in it

Bootstrapping Python packages is nontrivial. Chances are high that from the upgrade to 3.N-1, the bootstrap sequence has changed, as more packages switched to python3-sphinx etc. Start as early as possible.

For 3.7, I've used rpm-list-builder. The yaml recipe is archived on my GitHub. It has some comments. Note that rpm-list-builder has limitations:

• it doesn't bump releases, so you need to do that manually if you build one package once and the new build has new functionality (aka not just for tests)
• the "continue from here" future isn't very useful for us and was designed for different purpose, so what I did is that most of the yaml was commented out and I was uncommenting sections that I needed keeping finished stuff above, TODO stuff at the bottom
• everything is sequential. go do something different from time to time

What wasn't done for 3.7 but might have been a good idea is to restart the procedure once in a while. Packages early in the bootstrap sequence were build once with alphas and never tested with betas and rc. I suggest restarting the procedure on beta1 and on rc1.

Preserving the yaml recipe is good, you can use it for Fedora later.

Work in Progress Python 3.8 recipe: https://github.com/hroncok/rpm-list-builder/blob/python38/python38.yaml

In the Copr settings, add this external repo:

http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/

This is the command I've used:

rm work-directory/* -rf && rpmlb --download local --source-directory source-directory python37.yaml python37 --work-directory work-directory --build copr --copr-repo @python/python3.7

I had local fedpkg clones, so I could create quick hotfixes in them. Also a must for python3 package - you have it updated to 3.N, but rawhide still has 3.N-1.

Some packages will need a release bump
It is unfortunately required that you manually bump some packages, especially those with bootstrap bconds. For example when Fedora has python3-setuptools-41.6.0-1.fc32 and you rebuild it with bootstrap enabled without bumping, it is python3-setuptools-41.6.0-1.fc32~bootstrap and that sorts lower. This feature of the dist tag requires you to run rpmdev-bumpspec *.spec on such packages in the source directory prior to proceeding.

Rebuild everything else in copr

This shall give you the packages that need rebuild for Python 3.9:

$ (repoquery --refresh --repo=koji --source --whatrequires 'libpython3.8.so.1.0()(64bit)'; repoquery --repo=koji --source --whatrequires 'python(abi) = 3.8') | pkgname | sort | uniq | egrep -v '^python3$' > python38.pkgs && (repoquery --refresh --repo=python39 --source --whatrequires 'libpython3.9.so.1.0()(64bit)'; repoquery --repo=python39 --source --whatrequires 'python(abi) = 3.9') | pkgname | sort | uniq | egrep -v '^python3$' > python39.pkgs && python remove_closed.py python38.pkgs python39.pkgs todo.pkgs && wc -l *.pkgs

See the Rebuild everything else (in Koji) section for pkgname and remove_closed.py.

The koji repo is more up to date than rawhide. Use the following repo file if you don't already have it (save as /etc/yum.repos.d/koji.repo):

[koji]
name=koji
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/
enabled=0

[koji-source]
name=koji-source
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/src/
enabled=0

Inspect the resulting files and manually remove python3.X from all of them. Add all packages to copr:

$ for pkg in $(cat python38.pkgs); do echo $pkg; copr add-package-scm --clone-url https://src.fedoraproject.org/rpms/${pkg}.git --name $pkg --webhook-rebuild on --commit master @python/python3.9  || copr edit-package-scm --clone-url https://src.fedoraproject.org/rpms/${pkg}.git --name $pkg --webhook-rebuild on --commit master @python/python3.9; done

And build the ones that are still needed:

$ for pkg in $(cat todo.pkgs); do echo $pkg; copr build-package --nowait --name $pkg @python/python3.9; done

Use monitor_check.py to see and report the failures.

Once ready, move into Fedora proper

Ready can differ. For 3.7 it was when rc1 was released and all important stuff from the yaml recipe was built in copr (in fact, couple of packages were not, but the fix was known).

Request a side tag

Example: Side tag for Python 3.7 releng issue

You'll get a side tag name (target). It can differ. For 3.7 it was f29-python.

To build in the side tag, you do:

fedpkg build --target=<name>

To wait for builds to get into the repos, do:

koji wait-repo <name> --build=<nvr>

Reproduce the bootstrap sequence

Just get the recipe and rebuild in the side tag. rpm-list-builder could not do it with 3.7 so I've used a dirty script. Be prepared for failures, packages got updated in the meantime and new deps were added. Tough life.

Rebuild everything else

Now, this gets a bit magical but what I did is that I've used mini-mass-rebuild to rebuild everything. Remember to bump the release and use the side tag (the fedpkg-build.sh script needs to be adjusted).

What is everything? First, i get the packages that buildrequire python3-devel:

 dnf repoquery  --disablerepo='*' --enablerepo='rawhide-source' --whatrequires python3-devel

But I forgot the virtual provides:

dnf repoquery  --disablerepo='*' --enablerepo='rawhide' --provides python3-devel

Also a good idea what actually needs a rebuild is:

dnf repoquery --whatrequires 'libpython3.6m.so.1.0()(64bit)' --source
dnf repoquery --whatrequires 'python(abi) == 3.6' --source

Use 3.N-1 instead of 3.6.

A helpful pkgname command to pipe those things into:

#!/usr/bin/python3
import fileinput


for line in fileinput.input():
    print(line.rsplit("-", 2)[0])

Remember that you already built stuff that doesn't need an extra rebuild. To query the side tag repo, use:

baseurl=https://kojipkgs.fedoraproject.org/repos/<target_name>/latest/$basearch/

To remove a list of packages from the list of packages, I use remove_closed.py:

python3 remove_closed.py all_rawhide already_built remains

The same thing should be possible with the "comm" command; please test it and update the Wiki if it works:

comm -23 all_rawhide already_built > remains

There are also build-status.sh and build-failure-reason.sh in mini-mass-rebuild.

parallel -j 32 bash ../build-status.sh -- $(cat ../remains) | tee ../status
for i in free open closed failed canceled; do echo -e $i\\t$(grep " $i" ../status | wc -l); done
cat ../status | grep " closed" | cut -d" " -f1 | tee ../closed
python3 ../remove_closed.py ../remains ../closed ../remains

And:

parallel -j 32 bash ../build-failure-reason.sh -- $(cat ../remains) | tee ../reason
cat ../reason | grep ' build'  # FTBFS
cat ../reason | grep ' root'   # missing dependency

It's a good idea to mass rebuild everything that remains until the number of closed (built) packages stops at 0. Don't accidentally bump the release over and over again. Later, start investigating the build failure reasons. Untangle any circular deps if found. Add them to yaml recipe for next time. File bugs (block the tracking bug, create an alias (PYTHON3N)). E-mail maintainers. Cry.

Request a side tag merge

Once you feel things look good (ask QA to confirm) and the number of FTBFS is reasonably low (for 3.7 it was 300 (half waiting for deps)), ask releng to merge the side tag. Do this at least a week before the mass rebuild. Releng will most likely not respond within a day, so better plan a future date (a ~week in advance would work best). E-mail devel-announce.

It was not yet possible during 3.7, but check with nirik if creating a test compose from side tag is possible, it could prevent issues like this one (anaconda tracebacks, no compose). Without a proper compose, the rawhide repos are not updated.

When the side tag is being merged, be available. After the merge, stuff thet has higher nevr in rawhide will "win" over the side tag. Double check if everything that was bootstrapped (not everything from the recipe, but where actual bootstrapping was involved) has higher epoch:version-release in the side tag - rebuild it in the side tag again if it was bumped in regular rawhide, or you will need to bootstrap it again after the merge.

Once merged, releng will give you list of packages that had higher nevr in rawhide. Bump-rebuild them in rawhide once you can koji-wait for python3-3.N.0... build.

After a side tag merge

If there are still FTBFSes (they are), e-mail the maintainers again.

Monitor the state. Get the initial list of broken deps:

$ (dnf repoquery  --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --source && dnf repoquery  --disablerepo='*' --enablerepo='koji' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq > rawhide

Diff and update it regularly:

$ dnf clean all && (dnf repoquery  --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --source && dnf repoquery  --disablerepo='*' --enablerepo='koji' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq | diff -u rawhide -

When packages are fixed, check dependent packages. Rebuild those (they might get unblocked).

After the compose is built

The packages are in rawhide repos, yay! Mass rebuild will happen soon, bugs will be open automatically, your job is almost done here. Once the FTBFS bugs are opened, make the change tracking bug depend on them and mark any previously filled bugs as duplicates. You can bump the bugs in bulks (go to show dependency tree and change several).

Keep monitoring stuff. Also monitor devel mialing list for questions.

Rebuild everything in (the 3rd party repo that shall not be named), you can use mini-mass-rebuild with little tweaks. If you are a provenpackager in Fedora, you have the rights here as well, but if you never built anything in there, you need to ask. No nonfree stuff needed rebuilding during 3.7, only free (that might change in the future).

Update the fedora-python-classroom kickstart to not filter Python 3.N, but 3.N-1.

Before the beta freeze

Once the beta freeze is approaching, it's a good idea to ask FESCo to give permission to retire all remaining leaf packages with bugs in NEW. See the Mass retiring packages that still require Python 3.6 in Fedora 29 ticket for example. This removes cruft and it is apparently also the only way to scare some maintainers to fix their packages.

Broken upgrade path

It is likely that some packages that used to require Python 3.N-1 are removed at this point (not only by the abovementioned FESCo ticket). It is crucial to obsolete them, or users would be unable to upgrade to the new Fedora version, having problems like nothing provides python(abi) = 3.N-1 needed by coolpkg-0.1.2-3.fc31.noarch.

Such packages need to be obsoleted via the special fedora-obsolete-packages package. There is a script that can figure out all the needed lines to be added and a Bugzilla about this from Fedora 32 (Python 3.8) that explains the procedure.

It is likely that the list of obsoletes packages will be changing. Repeat this step to update it before the final freeze and also during the Fedora release lifetime.

After the beta freeze

All remaining FTBFS bugs block upgrades to the new Fedora version. Request freeze exceptions for all of them.

Also, create pythonN-1 package

This needs a new package review request. Ask releng to unretire it. If you will create a nice git merge commit and a PR, note that releng will push a revert commit to the retired repo, so do it afterwards to save yourself doing it twice.

Ideas for Python 3.9

Inspiration from Perl

Perl has their own build dependency resolver tailored to Perl. It deals only with Perl build/run-time dependencies, all others are ignored.

• https://repo.or.cz/Fedora-Rebuild.git

Perl has a common bootstrap macro throughout the Perl ecosystem. It turns the build dependency graph into a tree, that can be easily built.

• e.g. https://src.fedoraproject.org/rpms/perl-Alien-Build/blob/master/f/perl-Alien-Build.spec

Applied to Python

We could use the FPC-blessed "bootstrap" bcond, but we would have to set it for every package we're building. However, if we have a Python-specific bootstrap macro (e.g. python_global_bootstrap), we can set it centrally (e.g. in the python3 package) and then every Python package that uses this macro will be built using a bootstrap mode, without us needing to change its specfile.