SIGs/Python/UpgradingPython

Upgrading Python from 3.N-1 to 3.N

Here are the notes from upgrading Python from 3.6 to 3.7. They are created to save us time next time when upgrading to 3.8. It is not a comprehensive guide, but if it once gets to it, it'd be great.

For Python 3.9, see section Ideas for Python 3.9 at the bottom.

Create a Change Proposal

Before anything can happen, we need to create a Change Proposal. Follow the normal procedure, get inspired by previous changes, but make sure the process hasn't changed (copypasta is good, but check if the template hasn't been updated).

Create python3N package

This can happen as soon as the first alpha is released. If you have enough resources, try to do it even before. The python3X packages are designed to have flat/nonflat switches since python36, so you will use this package once you want to update python3.

Make a copr repo, bootstrap "important stuff" in it

Bootstrapping Python packages is nontrivial. Chances are high that from the upgrade to 3.N-1, the bootstrap sequence has changed, as more packages switched from python2-sphinx to python3-sphinx or from python3-pytoml to python3-toml etc. Start as early as possible.

For 3.7, I've used rpm-list-builder. The yaml recipe is archived on my GitHub. It has some comments. Note that rpm-list-builder has limitations:

• it doesn't bump releases, so you need to do that manually if you build one package once and the new build has new functionality (aka not just for tests)
• the "continue from here" future isn't very useful for us and was designed for different purpose, so what I did is that most of the yaml was commented out and I was uncommenting sections that I needed keeping finished stuff above, TODO stuff at the bottom
• everything is sequential. go do something different from time to time

What wasn't done for 3.7 but might have been a good idea is to restart the procedure once in a while. Packages early in the bootstrap sequence were built once with alphas and never tested with betas and rc. I suggest restarting the procedure later in the process.

Preserving the yaml recipe is good, you can use it for Fedora later.

• The Python 3.8 recipe: https://github.com/hroncok/rpm-list-builder/blob/python38/python38.yaml
• The Python 3.9 recipe: https://github.com/hroncok/rpm-list-builder/blob/python39/python39.yaml

In the Copr settings, add this external repo:

http://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/

And disable mock bootstrap.

This is the command I've used:

rm work-directory/* -rf && rpmlb --download local --source-directory source-directory python37.yaml python37 --work-directory work-directory --build copr --copr-repo @python/python3.7

I had local fedpkg clones, so I could create quick hotfixes in them. Also a must for python3 package - you have it updated to 3.N, but rawhide still has 3.N-1.

Some packages will need a release bump
It is unfortunately required that you manually bump some packages, especially those with bootstrap bconds. For example when Fedora has python3-setuptools-41.6.0-1.fc32 and you rebuild it with bootstrap enabled without bumping, it is python3-setuptools-41.6.0-1.fc32~bootstrap and that sorts lower. This feature of the dist tag requires you to run rpmdev-bumpspec *.spec on such packages in the source directory prior to proceeding.

Rebuild everything else in copr

This shall give you the packages that need rebuild for Python 3.9:

$ (repoquery --refresh --repo=koji --source --whatrequires 'libpython3.8.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.8' --whatrequires '/usr/bin/python3.8' --whatrequires 'python3.8dist(*)') | pkgname | sort | uniq | egrep -v '^python3$' > python38.pkgs && (repoquery --refresh --repo=python39 --source --whatrequires 'libpython3.9.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.9' --whatrequires '/usr/bin/python3.9' --whatrequires 'python3.9dist(*)') | pkgname | sort | uniq | egrep -v '^python3$' > python39.pkgs && comm -23 python38.pkgs python39.pkgs > todo.pkgs && wc -l *.pkgs

pkgname command is from fedora-packager 0.6.0.5+.

The koji repo is more up to date than rawhide. Use the following repo file if you don't already have it (save as /etc/yum.repos.d/koji.repo):

[koji]
name=koji
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/x86_64/
enabled=0

[koji-source]
name=koji-source
baseurl=http://kojipkgs.fedoraproject.org/repos/rawhide/latest/src/
enabled=0

Inspect the resulting files and manually remove python3.X from all of them. Add all packages to copr:

$ for pkg in $(cat python38.pkgs); do echo $pkg; copr add-package-distgit --name $pkg --webhook-rebuild on @python/python3.9  || copr edit-package-distgit --name $pkg --webhook-rebuild on @python/python3.9; done

And build the ones that are still needed:

$ for pkg in $(cat todo.pkgs); do echo $pkg; copr build-package --nowait --name $pkg @python/python3.9; done

Or, use moreutils-parallel (note that there are two packages in Fedora that provide the parallel command; you want moreutils-parallel):

$ parallel -j 12 copr build-package --nowait @python/python3.9 --name -- $(cat todo.pkgs)

Use monitor_check.py to see and report the failures.

Once ready, move into Fedora proper

Ready can differ. For 3.7 it was when rc1 was released and all important stuff from the yaml recipe was built in copr (in fact, couple of packages were not, but the fix was known).

Request a side tag

Example: Side tag for Python 3.7 releng issue

You'll get a side tag name (target). It can differ. For 3.7 it was f29-python.

To build in the side tag, you do:

fedpkg build --target=<name>

To wait for builds to get into the repos, do:

koji wait-repo <name> --build=<nvr>

If Koji uses the mock's bootstrap option, you need to request it to be disabled for the side tag in order to prevent broken dependencies of dnf in the bootstrap chroot. A releng member can do that via:

koji edit-tag <name> -x 'mock.use_bootstrap=0'

Once dnf and dnf-plugins-core (and/or possibly other Python packages needed in the bootstrap chroot in the future) are rebuilt and installable in the side tag, the mock's bootstrap option can be enabled again (possibly not worth the trouble unless not using it prevents some other builds).

Ask for autosigning to be enabled. That will significantly speed up the merging procedure.

Reproduce the bootstrap sequence

Just get the recipe and rebuild in the side tag. rpm-list-builder could not do it with 3.7 so I've used a dirty script. Be prepared for failures, packages got updated in the meantime and new deps were added. Tough life.

Rebuild everything else

Now, this gets a bit magical but what I did is that I've used mini-mass-rebuild to rebuild everything. Remember to bump the release and use the side tag (the fedpkg-build.sh script needs to be adjusted).

What is everything? First, i get the packages that buildrequire python3-devel:

 dnf repoquery  --disablerepo='*' --enablerepo='rawhide-source' --whatrequires python3-devel

But I forgot the virtual provides:

dnf repoquery  --disablerepo='*' --enablerepo='rawhide' --provides python3-devel

Also a good idea what actually needs a rebuild is:

dnf repoquery --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --whatrequires '/usr/bin/python3.6' --whatrequires 'python3.6dist(*)' --source

Use 3.N-1 instead of 3.6.

Remember that you already built stuff that doesn't need an extra rebuild. To query the side tag repo, use:

baseurl=https://kojipkgs.fedoraproject.org/repos/<target_name>/latest/$basearch/

To remove a list of packages from the list of packages, I use comm (requires sorted files):

comm -23 all_rawhide already_built > remains

There are also build-status.sh and build-failure-reason.sh in mini-mass-rebuild.

parallel -j 32 bash ../build-status.sh -- $(cat ../remains) | tee ../status
for i in free open closed failed canceled; do echo -e $i\\t$(grep " $i" ../status | wc -l); done
cat ../status | grep " closed" | cut -d" " -f1 | tee ../closed
comm -23 ../remains ../closed > ../remains

And:

parallel -j 32 bash ../build-failure-reason.sh -- $(cat ../remains) | tee ../reason
cat ../reason | grep ' build'  # FTBFS
cat ../reason | grep ' root'   # missing dependency

It's a good idea to mass rebuild everything that remains until the number of closed (built) packages stops at 0. Don't accidentally bump the release over and over again. Later, start investigating the build failure reasons. Untangle any circular deps if found. Add them to yaml recipe for next time. File bugs (block the tracking bug, create an alias (PYTHON3N)). E-mail maintainers. Cry.

Request a side tag merge

Once you feel things look good (ask QA to confirm) and the number of FTBFS is reasonably low (for 3.7 it was 300 (half waiting for deps)), ask releng to merge the side tag. Do this at least a week before the mass rebuild. Releng will most likely not respond within a day, so better plan a future date (a ~week in advance would work best). E-mail devel-announce.

It was not yet possible during 3.7, but check with nirik if creating a test compose from side tag is possible, it could prevent issues like this one (anaconda tracebacks, no compose). Without a proper compose, the rawhide repos are not updated.

When the side tag is being merged, be available. After the merge, stuff thet has higher nevr in rawhide will "win" over the side tag. Double check if everything that was bootstrapped (not everything from the recipe, but where actual bootstrapping was involved) has higher epoch:version-release in the side tag - rebuild it in the side tag again if it was bumped in regular rawhide, or you will need to bootstrap it again after the merge.

Once merged, releng will give you list of packages that had higher nevr in rawhide. Bump-rebuild them in rawhide once you can koji-wait for python3-3.N.0... build.

After a side tag merge

If there are still FTBFSes (they are), e-mail the maintainers again.

Monitor the state. Get the initial list of broken deps:

$ (dnf repoquery  --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq > rawhide

Diff and update it regularly:

$ dnf clean all && (dnf repoquery  --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq | diff -u rawhide -

When packages are fixed, check dependent packages. Rebuild those (they might get unblocked).

After the compose is built

The packages are in rawhide repos, yay! Mass rebuild will happen soon, bugs will be open automatically, your job is almost done here. Once the FTBFS bugs are opened, make the change tracking bug depend on them and mark any previously filled bugs as duplicates. You can bump the bugs in bulks (go to show dependency tree and change several).

Keep monitoring stuff. Also monitor devel mialing list for questions.

Rebuild everything in (the 3rd party repo that shall not be named), you can use mini-mass-rebuild with little tweaks. If you are a provenpackager in Fedora, you have the rights here as well, but if you never built anything in there, you need to ask. No nonfree stuff needed rebuilding during 3.7, only free (that might change in the future).

Update the fedora-python-classroom kickstart to not filter Python 3.N, but 3.N-1.

Rebuilds with new betas

When a new Python beta comes out, there can be two kinds of complications:

• The ABI changed, requiring rebuilds of all extencion modules
• The PYC magic number changed, requiring rebuilds of all Python modules

To find out if we have one of these situations, rebuild everything in COPR with the new Python beta and analyze the failures.

First, build the new Python beta release in COPR.

Then, get fresh lists copies of python38.pkgs and python39.pkgs. Combine them and filter out Python itself:

(repoquery --refresh --repo=koji --source --whatrequires 'libpython3.8.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.8') | pkgname | sort | uniq > python38.pkgs
(repoquery --refresh --repo=koji --source --whatrequires 'libpython3.9.so.1.0()(64bit)' --whatrequires 'python(abi) = 3.9') | pkgname | sort | uniq > python39.pkgs
cat python39.pkgs python38.pkgs | sort | uniq | egrep -v '^python3.9$' > torebuild.pkgs

Then, rebuild in COPR:

for pkg in $(torebuild.pkgs); do echo $pkg; copr build-package --nowait --background --name $pkg @python/python3.9; done

Analyze the failures and see what needs to be done when updating to the new beta in Fedora.

Before the beta freeze

Once the Fedora beta freeze is approaching, it's a good idea to ask FESCo to give permission to retire all remaining leaf packages with bugs in NEW. See the Mass retiring packages that still require Python 3.6 in Fedora 29 ticket for example. This removes cruft and it is apparently also the only way to scare some maintainers to fix their packages.

Also, the Developer Portal docs might need updating. This doesn't block anything, so file a bug about it (on a triaged component in Bugzilla), so someone with free cycles can do it someday.

Broken upgrade path

It is likely that some packages that used to require Python 3.N-1 are removed at this point (not only by the abovementioned FESCo ticket). It is crucial to obsolete them, or users would be unable to upgrade to the new Fedora version, having problems like nothing provides python(abi) = 3.N-1 needed by coolpkg-0.1.2-3.fc31.noarch.

Such packages need to be obsoleted via the special fedora-obsolete-packages package. There is a script that can figure out all the needed lines to be added and a Bugzilla about this from Fedora 32 (Python 3.8) that explains the procedure.

It is likely that the list of obsoletes packages will be changing. Repeat this step to update it before the final freeze and also during the Fedora release lifetime.

After the beta freeze

All remaining FTBFS bugs block upgrades to the new Fedora version. Request freeze exceptions for all of them.

Also, create pythonN-1 package

This needs a new package review request. Ask releng to unretire it. If you will create a nice git merge commit and a PR, note that releng will push a revert commit to the retired repo, so do it afterwards to save yourself doing it twice.

Ideas for Python 3.9

Inspiration from Perl

Perl has their own build dependency resolver tailored to Perl. It deals only with Perl build/run-time dependencies, all others are ignored.

• https://repo.or.cz/Fedora-Rebuild.git

Perl has a common bootstrap macro throughout the Perl ecosystem. It turns the build dependency graph into a tree, that can be easily built.

• e.g. https://src.fedoraproject.org/rpms/perl-Alien-Build/blob/rawhide/f/perl-Alien-Build.spec

Applied to Python

We could use the FPC-blessed "bootstrap" bcond, but we would have to set it for every package we're building. However, if we have a Python-specific bootstrap macro (e.g. python_global_bootstrap), we can set it centrally (e.g. in the python3 package) and then every Python package that uses this macro will be built using a bootstrap mode, without us needing to change its specfile.