Upgrading Python from 3.N-1 to 3.N
Here are the notes from upgrading Python from 3.6 to 3.7. They are created to save us time next time when upgrading to 3.8. It is not a comprehensive guide, but if it once gets to it, it'd be great.
Create a Change Proposal
Before anything can happen, we need to create a Change Proposal. Follow the normal procedure, get inspired by previous changes, but make sure the process hasn't changed (copypasta is good, but check if the template hasn't been updated).
Create python3N package
This can happen as soon as the first alpha is released. If you have enough resources, try to do it even before.
The python3X packages are designed to have flat/nonflat switches since 
python36
Make a copr repo, bootstrap "important stuff" in it
Bootstrapping Python packages is nontrivial. Chances are high that from the upgrade to 3.N-1, the bootstrap sequence has changed, as more packages switched to python3-sphinx etc. Start as early as possible.
For 3.7, I've used rpm-list-builder. The yaml recipe is archived on my GitHub. It has some comments. Note that rpm-list-builder has limitations:
- it doesn't bump releases, so you need to do that manually if you build one package once and the new build has new functionality (aka not just for tests)
- the "continue from here" future isn't very useful for us and was designed for different purpose, so what I did is that most of the yaml was commented out and I was uncommenting sections that I needed keeping finished stuff above, TODO stuff at the bottom
- everything is sequential. go do something different from time to time
When
What wasn't done for 3.7 but might have been a good idea is to restart the procedure once in a while. Packages early in the bootstrap sequence were build once with alphas and never tested with betas and rc. I suggest restarting the procedure on beta1 and on rc1.
Preserving the yaml recipe is good, you can use it for Fedora later.
This is the command I've used:
rm work-directory/* -rf && rpmlb --download local --source-directory source-directory python37.yaml python37 --work-directory work-directory --build copr --copr-repo @python/python3.7
I had local fedpkg clones, so I could create quick hotfixes in them. Also a must for
Once ready, move into Fedora proper
Ready can differ. For 3.7 it was when rc1 was released and all important stuff from the yaml recipe was built in copr (in fact, couple of packages were not, but the fix was known).
Request a side tag
Example: Side tag for Python 3.7 releng issue
You'll get a side tag name (target). It can differ. For 3.7 it was f29-python.
To build in the side tag, you do:
fedpkg build --target=<name>
To wait for builds to get into the repos, do:
koji wait-repo <name> --build=<nvr>
Reproduce the bootstrap sequence
Just get the recipe and rebuild in the side tag. rpm-list-builder could not do it with 3.7 so I've used a dirty script. Be prepared for failures, packages got updated in the meantime and new deps were added. Tough life.
Rebuild everything else
Now, this gets a bit magical but what I did is that I've used mini-mass-rebuild to rebuild everything. Remember to bump the release and use the side tag (the fedpkg-build.sh script needs to be adjusted).
What is everything? First, i get the packages that buildrequire
dnf repoquery --disablerepo='*' --enablerepo='rawhide-source' --whatrequires python3-devel
But I forgot the virtual provides:
dnf repoquery --disablerepo='*' --enablerepo='rawhide' --provides python3-devel
Also a good idea what actually needs a rebuild is:
dnf repoquery --whatrequires 'libpython3.6m.so.1.0()(64bit)' --source dnf repoquery --whatrequires 'python(abi) == 3.6' --source
Use 3.N-1 instead of 3.6.
A helpful pkgname command to pipe those things into:
#!/usr/bin/python3
import fileinput
for line in fileinput.input():
print('-'.join(line.split('-')[:-2]))
Remember that you already built stuff that doesn't need an extra rebuild. To remove a list of packages from the list of packages, I use remove_closed.py:
python3 remove_closed.py all_rawhide already_built remains
There are also build-status.sh and build-failure-reason.sh in mini-mass-rebuild.
parallel -j 32 bash ../build-status.sh -- $(cat ../remains) | tee ../status for i in free open closed failed canceled; do echo -e $i\\t$(grep " $i" ../status | wc -l); done cat ../status | grep " closed" | cut -d" " -f1 | tee ../closed python3 ../remove_closed.py ../remains ../closed ../remains
And:
parallel -j 32 bash ../build-failure-reason.sh -- $(cat ../remains) | tee ../reason cat ../reason | grep ' build' # FTBFS cat ../reason | grep ' root' # missing dependency
It's a good idea to mass rebuild everything that remains until the number of closed (built) packages stops at 0. Don't forget to bump the release over and over again. Later, start investigating the build failure reasons. Untangle any circular deps if found. Add them to yaml recipe for next time. File bugs (block the tracking bug, create an alias (PYTHON3N)). E-mail maintainers. Cry.
Request a side tag merge
Once you feel things look good (ask QA to confirm) and the number of FTBFS is reasonably low (for 3.7 it was 300 (half waiting for deps)), ask releng to merge the side tag. Do this at least a week before the mass rebuild. Releng will most likely not respond within a day, so better plan a future date (a ~week in advance would work best). E-mail devel-announce.
It was not yet possible during 3.7, but check with nirik if creating a test compose from side tag is possible, it could prevent issues like this one (anaconda tracebacks, no compose). Without a proper compose, the rawhide repos are not updated.
When the side tag is being merged, be available. After the merge, stuff thet has higher nevr in rawhide will "win" over the side tag.
Double check if
Once merged, releng will give you list of packages that had higher nevr in rawhide. Bump-rebuild them in rawhide once you can koji-wait for python3-3.N.0... build.
After a side tag merge
If there are still FTBFSes (they are), e-mail the maintainers again.
Monitor the state. Get the initial list of broken deps:
$ (dnf repoquery --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --source && dnf repoquery --disablerepo='*' --enablerepo='koji' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq > rawhide
Diff and update it regularly:
$ dnf clean all && (dnf repoquery --disablerepo='*' --enablerepo='koji' --whatrequires 'libpython3.6m.so.1.0()(64bit)' --source && dnf repoquery --disablerepo='*' --enablerepo='koji' --whatrequires 'python(abi) == 3.6' --source) | pkgname | sort | uniq | diff -u rawhide -
When packages are fixed, check dependent packages. Rebuild those (they might get unblocked).
After the compose is built
The packages are in rawhide repos, yay! Mass rebuild will happen soon, bugs will be open automatically, your job is almost done here. Once the FTBFS bugs are opened, make the change tracking bug depend on them and mark any previously filled bugs as duplicates. You can bump the bugs in bulks (go to show dependency tree and change several).
Keep monitoring stuff. Also monitor devel mialing list for questions.
Rebuild everything in (the 3rd party repo that shall not be named), you can use mini-mass-rebuild with little tweaks. If you are a provenpackager in Fedora, you have the rights here as well, but if you never built anything in there, you need to ask. No nonfree stuff needed rebuilding during 3.7, only free (that might change in the future).
Also, create pythonN-1 package
This needs a new package review request. Ask releng to unretire it. If you will create a nice git merge commit and a PR, note that releng will push a revert commit to the retired repo, so do it afterwards to save yourself doing it twice.