Latest revision as of 19:51, 19 April 2021

Reduce dependencies on python3-setuptools

Summary

We'll do two things to reduce the number of Python packages unnecessarily Requiring python3-setuptools:

python3-devel will no longer Require python3-setuptools
packages with console_scripts/gui_scripts entry points will no longer automatically Require python3-setuptools

Owner

Name: Tomáš Hrnčiar
Name: Miro Hrončok
Email: python-maint@redhat.com

Current status

Targeted release: Fedora Linux 35
Last updated: 2021-04-19
FESCo issue: #2590
Tracker bug: #1946642
Release notes tracker: <will NOT be assigned by the Wrangler>

Detailed Description

Historically, for over a decade, setuptools was omnipresent in Python packaging. For that reason, the python3-devel package required python3-setuptools. This is however changing, with PEP517- and PEP518-based upstream packaging, setuptools changed to a very popular yet only optional way of building packages, many upstreams switched to poetry or flit instead. Since we received multiple user reports about an unnecessary dependency, we wanted to get rid of the dependency, but it was not possible until now.

We'll do the two following changes (neither one depends on the other):

python3-devel will stop Requiring python3-setuptools

When we wanted to remove the dependency before, we have encountered a problem, which is best summarized in the Python spec file:

# This is not "API" (packages that need setuptools should still BuildRequire it)
# However some packages apparently can build both with and without setuptools
# producing egg-info as file or directory (depending on setuptools presence).
# Directory-to-file updates are problematic in RPM, so we ensure setuptools is
# installed when -devel is required.
# See https://bugzilla.redhat.com/show_bug.cgi?id=1623914
# See https://fedoraproject.org/wiki/Packaging:Directory_Replacement
Requires: (python3-setuptools if rpm-build)

To avoid this problem, python3-devel currently requires (python3-setuptools if rpmbuild) for backwards compatibility. Other than that, python3-setuptools was always brought into the buildroot transitively with python3-rpm-generators. Since the generators no longer use setuptools but packaging we can finally remove the dependency.

Packages that don't BuildRequire python3-setuptools but use setuptools during the build will fall into one of the following three categories:

They will fail to build from source with errors like: ModuleNotFoundError: No module named 'setuptools'.
They will successfully build from source because they will fallback to the standard library distutils module - their .egg-info directory will become a text file instead, causing errors when RPM tries to update the package (and they will possibly miss some runtime dependencies because the metadata generated by distutils is not complete).
They will continue to build unchanged because one or more packages they BuildRequire pulls in python3-setuptools transitively - at any point in the future, this can change and either (1) or (2) will happen.

We tested this change in the past when we asked other maintainers to explicitly BuildRequire python3-setuptools. Some of them responded, but many did not. Our analysis was based on grepping sources for any mention of setuptools, such as import setuptools or from setuptools import . Nowadays, the same grep-based query showed us roughly the same number (~350) of packages relying on python3-setuptools without BuildRequiring it. To see the direct impact, we have also briefly removed the Requires from python3-devel package used in testing python3.10 in COPR. It revealed only 140 packages that fail to build from the source because of the change, however many packages bring in python3-setuptools transitively. Some of the packages might build successfully even without python3-setuptools, because they fallback to distutils (from the Python standard library) if dependency on setuptools is not satisfied.

To avoid such problems we will:

Mass update packages we analyzed as impacted to add the missing BuildRequires on python3-setuptools.
Do the change while doing the update from Python 3.9 to Python 3.10 so the path of the .egg-info changes from /usr/lib(64)/python3.9/site-packages/...egg-info to /usr/lib(64)/python3.10/site-packages/...egg-info, hence no longer causing troubles on upgrades if some packages actually do change it from directory to file (and changing it the other way around in the future is safe).

How do we mass update the spec files

You can find the script we will use here. The script tries to detect what notation for BuildRequires definition each spec file uses.

Example of the most common occurrences:

BuildRequires: python%{python3_pkgversion}-setuptools
BuildRequires: %{py3_dist setuptools}
BuildRequires: python3-setuptools
BuildRequires: python3dist(setuptools)

We will try to keep the added BuildRequires as much consistent as possible with the existing Python-related BuildRequires (including e.g. whitespace). We will not bump the release or add a changelog entry (the Python 3.10 rebuild will take care of it). To show how the proposed modifications will look like we plan to create a git repository with all the spec files and commit the change there. The diff was shared on the devel and python-devel mailing lists, so maintainers can easily see the upcoming change and can provide feedback to us. You can find changes done by the script here

When do we update the spec files

We plan to mass update the spec files in the middle of April 2021, if you wish to opt-out from the mass update, please let us know prior to 15th of April 2021.

What spec files we will update

Below is the list of packages we plan to add BuildRequires for python3-setuptools to. If you wish to opt-out of this update, please edit this page and move your package to the list of packages to be ignored (listed below this one).

Packages to be updated

0ad ignatenkobrain pcpa pwalter
OpenMolcas jussilehtola
PyGreSQL hhorak jmlich odubaj panovotn pkubat praiskup
R2spec pingou
ViTables tnorth zbyszek
ansible-review dcallagh ttrinks
barman slaanesh tsao
blender design-sw hobbes1069 ignatenkobrain kwizart luya roma s4504kr slaanesh
btrfs-progs ignatenkobrain josef ngompa sandeen
bumpversion duriantang jdornak
calypso rathann
cinch greghellings
commissaire-client mbarnes smilner
cppcheck c72578 jussilehtola sgrubb
cranc lenkaseg
crudini apevec jruzicka pbrady
cxxtest mgieseki
datanommer ralph
dlib bizdelnick luya
ec2-hibinit-agent davdunc
electrum tredaell
fbthrift dcavalca filbranden salimma
fedora-messaging abompard
firefox alexl caolanm erack gecko-maint jgrulich kalev kengert mbarnes rhughes rstrode sharkcz stransky tpopela ueno xhorak
folly dcavalca filbranden salimma
fontforge frixxon kevin pnemade
gajim michich suraia
gau2grid jussilehtola
gfal2-python adev andreamanzi
ginga lupinix
git-filter-repo asn
gns3-gui kwizart
gns3-net-converter kwizart
gns3-server kwizart nucleo
gpsd fab mlichvar ttorling
h5py stevetraylen terjeros
khard mathstuf sdyroff
kismon fab
koji ausil kevin mikem puiterwijk
lammps ellio167 junghans
legofy lkf williamjmorenor
libcaca hubbitus slaanesh thias
libiptcdata dcm hobbes1069 jchaloup
libnuml sagitter
libreoffice caolanm dtardon erack sbergmann
lldb airlied daveisfera jankratochvil sergesanspaille siddharths tstellar
llvm dmalcolm ignatenkobrain jakub jistone kyle scottt sergesanspaille siddharths tstellar
llvm10 sergesanspaille tstellar
llvm11 sergesanspaille tstellar
llvm7.0 jistone petersen sergesanspaille tstellar
llvm9.0 jistone sergesanspaille tstellar
mercurial kiilerix nbecker pstodulk
mod_wsgi jdornak jkaluza jorton lmacken mrunge
moose zbyszek
mypaint avsej
mysql-connector-python hhorak hubbitus hvad mschorm
nototools mfabian pwu
officeparser rebus
offlineimap cicku dodji sergesanspaille teuf
openbabel alexpl jussilehtola rathann
percol hubbitus
poezio fantom louizatakk
porcupine kushal
protontricks atim
protonvpn-cli calexandru2018 jflory7
prunerepo frostyx praiskup
pyaudio chkr
pyhunspell mfabian
pyodbc fjanus hhorak
pyscard orion sjenning
pyserial stingray
python-CommonMark jujens
python-GridDataFormats rathann
python-OBD rathann
python-Pyped uggla
python-Rtree volter
python-acoustid terjeros
python-agate jujens
python-aiodns fantom
python-ansicolors fab
python-apprise lead2gold
python-apsw cicku dfateyev maci
python-arviz sergiopr
python-astroplan sergiopr
python-astropy-healpix lupinix
python-astroquery lupinix
python-astroscrappy lupinix
python-asttokens zbyszek
python-audioread terjeros
python-autobahn fab jujens
python-autopep8 mrunge ndipanov
python-b2sdk jonny
python-bloom cottsay rmattes
python-blosc tnorth zbyszek
python-box dmsimard fab
python-btchip jonny xenithorb
python-cached_property adamwill immanetize
python-carbon jsteffan piotrp
python-ccdproc lupinix
python-certbot-apache nb
python-chai kevin pingou ralph
python-cmigemo hubbitus
python-colorspacious rathann
python-construct moezroy terjeros
python-contextlib2 abompard pingou ralph tjikkun
python-cookiecutter chedi wakko666
python-crochet abompard
python-css-parser zbyszek
python-dbfread jujens
python-debrepo ktdreyer
python-dialog mjakubicek noodles raphgro sundaram zbyszek
python-dijitso limb
python-django-contact-form mrunge
python-django-health-check dmsimard
python-django-registration orphan
python-django-reversion mrunge
python-django-tagging jdornak mrunge piotrp
python-django-tastypie bkabrda cquad mrunge stevetraylen
python-docx kushal124
python-dtfabric fab
python-dukpy zbyszek
python-editorconfig barracks510
python-ephem fab
python-et_xmlfile jujens
python-etcd mbarnes smilner
python-fasteners mrunge
python-ffc zbyszek
python-fields cottsay
python-fisx zbyszek
python-fitsio lupinix
python-flake8-docstrings cottsay
python-flask-gravatar devrim orphan
python-flask-htmlmin devrim iztokf
python-flask-paranoid devrim orphan
python-flask-rstpages rmarko
python-flask-security devrim orphan
python-flask-sphinx-themes devrim orphan
python-flask-wtf-decorators frostyx
python-formats uggla
python-fypp rathann
python-gevent-eventemitter atim
python-google-i18n-address pwouters
python-googletrans lyessaadi
python-graphql-relay fab
python-heapdict qulogic
python-html5-parser kevin
python-htmlmin jujens
python-humblewx rickardlindberg
python-hupper kevin
python-i3ipc alebastr
python-inotify jfilak stevetraylen terjeros
python-jep raphgro
python-jinja2-cli jujens
python-jinja2-time chedi wakko666
python-jnius raphgro
python-joblib besser82 ignatenkobrain sergiopr
python-jsonmodels oanson
python-jsonrpclib ihrachyshka jonny
python-kerberos rcritten simo
python-kitchen kevin pingou ralph
python-landslide echevemaster salimma
python-lark-parser totol
python-leather jujens
python-libsass nonamedotc
python-libusb1 jonny
python-lmdb jruzicka pspacek tkrizek
python-logfury jonny
python-managesieve stevetraylen
python-matrix-nio ankursinha
python-meld3 kevin stevetraylen tsao
python-minibelt uggla
python-mmtf rathann
python-mnemonic jonny
python-music21 zbyszek
python-mwclient adamwill rdieter tuxbrewr
python-myhdl filiperosset
python-nbclient nonamedotc
python-nbxmpp michich suraia
python-networkmanager jdulaney
python-notario ktdreyer
python-oauth2 ignatenkobrain pjp spot sundaram
python-openoffice sharkcz
python-ouimeaux kni
python-pandas-datareader sergiopr
python-paste-script andreamanzi
python-patsy sergiopr
python-pbkdf2 jonny
python-pecan-notario ktdreyer
python-pelican firemanxbr mrunge
python-pexpect amcnabb fabiand ignatenkobrain radez swt2c tomspur
python-plaster-pastedeploy abompard
python-plumbum greghellings lorenzodalrio
python-polib cicku dchen diegobz dshea ivazquez moezroy suanand
python-precis_i18n michich
python-proteus sharkcz
python-publicsuffix2 rathann
python-pulsectl pfrields
python-pvc raphgro
python-pycares fantom
python-pycha potty sharkcz
python-pygeoip kevin ralph
python-pylons-sphinx-themes abompard
python-pymc3 sergiopr
python-pynn ankursinha
python-pyotp icon
python-pypng kevin ralph
python-pyqtgraph swt2c
python-pyramid_sawing abompard
python-pysb zbyszek
python-pysignals kni
python-pyswip pampelmuse
python-pyte terjeros
python-pytest-astropy-header sergiopr
python-pytest-mock fab jujens
python-pytest-repeat cottsay
python-pytest-watch jujens
python-pyvo lupinix
python-readthedocs-sphinx-ext jjames
python-recommonmark jujens
python-relatorio sharkcz
python-restructuredtext-lint jujens
python-requests-cache codeblock hobbes1069
python-retrying apevec
python-rmtest lberk mgoodwin nathans
python-rosdep cottsay rmattes thofmann
python-sanction kevin ralph
python-scikit-learn besser82 ignatenkobrain lupinix sergiopr
python-scrapy echevemaster
python-setuptools-lint jdulaney
python-shamir-mnemonic jonny
python-simplemediawiki lmacken potty ralph
python-slixmpp fantom louizatakk
python-smbc twaugh zdohnal
python-snappy jujens
python-social-auth-core cqi
python-soupsieve zbyszek
python-spdx jbertozzi
python-spdx-lookup jbertozzi
python-sphinxcontrib-bibtex jjames
python-statsd pabelanger tdecacqu
python-steam atim
python-tables tnorth zbyszek
python-tempdir rathann
python-timeout-decorator jcapitao
python-tinydb suanand
python-tortilla uggla
python-tqdm ignatenkobrain sgallagh
python-tree-format chedi wakko666
python-trezor jonny
python-twilio mich181189
python-txaio fab jujens
python-unidecode pjp sundaram
python-unidiff dcallagh
python-upt-cpan jbertozzi
python-upt-fedora jbertozzi
python-upt-pypi jbertozzi
python-upt-rubygems jbertozzi
python-urwidtrees ttomecek
python-vdf atim
python-wand barracks510
python-watchdog jsteffan jujens pingou
python-webencodings abompard
python-webpy mrunge
python-wsaccel jujens
python-xlwt leamas moezroy rathann
python-xvfbwrapper mrunge totol
python-yarl fab ignatenkobrain
python-zstandard rathann
python3-postgresql hhorak
python3-pytest-asyncio jujens
python3-saml dcallagh tchaikov
qemu berrange bonzini crobinso dwmw2 ehabkost jforbes lkundrak quintela rjones
rdkit giallu
root ellert
rpmspectool nphilipp
salt blarson dmurphy18 krionbsd
sepolicy_analysis vmojzis
solaar brouhaha rathann richardfearn tibbs
spec2scl jstanek
starcal hedayat
stomppy stevetraylen
sugar-speak callkalpa chimosky pbrobinson tuxbrewr
swid-tools adelton
swift-lang tachoknight
terminator dmaphy mattrose ohaessler
texlive-base spot
toot alciregi
translate-toolkit cicku dwayne petersen suanand
trellis lkundrak somlo
tryton sharkcz
trytond sharkcz
trytond-account sharkcz
trytond-account-be sharkcz
trytond-account-de-skr03 sharkcz
trytond-account-invoice sharkcz
trytond-account-invoice-history sharkcz
trytond-account-invoice-line-standalone sharkcz
trytond-account-product sharkcz
trytond-account-statement sharkcz
trytond-account-stock-anglo-saxon sharkcz
trytond-account-stock-continental sharkcz
trytond-analytic-account sharkcz
trytond-analytic-invoice sharkcz
trytond-analytic-purchase sharkcz
trytond-analytic-sale sharkcz
trytond-company sharkcz
trytond-company-work-time sharkcz
trytond-country sharkcz
trytond-currency sharkcz
trytond-dashboard sharkcz
trytond-google-maps sharkcz
trytond-ldap-authentication sharkcz
trytond-party sharkcz
trytond-party-siret sharkcz
trytond-product sharkcz
trytond-product-cost-fifo sharkcz
trytond-product-cost-history sharkcz
trytond-product-price-list sharkcz
trytond-project sharkcz
trytond-project-plan sharkcz
trytond-project-revenue sharkcz
trytond-purchase sharkcz
trytond-purchase-invoice-line-standalone sharkcz
trytond-sale sharkcz
trytond-sale-opportunity sharkcz
trytond-sale-price-list sharkcz
trytond-stock sharkcz
trytond-stock-forecast sharkcz
trytond-stock-inventory-location sharkcz
trytond-stock-location-sequence sharkcz
trytond-stock-product-location sharkcz
trytond-stock-supply sharkcz
trytond-stock-supply-day sharkcz
trytond-timesheet sharkcz
ubertooth avsej
uhd jskarvad
upt jbertozzi
uwsgi kad
watchman dcavalca filbranden salimma
wine-mono mooninite
winpdb spot
xrootd ellert simonm
xtensor-python sergesanspaille
yawn jsafrane miminar vcrhonek

Packages to be ignored

fontforge - false positive
gaupol - false positive: setuptools import is in setup-aeidon.py, which is unused; aeidon is instead built with setup.py using distutils only
poetry - has the setuptools imports in integration tests only, will add the requirement to test buildrequires manually if needed
python-poetry-core - seems like a false positive, will investigate
qemu - false positive
xrootd - false positive

Packages with console_scripts/gui_scripts entry points will no longer automatically Require python3.Xdist(setuptools)

Packages that use the console_scripts or gui_scripts entry points used to generate scripts for them that imported pkg_resources (from setuptools) on runtime and thus all such packages needed to Require setuptools on runtime. This is an example of such generated script from Fedora 32:

#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'fedpkg==1.40','console_scripts','fedpkg'
__requires__ = 'fedpkg==1.40'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', , sys.argv[0])
    sys.exit(
        load_entry_point('fedpkg==1.40', 'console_scripts', 'fedpkg')()
    )

The Python dist RPM dependency generator (/usr/lib/rpm/fileattrs/pythondist.attr) hence generated a Requires on python3.Xdist(setuptools) for all packages with scripts like the one above.

Starting from Python 3.8 and setuptools 47, this is no longer required, because the scripts will use importlib.metadata from the standard library instead of pkg_resources from setuptools. The scripts now looks like this:

#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'fedpkg==1.40','console_scripts','fedpkg'
import re
import sys

# for compatibility with easy_install; see #2198
__requires__ = 'fedpkg==1.40'

try:
    from importlib.metadata import distribution
except ImportError:
    try:
        from importlib_metadata import distribution
    except ImportError:
        from pkg_resources import load_entry_point
...

Since the scripts are generated during build, the packages no longer need to Require setuptools on runtime. The Python dist RPM dependency generator will no longer generate the Requires on python3.Xdist(setuptools).

The change is prepared in a pull request (https://src.fedoraproject.org/rpms/python-rpm-generators/pull-request/38). We plan to merge it as soon as it is ready. It only changes the behavior for Python 3.10 or newer, hence it will take affect in the Python 3.10 copr only, until Python is updated to 3.10 in Fedora proper.

Packages that actually import from the setuptools or pkg_resources module on runtime should explicitly require setuptools, but some of the upstreams may not do that. We don't expect many packages negatively affected by this, but since this is not strictly backwards compatible, we are only planning to do it in Fedora 35+.

Relevant Pull Requests:

Feedback

There was some discussion on various mailing list threads, but no other alternatives were proposed.

Benefit to Fedora

From Fedora Packaging Guidelines: It is important that your package list all necessary build dependencies using the BuildRequires: tag. As our testing showed, this is not true for many packages. Some of them can successfully build either with or without setuptools (they use try-except import and fallback to distutils from the standard library). Such packages are especially dangerous when not BuildRequiring setuptools -- they can produce different results depending on the presence of setuptools: either an .egg-info metadata directory (w/setuptools) or .egg-info text file (w/distutils). RPM has troubles when upgrading directories to files.

More Python applications will be installable without setutpools, hence contributing to the general minimization objective.

Scope

Some packages use distutils as a fallback in case setuptools is unavailable. The problem is that setuptools generates egg.info directory and distutils creates egg.info file. This would lead to failures caused by the known issue of RPM that cannot change directory to file. To avoid this we plan to do a mass spec update before we upgrade Python to version 3.10. This way we ensure that path will be correct after the mass rebuild.

Proposal owners:

Test affected packages in Copr.
Do the mass spec update to introduce missing BuildRequire to them.

Other developers: If your package requires setuptools on runtime, make sure it specifies it explicitly. There should be no additional work for developers other than that. There is a possibility that some packages will slip through our grep based query and fail to build, but we'll detect this in our testing Copr and fix them.

Release engineering: not needed for this Change

Policies and guidelines: already exist

Trademark approval: not needed for this Change

Alignment with Objectives: minimization

Upgrade/compatibility impact

All affected packages needs rebuild after the mass spec update but since we plan to do it with the upgrade to Python 3.10 we will rebuild it altogether. Users should not be affected by upgrades in any way.

How To Test

Interested testers can push explicit BuildRequire on setuptools to the distgit and see in our Python 3.10 COPR whether the package starts to build again. Webhook should trigger rebuild automatically on new commit in rawhide branch.

User Experience

Regular distro users shouldn't notice any change in system behaviour, except for packages that use setuptools or pkg_resources on runtime and fail to specify that dependency. Such packages will fail to run with ModuleNotFoundErrors and should be fixed by their maintainers.

Dependencies

300+ packages will need to BuildRequire setuptools explicitly. This will be done by the mass spec update.

Contingency Plan

Contingency mechanism: Commit removing python3-devel Requires on (python3-setuptools if rpmbuild) will be reverted. The change in Python dist RPM dependency generator will be reverted (Each of the contingency mechanisms can be applied independently.)
Contingency deadline: mass rebuild
Blocks release? No

Documentation

This page is a documentation, this is not a user facing change.

Release Notes

Search

Changes/Reduce dependencies on python3-setuptools: Difference between revisions