From Fedora Project Wiki
 
(65 intermediate revisions by 5 users not shown)
Line 2: Line 2:
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->
<!-- The actual name of your proposed change page should look something like: Changes/Your_Change_Proposal_Name.  This keeps all change proposals in the same namespace -->


= python3-devel won't pull in setuptools by default in rpmbuild =
= Reduce dependencies on python3-setuptools =
<!-- The name of your change proposal -->
<!-- The name of your change proposal -->


== Summary ==
== Summary ==
python3-devel currently requires python-setuptools if rpmbuild is used. Due to a change in RPM generators, this is not needed anymore hence we are removing it.
We'll do two things to reduce the number of Python packages unnecessarily Requiring python3-setuptools:
 
# python3-devel will no longer Require python3-setuptools
# packages with console_scripts/gui_scripts entry points will no longer automatically Require python3-setuptools
 
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->


Line 23: Line 27:


== Current status ==
== Current status ==
[[Category:ChangePageIncomplete]]
[[Category:ChangeAcceptedF35]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
Line 40: Line 44:
ON_QA -> change is fully code complete
ON_QA -> change is fully code complete
-->
-->
* FESCo issue: <will be assigned by the Wrangler>
* FESCo issue: [https://pagure.io/fesco/issue/2590 #2590]
* Tracker bug: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1946642 #1946642]
* Release notes tracker: <will be assigned by the Wrangler>
* Release notes tracker: <will NOT be assigned by the Wrangler>


== Detailed Description ==
== Detailed Description ==
<!-- Expand on the summary, if appropriate. A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
Historically, for over a decade, [https://pypi.org/project/setuptools/ setuptools] was omnipresent in Python packaging.
python3-devel currently requires python3-setuptools if rpmbuild is used. This was needed for RPM generators but since it adapted packaging, this is not needed anymore hence we are removing it. We tested this change in the past when we [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/GCPGM34ZGEOVUHSBGZTRYR5XKHTIJ3T7/ asked] other maintainers to explicitly BuildRequire python-3setuptools. Some of them responded, some not. Our analysis was based on greping sources for any mention of setuptools, such as ''import setuptools'' or ''from setuptools import''. Nowadays, the same grep based query showed us roughly the same number (about 335 + 72 that failed to be analysed) of packages relying on python3-setuptools. To see direct impact we removed this Require from python-devel package used in testing [https://copr.fedorainfracloud.org/coprs/g/python/python3.10/packages/ python3.10] COPR. It showed us 140 packages affected by this change. Even though it is fewer than expected 300+ packages, to avoid any breakages we plan to do a mass spec update on all of them to explicitly BuildRequire python3-setuptools. Some of them might build because they fallback to distutils if dependency on python3-setuptools is not satisfied.
For that reason, the python3-devel package required python3-setuptools.
This is however changing, with [https://www.python.org/dev/peps/pep-0517/ PEP517]- and [https://www.python.org/dev/peps/pep-0518/ PEP518]-based upstream packaging, setuptools changed to a very popular yet only optional way of building packages, many upstreams switched to poetry or flit instead.
Since we received multiple user reports about an unnecessary dependency, we wanted to [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/ARFZB56ULRQNT3FEAC4YXXZDIUHG4M5P/ get rid of the dependency], but it was not possible until now.
 
We'll do the two following changes (neither one depends on the other):
 
=== python3-devel will stop Requiring python3-setuptools ===
 
When we wanted to remove the dependency before, we have encountered a problem, which is best summarized in the Python spec file:
# This is not "API" (packages that need setuptools should still BuildRequire it)
# However some packages apparently can build both with and without setuptools
# producing egg-info as file or directory (depending on setuptools presence).
# Directory-to-file updates are problematic in RPM, so we ensure setuptools is
# installed when -devel is required.
# See https://bugzilla.redhat.com/show_bug.cgi?id=1623914
# See https://fedoraproject.org/wiki/Packaging:Directory_Replacement
Requires: (python3-setuptools if rpm-build)
 
To avoid this problem, python3-devel currently requires `(python3-setuptools if rpmbuild)` for backwards compatibility.
Other than that, python3-setuptools was always brought into the buildroot transitively with python3-rpm-generators.
Since the [https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/57QTZGF7FJPE2QQSGPJ3RGI4G5SLN6BM/ generators no longer use setuptools but packaging] we can finally remove the dependency.
 
Packages that don't BuildRequire python3-setuptools but use setuptools during the build will fall into one of the following three categories:
 
* They will fail to build from source with errors like: `ModuleNotFoundError: No module named 'setuptools'`.
* They will successfully build from source because they will fallback to the standard library distutils module - their `.egg-info` directory will become a text file instead, causing errors when RPM tries to update the package (and they will possibly miss some runtime dependencies because the metadata generated by distutils is not complete).
* They will continue to build unchanged because one or more packages they BuildRequire pulls in python3-setuptools transitively - at any point in the future, this can change and either (1) or (2) will happen.
 
We tested this change in the past when we [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/GCPGM34ZGEOVUHSBGZTRYR5XKHTIJ3T7/ asked] other maintainers to explicitly BuildRequire python3-setuptools. Some of them responded, but many did not. Our analysis was based on grepping sources for any mention of setuptools, such as `import setuptools` or `from setuptools import `. Nowadays, the same grep-based query showed us roughly the same number (~350) of packages relying on python3-setuptools without BuildRequiring it. To see the direct impact, we have also briefly removed the Requires from python3-devel package used in testing [https://copr.fedorainfracloud.org/coprs/g/python/python3.10/packages/ python3.10] in COPR. It revealed only 140 packages that fail to build from the source because of the change, however many packages bring in python3-setuptools transitively. Some of the packages might build successfully even without python3-setuptools, because they fallback to distutils (from the Python standard library) if dependency on setuptools is not satisfied.
 
To avoid such problems we will:
 
* Mass update packages we analyzed as impacted to add the missing BuildRequires on python3-setuptools.
* Do the change while doing the [[Changes/Python3.10|update from Python 3.9 to Python 3.10]] so the path of the `.egg-info` changes from `/usr/lib(64)/python3.9/site-packages/...egg-info` to `/usr/lib(64)/python3.10/site-packages/...egg-info`, hence no longer causing troubles on upgrades if some packages actually do change it from directory to file (and changing it the other way around in the future is safe).
 
==== How do we mass update the spec files ====
 
You can find the script we will use [https://github.com/hrnciar/add-setuptools-buildrequire here]. The script tries to detect what notation for BuildRequires definition each spec file uses.
 
Example of the most common occurrences:
 
BuildRequires: python%{python3_pkgversion}-setuptools
BuildRequires: %{py3_dist setuptools}
BuildRequires: python3-setuptools
BuildRequires: python3dist(setuptools)
 
We will try to keep the added BuildRequires as much consistent as possible with the existing Python-related BuildRequires (including e.g. whitespace). We will not bump the release or add a changelog entry (the Python 3.10 rebuild will take care of it).  To show how the proposed modifications will look like we plan to create a git repository with all the spec files and commit the change there. The diff [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/NTC35XOIEGBICNFG3CZIFPBRCK5K5MPX/ was] shared on the devel and python-devel mailing lists, so maintainers can easily see the upcoming change and can provide feedback to us. You can find changes done by the script [https://github.com/hrnciar/add-setuptools-buildrequire/compare/c082a1a...a60a447 here]
 
==== When do we update the spec files ====
 
We plan to mass update the spec files in the middle of April 2021, if you wish to opt-out from the mass update, please let us know prior to 15th of April 2021.
 
==== What spec files we will update ====
 
Below is the list of packages we plan to add BuildRequires for python3-setuptools to. If you wish to opt-out of this update, please edit this page and move your package to the list of packages to be ignored (listed below this one).
 
===== Packages to be updated =====
 
* 0ad                  ignatenkobrain pcpa pwalter
* OpenMolcas          jussilehtola
* PyGreSQL            hhorak jmlich odubaj panovotn pkubat praiskup
* R2spec              pingou
* ViTables            tnorth zbyszek
* ansible-review      dcallagh ttrinks
* barman              slaanesh tsao
* blender              design-sw hobbes1069 ignatenkobrain kwizart luya roma s4504kr slaanesh
* btrfs-progs          ignatenkobrain josef ngompa sandeen
* bumpversion          duriantang jdornak
* calypso              rathann
* cinch                greghellings
* commissaire-client  mbarnes smilner
* cppcheck            c72578 jussilehtola sgrubb
* cranc                lenkaseg
* crudini              apevec jruzicka pbrady
* cxxtest              mgieseki
* datanommer          ralph
* dlib                bizdelnick luya
* ec2-hibinit-agent    davdunc
* electrum            tredaell
* fbthrift            dcavalca filbranden salimma
* fedora-messaging    abompard
* firefox              alexl caolanm erack gecko-maint jgrulich kalev kengert mbarnes rhughes rstrode sharkcz stransky tpopela ueno xhorak
* folly                dcavalca filbranden salimma
* fontforge            frixxon kevin pnemade
* gajim                michich suraia
* gau2grid            jussilehtola
* gfal2-python        adev andreamanzi
* ginga                lupinix
* git-filter-repo      asn
* gns3-gui            kwizart
* gns3-net-converter  kwizart
* gns3-server          kwizart nucleo
* gpsd                fab mlichvar ttorling
* h5py                stevetraylen terjeros
* khard                mathstuf sdyroff
* kismon              fab
* koji                ausil kevin mikem puiterwijk
* lammps              ellio167 junghans
* legofy              lkf williamjmorenor
* libcaca              hubbitus slaanesh thias
* libiptcdata          dcm hobbes1069 jchaloup
* libnuml              sagitter
* libreoffice          caolanm dtardon erack sbergmann
* lldb                airlied daveisfera jankratochvil sergesanspaille siddharths tstellar
* llvm                dmalcolm ignatenkobrain jakub jistone kyle scottt sergesanspaille siddharths tstellar
* llvm10              sergesanspaille tstellar
* llvm11              sergesanspaille tstellar
* llvm7.0              jistone petersen sergesanspaille tstellar
* llvm9.0              jistone sergesanspaille tstellar
* mercurial            kiilerix nbecker pstodulk
* mod_wsgi            jdornak jkaluza jorton lmacken mrunge
* moose                zbyszek
* mypaint              avsej
* mysql-connector-python hhorak hubbitus hvad mschorm
* nototools            mfabian pwu
* officeparser        rebus
* offlineimap          cicku dodji sergesanspaille teuf
* openbabel            alexpl jussilehtola rathann
* percol              hubbitus
* poezio              fantom louizatakk
* porcupine            kushal
* protontricks        atim
* protonvpn-cli        calexandru2018 jflory7
* prunerepo            frostyx praiskup
* pyaudio              chkr
* pyhunspell          mfabian
* pyodbc              fjanus hhorak
* pyscard              orion sjenning
* pyserial            stingray
* python-CommonMark    jujens
* python-GridDataFormats rathann
* python-OBD          rathann
* python-Pyped        uggla
* python-Rtree        volter
* python-acoustid      terjeros
* python-agate        jujens
* python-aiodns        fantom
* python-ansicolors    fab
* python-apprise      lead2gold
* python-apsw          cicku dfateyev maci
* python-arviz        sergiopr
* python-astroplan    sergiopr
* python-astropy-healpix lupinix
* python-astroquery    lupinix
* python-astroscrappy  lupinix
* python-asttokens    zbyszek
* python-audioread    terjeros
* python-autobahn      fab jujens
* python-autopep8      mrunge ndipanov
* python-b2sdk        jonny
* python-bloom        cottsay rmattes
* python-blosc        tnorth zbyszek
* python-box          dmsimard fab
* python-btchip        jonny xenithorb
* python-cached_property adamwill immanetize
* python-carbon        jsteffan piotrp
* python-ccdproc      lupinix
* python-certbot-apache nb
* python-chai          kevin pingou ralph
* python-cmigemo      hubbitus
* python-colorspacious rathann
* python-construct    moezroy terjeros
* python-contextlib2  abompard pingou ralph tjikkun
* python-cookiecutter  chedi wakko666
* python-crochet      abompard
* python-css-parser    zbyszek
* python-dbfread      jujens
* python-debrepo      ktdreyer
* python-dialog        mjakubicek noodles raphgro sundaram zbyszek
* python-dijitso      limb
* python-django-contact-form mrunge
* python-django-health-check dmsimard
* python-django-registration orphan
* python-django-reversion mrunge
* python-django-tagging jdornak mrunge piotrp
* python-django-tastypie bkabrda cquad mrunge stevetraylen
* python-docx          kushal124
* python-dtfabric      fab
* python-dukpy        zbyszek
* python-editorconfig  barracks510
* python-ephem        fab
* python-et_xmlfile    jujens
* python-etcd          mbarnes smilner
* python-fasteners    mrunge
* python-ffc          zbyszek
* python-fields        cottsay
* python-fisx          zbyszek
* python-fitsio        lupinix
* python-flake8-docstrings cottsay
* python-flask-gravatar devrim orphan
* python-flask-htmlmin devrim iztokf
* python-flask-paranoid devrim orphan
* python-flask-rstpages rmarko
* python-flask-security devrim orphan
* python-flask-sphinx-themes devrim orphan
* python-flask-wtf-decorators frostyx
* python-formats      uggla
* python-fypp          rathann
* python-gevent-eventemitter atim
* python-google-i18n-address pwouters
* python-googletrans  lyessaadi
* python-graphql-relay fab
* python-heapdict      qulogic
* python-html5-parser  kevin
* python-htmlmin      jujens
* python-humblewx      rickardlindberg
* python-hupper        kevin
* python-i3ipc        alebastr
* python-inotify      jfilak stevetraylen terjeros
* python-jep          raphgro
* python-jinja2-cli    jujens
* python-jinja2-time  chedi wakko666
* python-jnius        raphgro
* python-joblib        besser82 ignatenkobrain sergiopr
* python-jsonmodels    oanson
* python-jsonrpclib    ihrachyshka jonny
* python-kerberos      rcritten simo
* python-kitchen      kevin pingou ralph
* python-landslide    echevemaster salimma
* python-lark-parser  totol
* python-leather      jujens
* python-libsass      nonamedotc
* python-libusb1      jonny
* python-lmdb          jruzicka pspacek tkrizek
* python-logfury      jonny
* python-managesieve  stevetraylen
* python-matrix-nio    ankursinha
* python-meld3        kevin stevetraylen tsao
* python-minibelt      uggla
* python-mmtf          rathann
* python-mnemonic      jonny
* python-music21      zbyszek
* python-mwclient      adamwill rdieter tuxbrewr
* python-myhdl        filiperosset
* python-nbclient      nonamedotc
* python-nbxmpp        michich suraia
* python-networkmanager jdulaney
* python-notario      ktdreyer
* python-oauth2        ignatenkobrain pjp spot sundaram
* python-openoffice    sharkcz
* python-ouimeaux      kni
* python-pandas-datareader sergiopr
* python-paste-script  andreamanzi
* python-patsy        sergiopr
* python-pbkdf2        jonny
* python-pecan-notario ktdreyer
* python-pelican      firemanxbr mrunge
* python-pexpect      amcnabb fabiand ignatenkobrain radez swt2c tomspur
* python-plaster-pastedeploy abompard
* python-plumbum      greghellings lorenzodalrio
* python-polib        cicku dchen diegobz dshea ivazquez moezroy suanand
* python-precis_i18n  michich
* python-proteus      sharkcz
* python-publicsuffix2 rathann
* python-pulsectl      pfrields
* python-pvc          raphgro
* python-pycares      fantom
* python-pycha        potty sharkcz
* python-pygeoip      kevin ralph
* python-pylons-sphinx-themes abompard
* python-pymc3        sergiopr
* python-pynn          ankursinha
* python-pyotp        icon
* python-pypng        kevin ralph
* python-pyqtgraph    swt2c
* python-pyramid_sawing abompard
* python-pysb          zbyszek
* python-pysignals    kni
* python-pyswip        pampelmuse
* python-pyte          terjeros
* python-pytest-astropy-header sergiopr
* python-pytest-mock  fab jujens
* python-pytest-repeat cottsay
* python-pytest-watch  jujens
* python-pyvo          lupinix
* python-readthedocs-sphinx-ext jjames
* python-recommonmark  jujens
* python-relatorio    sharkcz
* python-restructuredtext-lint jujens
* python-requests-cache codeblock hobbes1069
* python-retrying      apevec
* python-rmtest        lberk mgoodwin nathans
* python-rosdep        cottsay rmattes thofmann
* python-sanction      kevin ralph
* python-scikit-learn  besser82 ignatenkobrain lupinix sergiopr
* python-scrapy        echevemaster
* python-setuptools-lint jdulaney
* python-shamir-mnemonic jonny
* python-simplemediawiki lmacken potty ralph
* python-slixmpp      fantom louizatakk
* python-smbc          twaugh zdohnal
* python-snappy        jujens
* python-social-auth-core cqi
* python-soupsieve    zbyszek
* python-spdx          jbertozzi
* python-spdx-lookup  jbertozzi
* python-sphinxcontrib-bibtex jjames
* python-statsd        pabelanger tdecacqu
* python-steam        atim
* python-tables        tnorth zbyszek
* python-tempdir      rathann
* python-timeout-decorator jcapitao
* python-tinydb        suanand
* python-tortilla      uggla
* python-tqdm          ignatenkobrain sgallagh
* python-tree-format  chedi wakko666
* python-trezor        jonny
* python-twilio        mich181189
* python-txaio        fab jujens
* python-unidecode    pjp sundaram
* python-unidiff      dcallagh
* python-upt-cpan      jbertozzi
* python-upt-fedora    jbertozzi
* python-upt-pypi      jbertozzi
* python-upt-rubygems  jbertozzi
* python-urwidtrees    ttomecek
* python-vdf          atim
* python-wand          barracks510
* python-watchdog      jsteffan jujens pingou
* python-webencodings  abompard
* python-webpy        mrunge
* python-wsaccel      jujens
* python-xlwt          leamas moezroy rathann
* python-xvfbwrapper  mrunge totol
* python-yarl          fab ignatenkobrain
* python-zstandard    rathann
* python3-postgresql  hhorak
* python3-pytest-asyncio jujens
* python3-saml        dcallagh tchaikov
* qemu                berrange bonzini crobinso dwmw2 ehabkost jforbes lkundrak quintela rjones
* rdkit                giallu
* root                ellert
* rpmspectool          nphilipp
* salt                blarson dmurphy18 krionbsd
* sepolicy_analysis    vmojzis
* solaar              brouhaha rathann richardfearn tibbs
* spec2scl            jstanek
* starcal              hedayat
* stomppy              stevetraylen
* sugar-speak          callkalpa chimosky pbrobinson tuxbrewr
* swid-tools          adelton
* swift-lang          tachoknight
* terminator          dmaphy mattrose ohaessler
* texlive-base        spot
* toot                alciregi
* translate-toolkit    cicku dwayne petersen suanand
* trellis              lkundrak somlo
* tryton              sharkcz
* trytond              sharkcz
* trytond-account      sharkcz
* trytond-account-be  sharkcz
* trytond-account-de-skr03 sharkcz
* trytond-account-invoice sharkcz
* trytond-account-invoice-history sharkcz
* trytond-account-invoice-line-standalone sharkcz
* trytond-account-product sharkcz
* trytond-account-statement sharkcz
* trytond-account-stock-anglo-saxon sharkcz
* trytond-account-stock-continental sharkcz
* trytond-analytic-account sharkcz
* trytond-analytic-invoice sharkcz
* trytond-analytic-purchase sharkcz
* trytond-analytic-sale sharkcz
* trytond-company      sharkcz
* trytond-company-work-time sharkcz
* trytond-country      sharkcz
* trytond-currency    sharkcz
* trytond-dashboard    sharkcz
* trytond-google-maps  sharkcz
* trytond-ldap-authentication sharkcz
* trytond-party        sharkcz
* trytond-party-siret  sharkcz
* trytond-product      sharkcz
* trytond-product-cost-fifo sharkcz
* trytond-product-cost-history sharkcz
* trytond-product-price-list sharkcz
* trytond-project      sharkcz
* trytond-project-plan sharkcz
* trytond-project-revenue sharkcz
* trytond-purchase    sharkcz
* trytond-purchase-invoice-line-standalone sharkcz
* trytond-sale        sharkcz
* trytond-sale-opportunity sharkcz
* trytond-sale-price-list sharkcz
* trytond-stock        sharkcz
* trytond-stock-forecast sharkcz
* trytond-stock-inventory-location sharkcz
* trytond-stock-location-sequence sharkcz
* trytond-stock-product-location sharkcz
* trytond-stock-supply sharkcz
* trytond-stock-supply-day sharkcz
* trytond-timesheet    sharkcz
* ubertooth            avsej
* uhd                  jskarvad
* upt                  jbertozzi
* uwsgi                kad
* watchman            dcavalca filbranden salimma
* wine-mono            mooninite
* winpdb              spot
* xrootd              ellert simonm
* xtensor-python      sergesanspaille
* yawn                jsafrane miminar vcrhonek
 
===== Packages to be ignored =====
 
* fontforge - false positive
* gaupol - false positive: setuptools import is in setup-aeidon.py, which is unused; aeidon is instead built with setup.py using distutils only
* poetry - has the setuptools imports in integration tests only, will add the requirement to test buildrequires manually if needed
* python-poetry-core - seems like a false positive, will investigate
* qemu - false positive
* xrootd - false positive
 
=== Packages with console_scripts/gui_scripts entry points will no longer automatically Require python3.Xdist(setuptools) ===
 
Packages that use the `console_scripts` or `gui_scripts` entry points used to generate scripts for them that imported `pkg_resources` (from setuptools) on runtime and thus all such packages needed to Require setuptools on runtime. This is an example of such generated script from Fedora 32:
 
#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'fedpkg==1.40','console_scripts','fedpkg'
__requires__ = 'fedpkg==1.40'
import re
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('fedpkg==1.40', 'console_scripts', 'fedpkg')()
    )
 
The Python dist RPM dependency generator (`/usr/lib/rpm/fileattrs/pythondist.attr`) hence generated a Requires on `python3.Xdist(setuptools)` for all packages with scripts like the one above.
 
Starting from Python 3.8 and setuptools 47, this is no longer required, because the scripts will use `importlib.metadata` from the standard library instead of `pkg_resources` from setuptools. The scripts now looks like this:
 
#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'fedpkg==1.40','console_scripts','fedpkg'
import re
import sys
# for compatibility with easy_install; see #2198
__requires__ = 'fedpkg==1.40'
try:
    from importlib.metadata import distribution
except ImportError:
    try:
        from importlib_metadata import distribution
    except ImportError:
        from pkg_resources import load_entry_point
...
 
Since the scripts are generated during build, the packages no longer need to Require setuptools on runtime.
The Python dist RPM dependency generator will no longer generate the Requires on `python3.Xdist(setuptools)`.
 
The change is prepared in a pull request (https://src.fedoraproject.org/rpms/python-rpm-generators/pull-request/38). We plan to merge it as soon as it is ready. It only changes the behavior for Python 3.10 or newer, hence it will take affect in the Python 3.10 copr only, until Python is updated to 3.10 in Fedora proper.
 
Packages that actually import from the `setuptools` or `pkg_resources` module on runtime should explicitly require setuptools, but some of the upstreams may not do that. We don't expect many packages negatively affected by this, but since this is not strictly backwards compatible, we are only planning to do it in Fedora 35+.
 
Relevant Pull Requests:
 
* https://github.com/GNS3/gns3-gui/pull/3157
* https://github.com/GNS3/gns3-server/pull/1880
* https://github.com/PyCQA/isort/pull/1699
* https://github.com/aio-libs/aiosmtpd/pull/267
* https://github.com/andreikop/enki/pull/472
* https://github.com/ansible-community/ansible-bender/pull/250
* https://github.com/astropy/astropy/pull/11455
* https://github.com/celery/celery/pull/6704
* https://github.com/click-contrib/click-man/pull/48
* https://github.com/crossbario/autobahn-python/pull/1481
* https://github.com/deluge-torrent/deluge/pull/309
* https://github.com/fedora-infra/bodhi/pull/4199
* https://github.com/fsfe/reuse-tool/pull/342
* https://github.com/httpie/httpie/pull/1049
* https://github.com/jbarlow83/OCRmyPDF/pull/755
* https://github.com/mkdocs/mkdocs/pull/2347
* https://github.com/nphilipp/rpmspectool/pull/14
* https://github.com/nvbn/thefuck/pull/1180
* https://github.com/packit/packit/pull/1188
* https://github.com/pecan/pecan/pull/122
* https://github.com/persepolisdm/persepolis/pull/858
* https://github.com/pytest-dev/pytest/pull/8503
* https://github.com/ros-infrastructure/catkin_pkg/pull/305
* https://github.com/spyking-circus/spyking-circus/pull/351
* https://github.com/stephenfin/rst2txt/pull/1
* https://github.com/twisted/towncrier/pull/333
* https://github.com/willthames/ansible-review/pull/94
* https://github.com/wireservice/csvkit/pull/1115
* https://gitlab.com/goobook/goobook/-/merge_requests/14
* https://lab.louiz.org/poezio/poezio/-/merge_requests/208
* https://pagure.io/copr/copr/pull-request/1754
* https://pagure.io/fedora-business-cards/pull-request/12
* https://pagure.io/fedscm-admin/pull-request/65
* https://pagure.io/flatpak-module-tools/pull-request/13
* https://pagure.io/fm-orchestrator/pull-request/1695
* https://pagure.io/pagure/pull-request/5143
* https://pagure.io/rpmdeplint/pull-request/2
* https://review.opendev.org/c/zuul/zuul-client/+/783970
* https://review.opendev.org/c/zuul/zuul/+/783967
* https://src.fedoraproject.org/rpms/gns3-net-converter/pull-request/1
* https://src.fedoraproject.org/rpms/resultsdb/pull-request/
* https://twistedmatrix.com/trac/ticket/10164


== Feedback ==
== Feedback ==
<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->
<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->
There was some discussion on various mailing list threads, but no other alternatives were proposed.
* https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/GCPGM34ZGEOVUHSBGZTRYR5XKHTIJ3T7/
* https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/GCPGM34ZGEOVUHSBGZTRYR5XKHTIJ3T7/
* https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/BL7XMXVEHSDZDMH22YET3I4EK66PK4NI/
* https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/BL7XMXVEHSDZDMH22YET3I4EK66PK4NI/


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 81: Line 590:


From [https://docs.fedoraproject.org/en-US/packaging-guidelines/#buildrequires Fedora Packaging Guidelines]: ''It is important that your package list all necessary build dependencies using the BuildRequires: tag.'' As our testing showed, this is not true for many packages. Some of them can successfully build either with or without setuptools (they use try-except import and fallback to distutils from the standard library). Such packages are especially dangerous when not BuildRequiring setuptools -- they can produce different results depending on the presence of setuptools: either an .egg-info metadata directory (w/setuptools) or .egg-info text file (w/distutils). RPM has troubles when [https://docs.fedoraproject.org/en-US/packaging-guidelines/Directory_Replacement/ upgrading directories to files].
From [https://docs.fedoraproject.org/en-US/packaging-guidelines/#buildrequires Fedora Packaging Guidelines]: ''It is important that your package list all necessary build dependencies using the BuildRequires: tag.'' As our testing showed, this is not true for many packages. Some of them can successfully build either with or without setuptools (they use try-except import and fallback to distutils from the standard library). Such packages are especially dangerous when not BuildRequiring setuptools -- they can produce different results depending on the presence of setuptools: either an .egg-info metadata directory (w/setuptools) or .egg-info text file (w/distutils). RPM has troubles when [https://docs.fedoraproject.org/en-US/packaging-guidelines/Directory_Replacement/ upgrading directories to files].
More Python applications will be installable without setutpools, hence contributing to the general minimization objective.


== Scope ==
== Scope ==
 
Some packages use distutils as a fallback in case setuptools is unavailable. The problem is that setuptools generates egg.info directory and distutils creates egg.info file. This would lead to failures caused by the known issue of RPM that cannot change directory to file. To avoid this we plan to do a mass spec update before we upgrade Python to version 3.10. This way we ensure that path will be correct after the mass rebuild.
We will coordinate the work in a side tag and merge when ready.


* Proposal owners:
* Proposal owners:
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
*# Test affected packages in Copr.
# Test affected packages in Copr.
*# Do the mass spec update to introduce missing BuildRequire to them.
# Do the mass spec update to introduce missing BuildRequire to them.


* Other developers: There should be no additional work for other developers. BuildRequiring setuptools might not be enough for all packages, but we will take care of them manually if it happens.
* Other developers: If your package requires setuptools on runtime, make sure it specifies it explicitly. There should be no additional work for developers other than that. There is a possibility that some packages will slip through our grep based query and fail to build, but we'll detect this in our testing Copr and fix them.
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Release engineering: [https://pagure.io/releng/issues #Releng issue number] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Release engineering: not needed for this Change <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->


* Policies and guidelines: N/A (not needed for this Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Policies and guidelines: already exist <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. Please submit a pull request with the proposed changes before submitting your Change proposal. -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. Please submit a pull request with the proposed changes before submitting your Change proposal. -->


* Trademark approval: N/A (not needed for this Change)
* Trademark approval: not needed for this Change
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://pagure.io/Fedora-Council/tickets/issues ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://pagure.io/Fedora-Council/tickets/issues ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->


* Alignment with Objectives: N/A
* Alignment with Objectives: minimization
<!-- Does your proposal align with the current Fedora Objectives: https://docs.fedoraproject.org/en-US/project/objectives/ ? It's okay if it doesn't, but it's something to consider -->
<!-- Does your proposal align with the current Fedora Objectives: https://docs.fedoraproject.org/en-US/project/objectives/ ? It's okay if it doesn't, but it's something to consider -->


Line 111: Line 621:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
All affected packages needs rebuild after the mass spec update.
All affected packages needs rebuild after the mass spec update but since we plan to do it with the upgrade to Python 3.10 we will rebuild it altogether.
Users should not be affected by upgrades in any way.


== How To Test ==
== How To Test ==
Line 129: Line 640:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
Interested testers can push explicit BuildRequire on setuptools to the distgit and see in our [https://copr.fedorainfracloud.org/coprs/g/python/python3.10/ Python 3.10 COPR] whether the package starts to build again. Webhook should trigger rebuild automatically on new commit in rawhide branch.
Interested testers can push explicit BuildRequire on setuptools to the distgit and see in our [https://copr.fedorainfracloud.org/coprs/g/python/python3.10/ Python 3.10 COPR] whether the package starts to build again. Webhook should trigger rebuild automatically on new commit in rawhide branch.


Line 144: Line 654:
-->
-->


Regular distro users shouldn't notice any change in system behaviour.
Regular distro users shouldn't notice any change in system behaviour,
except for packages that use setuptools or pkg_resources on runtime and fail to specify that dependency.
Such packages will fail to run with ModuleNotFoundErrors and should be fixed by their maintainers.


== Dependencies ==
== Dependencies ==
Line 150: Line 662:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
140 packages will need to BuildRequire setuptools explicitly. This will be done by the mass spec update.
300+ packages will need to BuildRequire setuptools explicitly. This will be done by the mass spec update.


== Contingency Plan ==
== Contingency Plan ==


<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency mechanism: Commit removing Requires on python-setuptools will be reverted. <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency mechanism: Commit removing python3-devel Requires on `(python3-setuptools if rpmbuild)` will be reverted. The change in Python dist RPM dependency generator will be reverted (Each of the contingency mechanisms can be applied independently.) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
* Contingency deadline: mass rebuild  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency deadline: mass rebuild  <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->

Latest revision as of 19:51, 19 April 2021


Reduce dependencies on python3-setuptools

Summary

We'll do two things to reduce the number of Python packages unnecessarily Requiring python3-setuptools:

  1. python3-devel will no longer Require python3-setuptools
  2. packages with console_scripts/gui_scripts entry points will no longer automatically Require python3-setuptools


Owner

Current status

  • Targeted release: Fedora Linux 35
  • Last updated: 2021-04-19
  • FESCo issue: #2590
  • Tracker bug: #1946642
  • Release notes tracker: <will NOT be assigned by the Wrangler>

Detailed Description

Historically, for over a decade, setuptools was omnipresent in Python packaging. For that reason, the python3-devel package required python3-setuptools. This is however changing, with PEP517- and PEP518-based upstream packaging, setuptools changed to a very popular yet only optional way of building packages, many upstreams switched to poetry or flit instead. Since we received multiple user reports about an unnecessary dependency, we wanted to get rid of the dependency, but it was not possible until now.

We'll do the two following changes (neither one depends on the other):

python3-devel will stop Requiring python3-setuptools

When we wanted to remove the dependency before, we have encountered a problem, which is best summarized in the Python spec file:

# This is not "API" (packages that need setuptools should still BuildRequire it)
# However some packages apparently can build both with and without setuptools
# producing egg-info as file or directory (depending on setuptools presence).
# Directory-to-file updates are problematic in RPM, so we ensure setuptools is
# installed when -devel is required.
# See https://bugzilla.redhat.com/show_bug.cgi?id=1623914
# See https://fedoraproject.org/wiki/Packaging:Directory_Replacement
Requires: (python3-setuptools if rpm-build)

To avoid this problem, python3-devel currently requires (python3-setuptools if rpmbuild) for backwards compatibility. Other than that, python3-setuptools was always brought into the buildroot transitively with python3-rpm-generators. Since the generators no longer use setuptools but packaging we can finally remove the dependency.

Packages that don't BuildRequire python3-setuptools but use setuptools during the build will fall into one of the following three categories:

  • They will fail to build from source with errors like: ModuleNotFoundError: No module named 'setuptools'.
  • They will successfully build from source because they will fallback to the standard library distutils module - their .egg-info directory will become a text file instead, causing errors when RPM tries to update the package (and they will possibly miss some runtime dependencies because the metadata generated by distutils is not complete).
  • They will continue to build unchanged because one or more packages they BuildRequire pulls in python3-setuptools transitively - at any point in the future, this can change and either (1) or (2) will happen.

We tested this change in the past when we asked other maintainers to explicitly BuildRequire python3-setuptools. Some of them responded, but many did not. Our analysis was based on grepping sources for any mention of setuptools, such as import setuptools or from setuptools import . Nowadays, the same grep-based query showed us roughly the same number (~350) of packages relying on python3-setuptools without BuildRequiring it. To see the direct impact, we have also briefly removed the Requires from python3-devel package used in testing python3.10 in COPR. It revealed only 140 packages that fail to build from the source because of the change, however many packages bring in python3-setuptools transitively. Some of the packages might build successfully even without python3-setuptools, because they fallback to distutils (from the Python standard library) if dependency on setuptools is not satisfied.

To avoid such problems we will:

  • Mass update packages we analyzed as impacted to add the missing BuildRequires on python3-setuptools.
  • Do the change while doing the update from Python 3.9 to Python 3.10 so the path of the .egg-info changes from /usr/lib(64)/python3.9/site-packages/...egg-info to /usr/lib(64)/python3.10/site-packages/...egg-info, hence no longer causing troubles on upgrades if some packages actually do change it from directory to file (and changing it the other way around in the future is safe).

How do we mass update the spec files

You can find the script we will use here. The script tries to detect what notation for BuildRequires definition each spec file uses.

Example of the most common occurrences:

BuildRequires: python%{python3_pkgversion}-setuptools
BuildRequires: %{py3_dist setuptools}
BuildRequires: python3-setuptools
BuildRequires: python3dist(setuptools)

We will try to keep the added BuildRequires as much consistent as possible with the existing Python-related BuildRequires (including e.g. whitespace). We will not bump the release or add a changelog entry (the Python 3.10 rebuild will take care of it). To show how the proposed modifications will look like we plan to create a git repository with all the spec files and commit the change there. The diff was shared on the devel and python-devel mailing lists, so maintainers can easily see the upcoming change and can provide feedback to us. You can find changes done by the script here

When do we update the spec files

We plan to mass update the spec files in the middle of April 2021, if you wish to opt-out from the mass update, please let us know prior to 15th of April 2021.

What spec files we will update

Below is the list of packages we plan to add BuildRequires for python3-setuptools to. If you wish to opt-out of this update, please edit this page and move your package to the list of packages to be ignored (listed below this one).

Packages to be updated
  • 0ad ignatenkobrain pcpa pwalter
  • OpenMolcas jussilehtola
  • PyGreSQL hhorak jmlich odubaj panovotn pkubat praiskup
  • R2spec pingou
  • ViTables tnorth zbyszek
  • ansible-review dcallagh ttrinks
  • barman slaanesh tsao
  • blender design-sw hobbes1069 ignatenkobrain kwizart luya roma s4504kr slaanesh
  • btrfs-progs ignatenkobrain josef ngompa sandeen
  • bumpversion duriantang jdornak
  • calypso rathann
  • cinch greghellings
  • commissaire-client mbarnes smilner
  • cppcheck c72578 jussilehtola sgrubb
  • cranc lenkaseg
  • crudini apevec jruzicka pbrady
  • cxxtest mgieseki
  • datanommer ralph
  • dlib bizdelnick luya
  • ec2-hibinit-agent davdunc
  • electrum tredaell
  • fbthrift dcavalca filbranden salimma
  • fedora-messaging abompard
  • firefox alexl caolanm erack gecko-maint jgrulich kalev kengert mbarnes rhughes rstrode sharkcz stransky tpopela ueno xhorak
  • folly dcavalca filbranden salimma
  • fontforge frixxon kevin pnemade
  • gajim michich suraia
  • gau2grid jussilehtola
  • gfal2-python adev andreamanzi
  • ginga lupinix
  • git-filter-repo asn
  • gns3-gui kwizart
  • gns3-net-converter kwizart
  • gns3-server kwizart nucleo
  • gpsd fab mlichvar ttorling
  • h5py stevetraylen terjeros
  • khard mathstuf sdyroff
  • kismon fab
  • koji ausil kevin mikem puiterwijk
  • lammps ellio167 junghans
  • legofy lkf williamjmorenor
  • libcaca hubbitus slaanesh thias
  • libiptcdata dcm hobbes1069 jchaloup
  • libnuml sagitter
  • libreoffice caolanm dtardon erack sbergmann
  • lldb airlied daveisfera jankratochvil sergesanspaille siddharths tstellar
  • llvm dmalcolm ignatenkobrain jakub jistone kyle scottt sergesanspaille siddharths tstellar
  • llvm10 sergesanspaille tstellar
  • llvm11 sergesanspaille tstellar
  • llvm7.0 jistone petersen sergesanspaille tstellar
  • llvm9.0 jistone sergesanspaille tstellar
  • mercurial kiilerix nbecker pstodulk
  • mod_wsgi jdornak jkaluza jorton lmacken mrunge
  • moose zbyszek
  • mypaint avsej
  • mysql-connector-python hhorak hubbitus hvad mschorm
  • nototools mfabian pwu
  • officeparser rebus
  • offlineimap cicku dodji sergesanspaille teuf
  • openbabel alexpl jussilehtola rathann
  • percol hubbitus
  • poezio fantom louizatakk
  • porcupine kushal
  • protontricks atim
  • protonvpn-cli calexandru2018 jflory7
  • prunerepo frostyx praiskup
  • pyaudio chkr
  • pyhunspell mfabian
  • pyodbc fjanus hhorak
  • pyscard orion sjenning
  • pyserial stingray
  • python-CommonMark jujens
  • python-GridDataFormats rathann
  • python-OBD rathann
  • python-Pyped uggla
  • python-Rtree volter
  • python-acoustid terjeros
  • python-agate jujens
  • python-aiodns fantom
  • python-ansicolors fab
  • python-apprise lead2gold
  • python-apsw cicku dfateyev maci
  • python-arviz sergiopr
  • python-astroplan sergiopr
  • python-astropy-healpix lupinix
  • python-astroquery lupinix
  • python-astroscrappy lupinix
  • python-asttokens zbyszek
  • python-audioread terjeros
  • python-autobahn fab jujens
  • python-autopep8 mrunge ndipanov
  • python-b2sdk jonny
  • python-bloom cottsay rmattes
  • python-blosc tnorth zbyszek
  • python-box dmsimard fab
  • python-btchip jonny xenithorb
  • python-cached_property adamwill immanetize
  • python-carbon jsteffan piotrp
  • python-ccdproc lupinix
  • python-certbot-apache nb
  • python-chai kevin pingou ralph
  • python-cmigemo hubbitus
  • python-colorspacious rathann
  • python-construct moezroy terjeros
  • python-contextlib2 abompard pingou ralph tjikkun
  • python-cookiecutter chedi wakko666
  • python-crochet abompard
  • python-css-parser zbyszek
  • python-dbfread jujens
  • python-debrepo ktdreyer
  • python-dialog mjakubicek noodles raphgro sundaram zbyszek
  • python-dijitso limb
  • python-django-contact-form mrunge
  • python-django-health-check dmsimard
  • python-django-registration orphan
  • python-django-reversion mrunge
  • python-django-tagging jdornak mrunge piotrp
  • python-django-tastypie bkabrda cquad mrunge stevetraylen
  • python-docx kushal124
  • python-dtfabric fab
  • python-dukpy zbyszek
  • python-editorconfig barracks510
  • python-ephem fab
  • python-et_xmlfile jujens
  • python-etcd mbarnes smilner
  • python-fasteners mrunge
  • python-ffc zbyszek
  • python-fields cottsay
  • python-fisx zbyszek
  • python-fitsio lupinix
  • python-flake8-docstrings cottsay
  • python-flask-gravatar devrim orphan
  • python-flask-htmlmin devrim iztokf
  • python-flask-paranoid devrim orphan
  • python-flask-rstpages rmarko
  • python-flask-security devrim orphan
  • python-flask-sphinx-themes devrim orphan
  • python-flask-wtf-decorators frostyx
  • python-formats uggla
  • python-fypp rathann
  • python-gevent-eventemitter atim
  • python-google-i18n-address pwouters
  • python-googletrans lyessaadi
  • python-graphql-relay fab
  • python-heapdict qulogic
  • python-html5-parser kevin
  • python-htmlmin jujens
  • python-humblewx rickardlindberg
  • python-hupper kevin
  • python-i3ipc alebastr
  • python-inotify jfilak stevetraylen terjeros
  • python-jep raphgro
  • python-jinja2-cli jujens
  • python-jinja2-time chedi wakko666
  • python-jnius raphgro
  • python-joblib besser82 ignatenkobrain sergiopr
  • python-jsonmodels oanson
  • python-jsonrpclib ihrachyshka jonny
  • python-kerberos rcritten simo
  • python-kitchen kevin pingou ralph
  • python-landslide echevemaster salimma
  • python-lark-parser totol
  • python-leather jujens
  • python-libsass nonamedotc
  • python-libusb1 jonny
  • python-lmdb jruzicka pspacek tkrizek
  • python-logfury jonny
  • python-managesieve stevetraylen
  • python-matrix-nio ankursinha
  • python-meld3 kevin stevetraylen tsao
  • python-minibelt uggla
  • python-mmtf rathann
  • python-mnemonic jonny
  • python-music21 zbyszek
  • python-mwclient adamwill rdieter tuxbrewr
  • python-myhdl filiperosset
  • python-nbclient nonamedotc
  • python-nbxmpp michich suraia
  • python-networkmanager jdulaney
  • python-notario ktdreyer
  • python-oauth2 ignatenkobrain pjp spot sundaram
  • python-openoffice sharkcz
  • python-ouimeaux kni
  • python-pandas-datareader sergiopr
  • python-paste-script andreamanzi
  • python-patsy sergiopr
  • python-pbkdf2 jonny
  • python-pecan-notario ktdreyer
  • python-pelican firemanxbr mrunge
  • python-pexpect amcnabb fabiand ignatenkobrain radez swt2c tomspur
  • python-plaster-pastedeploy abompard
  • python-plumbum greghellings lorenzodalrio
  • python-polib cicku dchen diegobz dshea ivazquez moezroy suanand
  • python-precis_i18n michich
  • python-proteus sharkcz
  • python-publicsuffix2 rathann
  • python-pulsectl pfrields
  • python-pvc raphgro
  • python-pycares fantom
  • python-pycha potty sharkcz
  • python-pygeoip kevin ralph
  • python-pylons-sphinx-themes abompard
  • python-pymc3 sergiopr
  • python-pynn ankursinha
  • python-pyotp icon
  • python-pypng kevin ralph
  • python-pyqtgraph swt2c
  • python-pyramid_sawing abompard
  • python-pysb zbyszek
  • python-pysignals kni
  • python-pyswip pampelmuse
  • python-pyte terjeros
  • python-pytest-astropy-header sergiopr
  • python-pytest-mock fab jujens
  • python-pytest-repeat cottsay
  • python-pytest-watch jujens
  • python-pyvo lupinix
  • python-readthedocs-sphinx-ext jjames
  • python-recommonmark jujens
  • python-relatorio sharkcz
  • python-restructuredtext-lint jujens
  • python-requests-cache codeblock hobbes1069
  • python-retrying apevec
  • python-rmtest lberk mgoodwin nathans
  • python-rosdep cottsay rmattes thofmann
  • python-sanction kevin ralph
  • python-scikit-learn besser82 ignatenkobrain lupinix sergiopr
  • python-scrapy echevemaster
  • python-setuptools-lint jdulaney
  • python-shamir-mnemonic jonny
  • python-simplemediawiki lmacken potty ralph
  • python-slixmpp fantom louizatakk
  • python-smbc twaugh zdohnal
  • python-snappy jujens
  • python-social-auth-core cqi
  • python-soupsieve zbyszek
  • python-spdx jbertozzi
  • python-spdx-lookup jbertozzi
  • python-sphinxcontrib-bibtex jjames
  • python-statsd pabelanger tdecacqu
  • python-steam atim
  • python-tables tnorth zbyszek
  • python-tempdir rathann
  • python-timeout-decorator jcapitao
  • python-tinydb suanand
  • python-tortilla uggla
  • python-tqdm ignatenkobrain sgallagh
  • python-tree-format chedi wakko666
  • python-trezor jonny
  • python-twilio mich181189
  • python-txaio fab jujens
  • python-unidecode pjp sundaram
  • python-unidiff dcallagh
  • python-upt-cpan jbertozzi
  • python-upt-fedora jbertozzi
  • python-upt-pypi jbertozzi
  • python-upt-rubygems jbertozzi
  • python-urwidtrees ttomecek
  • python-vdf atim
  • python-wand barracks510
  • python-watchdog jsteffan jujens pingou
  • python-webencodings abompard
  • python-webpy mrunge
  • python-wsaccel jujens
  • python-xlwt leamas moezroy rathann
  • python-xvfbwrapper mrunge totol
  • python-yarl fab ignatenkobrain
  • python-zstandard rathann
  • python3-postgresql hhorak
  • python3-pytest-asyncio jujens
  • python3-saml dcallagh tchaikov
  • qemu berrange bonzini crobinso dwmw2 ehabkost jforbes lkundrak quintela rjones
  • rdkit giallu
  • root ellert
  • rpmspectool nphilipp
  • salt blarson dmurphy18 krionbsd
  • sepolicy_analysis vmojzis
  • solaar brouhaha rathann richardfearn tibbs
  • spec2scl jstanek
  • starcal hedayat
  • stomppy stevetraylen
  • sugar-speak callkalpa chimosky pbrobinson tuxbrewr
  • swid-tools adelton
  • swift-lang tachoknight
  • terminator dmaphy mattrose ohaessler
  • texlive-base spot
  • toot alciregi
  • translate-toolkit cicku dwayne petersen suanand
  • trellis lkundrak somlo
  • tryton sharkcz
  • trytond sharkcz
  • trytond-account sharkcz
  • trytond-account-be sharkcz
  • trytond-account-de-skr03 sharkcz
  • trytond-account-invoice sharkcz
  • trytond-account-invoice-history sharkcz
  • trytond-account-invoice-line-standalone sharkcz
  • trytond-account-product sharkcz
  • trytond-account-statement sharkcz
  • trytond-account-stock-anglo-saxon sharkcz
  • trytond-account-stock-continental sharkcz
  • trytond-analytic-account sharkcz
  • trytond-analytic-invoice sharkcz
  • trytond-analytic-purchase sharkcz
  • trytond-analytic-sale sharkcz
  • trytond-company sharkcz
  • trytond-company-work-time sharkcz
  • trytond-country sharkcz
  • trytond-currency sharkcz
  • trytond-dashboard sharkcz
  • trytond-google-maps sharkcz
  • trytond-ldap-authentication sharkcz
  • trytond-party sharkcz
  • trytond-party-siret sharkcz
  • trytond-product sharkcz
  • trytond-product-cost-fifo sharkcz
  • trytond-product-cost-history sharkcz
  • trytond-product-price-list sharkcz
  • trytond-project sharkcz
  • trytond-project-plan sharkcz
  • trytond-project-revenue sharkcz
  • trytond-purchase sharkcz
  • trytond-purchase-invoice-line-standalone sharkcz
  • trytond-sale sharkcz
  • trytond-sale-opportunity sharkcz
  • trytond-sale-price-list sharkcz
  • trytond-stock sharkcz
  • trytond-stock-forecast sharkcz
  • trytond-stock-inventory-location sharkcz
  • trytond-stock-location-sequence sharkcz
  • trytond-stock-product-location sharkcz
  • trytond-stock-supply sharkcz
  • trytond-stock-supply-day sharkcz
  • trytond-timesheet sharkcz
  • ubertooth avsej
  • uhd jskarvad
  • upt jbertozzi
  • uwsgi kad
  • watchman dcavalca filbranden salimma
  • wine-mono mooninite
  • winpdb spot
  • xrootd ellert simonm
  • xtensor-python sergesanspaille
  • yawn jsafrane miminar vcrhonek
Packages to be ignored
  • fontforge - false positive
  • gaupol - false positive: setuptools import is in setup-aeidon.py, which is unused; aeidon is instead built with setup.py using distutils only
  • poetry - has the setuptools imports in integration tests only, will add the requirement to test buildrequires manually if needed
  • python-poetry-core - seems like a false positive, will investigate
  • qemu - false positive
  • xrootd - false positive

Packages with console_scripts/gui_scripts entry points will no longer automatically Require python3.Xdist(setuptools)

Packages that use the console_scripts or gui_scripts entry points used to generate scripts for them that imported pkg_resources (from setuptools) on runtime and thus all such packages needed to Require setuptools on runtime. This is an example of such generated script from Fedora 32:

#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'fedpkg==1.40','console_scripts','fedpkg'
__requires__ = 'fedpkg==1.40'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', , sys.argv[0])
    sys.exit(
        load_entry_point('fedpkg==1.40', 'console_scripts', 'fedpkg')()
    )

The Python dist RPM dependency generator (/usr/lib/rpm/fileattrs/pythondist.attr) hence generated a Requires on python3.Xdist(setuptools) for all packages with scripts like the one above.

Starting from Python 3.8 and setuptools 47, this is no longer required, because the scripts will use importlib.metadata from the standard library instead of pkg_resources from setuptools. The scripts now looks like this:

#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'fedpkg==1.40','console_scripts','fedpkg'
import re
import sys

# for compatibility with easy_install; see #2198
__requires__ = 'fedpkg==1.40'

try:
    from importlib.metadata import distribution
except ImportError:
    try:
        from importlib_metadata import distribution
    except ImportError:
        from pkg_resources import load_entry_point
...

Since the scripts are generated during build, the packages no longer need to Require setuptools on runtime. The Python dist RPM dependency generator will no longer generate the Requires on python3.Xdist(setuptools).

The change is prepared in a pull request (https://src.fedoraproject.org/rpms/python-rpm-generators/pull-request/38). We plan to merge it as soon as it is ready. It only changes the behavior for Python 3.10 or newer, hence it will take affect in the Python 3.10 copr only, until Python is updated to 3.10 in Fedora proper.

Packages that actually import from the setuptools or pkg_resources module on runtime should explicitly require setuptools, but some of the upstreams may not do that. We don't expect many packages negatively affected by this, but since this is not strictly backwards compatible, we are only planning to do it in Fedora 35+.

Relevant Pull Requests:

Feedback

There was some discussion on various mailing list threads, but no other alternatives were proposed.

Benefit to Fedora

From Fedora Packaging Guidelines: It is important that your package list all necessary build dependencies using the BuildRequires: tag. As our testing showed, this is not true for many packages. Some of them can successfully build either with or without setuptools (they use try-except import and fallback to distutils from the standard library). Such packages are especially dangerous when not BuildRequiring setuptools -- they can produce different results depending on the presence of setuptools: either an .egg-info metadata directory (w/setuptools) or .egg-info text file (w/distutils). RPM has troubles when upgrading directories to files.

More Python applications will be installable without setutpools, hence contributing to the general minimization objective.

Scope

Some packages use distutils as a fallback in case setuptools is unavailable. The problem is that setuptools generates egg.info directory and distutils creates egg.info file. This would lead to failures caused by the known issue of RPM that cannot change directory to file. To avoid this we plan to do a mass spec update before we upgrade Python to version 3.10. This way we ensure that path will be correct after the mass rebuild.

  • Proposal owners:
  1. Test affected packages in Copr.
  2. Do the mass spec update to introduce missing BuildRequire to them.
  • Other developers: If your package requires setuptools on runtime, make sure it specifies it explicitly. There should be no additional work for developers other than that. There is a possibility that some packages will slip through our grep based query and fail to build, but we'll detect this in our testing Copr and fix them.
  • Release engineering: not needed for this Change
  • Policies and guidelines: already exist
  • Trademark approval: not needed for this Change
  • Alignment with Objectives: minimization

Upgrade/compatibility impact

All affected packages needs rebuild after the mass spec update but since we plan to do it with the upgrade to Python 3.10 we will rebuild it altogether. Users should not be affected by upgrades in any way.

How To Test

Interested testers can push explicit BuildRequire on setuptools to the distgit and see in our Python 3.10 COPR whether the package starts to build again. Webhook should trigger rebuild automatically on new commit in rawhide branch.

User Experience

Regular distro users shouldn't notice any change in system behaviour, except for packages that use setuptools or pkg_resources on runtime and fail to specify that dependency. Such packages will fail to run with ModuleNotFoundErrors and should be fixed by their maintainers.

Dependencies

300+ packages will need to BuildRequire setuptools explicitly. This will be done by the mass spec update.

Contingency Plan

  • Contingency mechanism: Commit removing python3-devel Requires on (python3-setuptools if rpmbuild) will be reverted. The change in Python dist RPM dependency generator will be reverted (Each of the contingency mechanisms can be applied independently.)
  • Contingency deadline: mass rebuild
  • Blocks release? No

Documentation

This page is a documentation, this is not a user facing change.

Release Notes