Changes/PythonPackagingGuidelines202x

Python Packaging Guidelines overhaul

Summary

The Python Packaging guidelines will be rewritten, with the major changes being PyPI parity and usage of upstream metadata.

A new set of macros, pyproject-rpm-macros, written in mind with the new guidelines and with upstream best practices, are documented in the new guidelines as practical examples.

The older (a.k.a. "201x-era") Python Packaging guidelines will remain in effect as an option (until retired by another Fedora Change).

Owner

• Name: Petr Viktorin, Python SIG
• Email: pviktori@redhat.com, python-devel@lists.fedoraproject.org

Current status

• Targeted release: Fedora Linux 35
• Last updated: 2021-06-30
• FESCo issue: #2625
• Tracker bug: #1977883
• Release notes tracker: <will not be assigned by the Wrangler: not user-facing>

Detailed Description

The proposed new Python packaging guidelines are available at this Pull request: https://pagure.io/packaging-committee/pull-request/1072

A rendered preview (which might be deleted by the time you read this) is at: http://100.26.217.43/packaging-committee-pr1072/packaging-guidelines/Python/

Not removing older guidelines

The current ("201x-era") guidelines will stay in effect as an option for packages that haven't migrated yet or those that cannot follow the new guidelines for whatever unforeseen reason.

They will be retired in another Fedora Change, some time after the vast majority of Python packages follow the new guidelines and there are no known blockers for the remaining ones. There's no rush; it might well take a decade.

Guideline Changes

See an external document for a detailed list of changes to MUST/SHOULD rules. The major ones are listed here:

Requiring python3-devel

All packages that use Python at run- or build-time will need to BuildRequire python3-devel. This package brings the necessary macros and settings, and it will enable automated or manual checks. (For example, Python maintainers use this requirement to list packages that use Python in some way and might be affected by planned changes.)

PyPI Parity

Machine-readable metadata (distribution names in dist-info directories on disk and the corresponding python3.Xdist(foo) RPM provides) will match the Python Package Index (PyPI).

This solves a namespace issue. Python packaging tools use a flat namespace, and PyPI is the place where open-source Python packages are generally published, so users and tools assume a package called requests is whatever requests means on PyPI. While this is not ideal (especially for private packages), it makes sense for Fedora to align with the PyPI namespace.

Note that Fedora package names are not affected – just the Python packaging metadata on disk and virtual RPM Provides generated from it.

The new guidelines cover what to do for packages that cannot be registered on PyPI. The Change owner is prepared to help with PyPI registration.

Note that names found in Fedora but not on PyPI have been reserved on PyPI to avoid being taken by unrelated projects.

Upstream metadata

Upstream “dist-info” metadata for Python packages is now standardized and shareable with other distributors (be it Linux distros or others). As much as possible, RPM metadata should be automatically taken from there, rather be duplicated in the spec file (and likely diverge over time). This includes run-time and build-time requirements for Python packages, tests and test requirements for packages that use the tox tool, and Extras (optional features).

Packagers are expected to treat metadata bugs as any other bugs. (That is, ideally: patch them and present the patches upstream).

Naming

The new guidelines explain the various names a Python package can have (Fedora package name, project name, import name) and begs developers/maintainers to keep them synchronized if possible.

Python 3

The new guidelines only cover Python 3.

Python 2 does not need its own guidelines. python2.7 is deprecated and as of this writing, only about 35 packages use it, usually under individual FESCo exceptions.

Python 4 is not planned upstream.

Tests, linters and coverage

Running upstream tests or at least smoke testing the packaged Python modules is mandatory. Running linters (e.g. pylint, pyflakes, python-black) or test coverage measurements (e.g. python-coverage) is discouraged.

Non-Guideline Changes

The python3-devel package, a mandatory build-time requirement for all packages that use Python, will require pyproject-rpm-macros, helpers designed for the new guidelines.

While this small package is not always necessary, it will make life easier for most Python packagers.

pyproject-rpm-macros is currently documented as provisional; that label will be removed and a 1.0 release will be made.

Example spec file with changes

Example of a spec file following the old guidelines:

Name:           python-pello
Version:        1.0.2
Release:        1%{?dist}
Summary:        Example Python library

License:        MIT
URL:            https://github.com/fedora-python/Pello
Source0:        %{url}/archive/v%{version}/Pello-%{version}.tar.gz

BuildArch:      noarch
BuildRequires:  python3-devel

# Build dependencies needed to be specified manually
BuildRequires:  python3-setuptools

# Test dependencies needed to be specified manually
BuildRequires:  python3-pytest >= 3

# Runtime dependencies needed to be specified manually on build time to run tests
BuildRequires:  python3-blessings

%global _description %{expand:
A python module which provides a convenient example.
This description provides some details.}

%description %_description

%package -n python3-pello
Summary:        %{summary}
Recommends:     python3-pello+color

%description -n python3-pello %_description


# The metadata directory needed to be specified manually for Python extras
%python_extras_subpkg -n python3-pello color -i %{python3_sitelib}/*.egg-info


%prep
%autosetup -p1 -n Pello-%{version}


%build
# The macro only supported projects with setup.py
%py3_build


%install
# The macro only supported projects with setup.py
%py3_install


%check
# The old guidelines mentioned one way of running tests which is deprecated
# and in many cases lead to unexpected results (such as: 0 tests run, success)
# Alternatively, %%pytest can be used with the old macros as well
%{python3} setup.py test


%files -n python3-pello
%doc README.md
%license LICENSE.txt
%{_bindir}/pello_greeting

# The library files needed to be listed manually,
# often with extra care about the .pyc cache (although not in this example)
%{python3_sitelib}/pello/

# The metadata files needed to be listed manually
%{python3_sitelib}/Pello-*.egg-info/

Example of a spec file following the new guidelines:

Name:           python-pello
Version:        1.0.2
Release:        1%{?dist}
Summary:        Example Python library

License:        MIT
URL:            https://github.com/fedora-python/Pello
Source0:        %{url}/archive/v%{version}/Pello-%{version}.tar.gz

BuildArch:      noarch
BuildRequires:  python3-devel

%global _description %{expand:
A python module which provides a convenient example.
This description provides some details.}

%description %_description

%package -n python3-pello
Summary:        %{summary}
Recommends:     python3-pello+color

%description -n python3-pello %_description


%pyproject_extras_subpkg -n python3-pello color


%prep
%autosetup -p1 -n Pello-%{version}


%generate_buildrequires
# The build/test/runtime BuildRequires are generated from upstream metadata
%pyproject_buildrequires -t


%build
# The macro supports setup.py-based and pyproject.toml-based build
%pyproject_wheel


%install
# The macro supports setup.py-based and pyproject.toml-based build
%pyproject_install

# Library and metadata files can be saved automatically
%pyproject_save_files pello


%check
# test can run via the upstream de-facto standard way
# tox based tests run in Fedora's environment, not a Python virtual environment
# for projects without tox, %%pytest is preferred
%tox


# For python3-pello, %%{pyproject_files} handles code files, but
# executables, documentation and license must be listed in the spec file:
%files -n python3-pello -f %{pyproject_files}
%doc README.md
%license LICENSE.txt
%{_bindir}/pello_greeting

Feedback

The new guidelines were discussed extensively on the Python SIG list discussion thread.

Two major objections remain. Both were best voiced by Neal Gompa:

Burdening packagers

Neal does “not necessarily disagree that PyPI and Fedora pythonXdist() names should match, but [he disagrees] on burdening packagers with this.”

However, actually pushing a package to PyPI (and maintaining it there) is not necessary. The minimal-effort solution is to reserve the name on PyPI so that a conflicting package can not be created there.

The author of this change is willing to do this work for all Python packages whose upstream is not on PyPI; packagers need to send a request to the Python SIG list. If something happens to me, Red Hat's python-maint team is also ready to do this, and in the event of catastrophic management change, PyPI maintainers can be e-mailed directly.

This is all described in the new guidelines (see the PyPI parity section).

The new guidelines discourage the minimal-effort solution because it is suboptimal for users (and for the wider Python ecosystem).

(It was further suggested that the registration process should be automatic to minimize burden on packagers. It will be automated if it's too much for humans to handle, but we expect there won't be enough requests to make automation worth it.)

Conditioning names on PyPI

Neal also fundamentally disagrees “on conditioning those names on PyPI, because frankly no other ecosystem in Fedora works that way. Not Rust, not Go, not Perl, not OCaml, not anything.”

It's true that no other ecosystem in Fedora does this, but the author of this change believes that they should: if the ecosystem uses a flat namespace of dependencies, its Fedora packages should use the ecosystem-wide namespace rather than a Fedora-specific one. RPM predates global per-ecosystem directories of free/open software, but they are here and users (at least those affected by these issues) are used to them.

For Python specifically, PyPI is already a de-facto standard: if you see a name in install_requires in setup.py of some random upstream project you want to package, you can assume that it's the PyPI name.

It is in the best interest of Fedora's users to use PyPI names for dependency resolution.

Mandatory tests

When this change was proposed, the guidelines contained a rule that said running upstream tests in %check or Fedora CI is a MUST. Based on the strong push back against that rule, it was relaxed to a SHOULD. However at least smoke testing (e.g. by importing the Python modules) is still mandatory. Mentions of the Fedora CI were removed from the guidelines.

Benefit to Fedora

The new packaging guidelines should encourage sharing packaging work with upstream developers (and through them, with other distros). Hopefully, they'll also free up some packagers' time.

PyPI parity will hopefully enable automated ways of mixing Fedora packages with upstream ones.

The updated examples and notes will be more useful to today's packagers.

The older (a.k.a. "201x-era") Python Packaging guidelines only cover projects that use setup.py-based installation (via setuptools or distutils). The new guidelines and macros support more "build backends" (such as setup.py-less setuptools, poetry, flit or others) transparently via PEP 517 (and they continue to work with projects that use setup.py to install).

Scope

• Policies and guidelines: https://pagure.io/packaging-committee/pull-request/1072

• Proposal owners:
  • Make python3-devel require pyproject-rpm-macros
  • Remove provisional status from pyproject-rpm-macros; release & package version 1.0.
  • Handle PyPI name registration/reservation requests from other packagers.

• Other developers: Maintainers of packages that use Python should gradually switch to the new guidelines

• Release engineering: https://pagure.io/releng/issue/10164

• Trademark approval: N/A (not needed for this Change)

• Alignment with Objectives: N/A

Upgrade/compatibility impact

None expected. Any compatibility impact on individual packages would be a bug.

How To Test

If you maintain a package that uses Python, try to follow the new guidelines. Any major omission or ambiguity is a bug. (Minor/unique cases should be discussed on the Pytohn SIG list instead; the document is already quite long.)

Until this change proposal is approved and enabled, you'll need to BuildRequire pyproject-rpm-macros to have the new macros available.

Updated packages should be tested normally, as with any other package update.

User Experience

No direct impact on end users.

Dependencies

PyPI maintainers are aware of the general direction the Guidelines, and have been supportive.

Contingency Plan

• Contingency mechanism: Revert to the older guidelines
• Contingency deadline: N/A (Any time until old guidelines are retired, in a separate Change)
• Blocks release? No

Documentation

The new guidelines are the documentation. See Detailed Description above for links.

Release Notes

Fedora's Python Packaging guidelines have been rewritten to align with modern Python packaging practices and better integrate with the wider Python ecosystem (specifically, the Python Package Index).