From Fedora Project Wiki

Revision as of 12:34, 19 March 2018 by Churchyard (talk | contribs)


Work In Progress
Things described on this page are Work In Progress. Talk to the Python SIG if you want to discuss it.

Summary

Currently, in Fedora package names, executables etc., python without a version number generally means Python 2. Given that the upstream support for Python 2 will end soon, we want to prepare Fedora for a transition to Python 3 as a default. Before switching python to refer to Python 3, a lot of preparatory work needs to be done and this page explains our plan to achieve this.

Wait, wasn't Python 3 as Default already done in Fedora 23? Yes it was, but part of that change proposal was that /usr/bin/python and unversioned package names starting with python will still mean Python 2, in compliance with upstream PEP 394. This change is about what happens once PEP 394 is updated (and we will drive that update if needed):

  • /usr/bin/python will mean Python 3
  • dnf install python will mean Python 3
  • dnf install python-foo will mean Python 3 version of foo
  • pip, virtualenv, flask, pytest, flake8, sphinx-build and others like so will all run the Python 3 version, the Python 2 version will need to be run using the -2 (or 2) suffix

Owner

Current status

  • Last updated: 2018-03-19

Currently (in Fedora 27):

  • /usr/bin/python means Python 2
  • dnf install python means Python 2
  • dnf install python-foo means Python 2 version of foo
  • pip, virtualenv, flask, pycodestyle, flake8, sphinx-build and others like so (almost) all run the Python 2 version, the Python 3 version needs to be run using the -3 (or 3) suffix

/usr/bin/python

/usr/bin/python is a symbolic link to /usr/bin/python2 and belongs to the python2 package. This also means that packages that require /usr/bin/python (usually due to shebangs) de facto require Python 2.

The python package

Since Fedora 26, there is no python package, only python3 and python2. python is virtually provided from python2.

Packages with Python modules

According to the guidelines, built (binary) RPM packages with python in their names MUST use the python2- or python3- prefix. python- prefix (or -python suffix) is forbidden. The name with the python- prefix shall be provided from the python2- package using the %python_provide macro. This macro shall be used in both Python 2 and 3 subpackages, while it is currently a no-op on Python 3, but allows an easy switch in the future.

However, it wasn't always so and plenty of packages do not follow those guidelines yet.

The current situation in Fedora is as follows ("correct" means "follows current packaging guidelines"):

  • Correct package supporting Python 2 and 3 builds these binary RPMs:
    • python2-foo (provides python-foo as a virtual provide via %python_provides)
    • python3-foo (uses %python_provides, which is currently no-op)
  • "Misnamed" package supporting py2/3 builds these binary RPMs:
    • python-foo (the py2 version; may provide python2-foo virtually, but usually doesn't)
    • python3-foo
  • Correct py2-only packages are named:
    • python2-foo (provides python-foo as a virtual provide via %python_provides)
  • Correct py3-only packages are named:
    • python3-foo (uses %python_provides, which is currently no-op)
  • "Misnamed" py2-only package might be named:
    • python-foo, or
    • pyfoo, or
    • foo-python, etc.
  • There are (close to) no "misnamed" py3-only packages

We are tracking statistics about this in the Fedora Python 3 Porting Database. There is close to a thousand packages that violate the naming guidelines by the time of Fedora 26.

Detailed Description

While the default version of Python in Fedora is Python 3, the python command still invokes a Python 2 interpreter, and whenever one installs a python-foo package, they get a Python 2 version of this package. Similarly, when a Python developer wants to run a familiar tool, such as virtualenv, twine, flake8, pytest and others, the Python 2 version is run. This has all been decided in long discussions around the Python 3 as Default Change, where the main argument was PEP 394 that says: for the time being, all distributions should ensure that python refers to [...] python2.

However, Python 2 is slowly approaching its end of existence and will no longer be supported upstream after 2020-01-01. Therefore we would like to start the transition to a state where python means python3. PEP 394 says: there will eventually come a time where the third party ecosystem surrounding Python 3 is sufficiently mature for this recommendation to be updated to suggest that the python symlink refer to python3 rather than python2. We want to be ready for this update, and furthermore we want to make this update happen as soon as possible, but not later than 2020-01-01.

This requires a lot of preparatory work before we do the actual switch. We need to make sure that nothing uses python to imply Python 2 neither in binary RPM names nor in dependencies' names or shebangs, but rather explicitly specifies a Python version by using either python2 or python3. The state we want to achieve in Fedora to be ready for the switch, is where python-foo package names are only virtually provided with the %python_provide macro, so we can switch the macro to mean no-op on Python 2 and create the virtual provide on Python 3 packages.

Other Linux Distributions and OSes

Some Linux distributions like Arch Linux (2010) or OpenMandriva (2017) have already made /usr/bin/python pointing to Python 3, while others like openSUSE or Debian/Ubuntu are working towards handling the transition in a more conservative way. Gentoo uses a configurable option with Python 3 as default.

Open source package manager homebrew for macOS X has renamed the python3 package to just python on 2018-03-01.

Transition Steps

Phase 1: Eliminate the python- prefix from the Python 2 package names and /usr/bin/python from shebangs

  • Targeted release for completion: Fedora 28 ~= first half of 2018
  • Fedora Change: Not needed
  • Description: Ensure that all python-foo names are virtual provides declarations, with the actual RPMs themselves being named python2-foo. python-foo names are also not to be used to declare the dependencies of a Python 2 package. Ensure that no scripts we ship in Fedora use the /usr/bin/python shebang and no packages require /usr/bin/python. All scripts shall explicitly use /usr/bin/python2 or /usr/bin/python3 and all packages shall explicitly require those as well. (This also applies to the /usr/bin/env python shebang.) This is a preparatory phase to clean up Fedora's Python packages and ensure consistency not only for new Python packages, but also for existing ones. This allows us to proceed with Phase 2.

Interlude: Update PEP 394 (not a Fedora Change)

  • Targeted date: Before Fedora 32 branching
  • Formal proposal: TBD (informal at Python's Linux SIG)
  • Description: Work with CPython upstream to change the PEP 394 so that it says: all distributions should ensure that python refers to [...] python3.

Phase 2: Switch python to refer to python3

  • Targeted release: Fedora 32 ~= first half of 2020
  • Fedora Change: TBD
  • Description: Make python mean Python 3 in Fedora. This would mean that installing a python-foo package will imply a Python 3 version on the package, and invoking python or e.g. pytest will run a Python 3 version of the interpreter/tool. The steps needed to achieve this will be done in combination with proposing changes to the upstream guidance in PEP 394. Note that after Phase 1, nothing in Fedora uses /usr/bin/python any more, so it should be safe for sysadmins to flip the symblink back to python2 and we should provide a documented and supported way of doing so.
  • Steps:
    • Switch %python_provides macro to provide python-foo from python3-foo instead of python2-foo
    • Switch /usr/bin/python to point to /usr/bin/python3

Phase 3: Maybe get rid of Python 2

  • Targeted release: Fedora 33
  • Description: Python 2 is no longer supported at this point and is a dead upstream. A discussion shall be made whether to remove it from Fedora entirely and when. The current maintainers of python2 will likely orphan it. However, we realize plenty of stuff in Fedora would still require it at this point, so we expect Python SIG to continue maintaining it for some time. Also, packaging a project like tauthon might be worth considering.

Benefit to Fedora

Fedora has long been a one of the leaders of the Make Python 3 the Default Python initiative, having Python 3 as the "default Python" since Fedora 23 including a Fedora Workstation without Python 2, tracking the status of Python 3 compatibility and actively getting involved in porting upstream projects, such as (but not limited to) Samba. However, there are other distros (such as Arch) where python means Python 3 for quite some time.

This transition plan simplifies the final transition to Python 3 when the upstream support for Python 2 ends.

How can I help?

Go trough the packages you maintain and see if they FAIL some taskotron checks. If so, please fix your packages to follow the new rules. You can also see a list of packages breaking the naming policy on PortingDB.

Scope

  • Proposal owners:

Proposal owners are to communicate the idea to the packagers and rename the affected packages as well as fix the requirements via Mass_package_changes procedure.

  • Other developers:

Package maintainers are expected to fix package requirements generated from shebangs by making sure that no scripts in their packages use /usr/bin/python or /usr/bin/env python shebangs. Package maintainers will also be asked to review Pagure PRs with automated patches fixing the ambiguous Python 2 dependency declarations.

  • Policies and guidelines:
    • Using python2/python3 instead of python in binary RPM package name guidelines, fpc
    • Using python2/python3 instead of python in (Build)Requires, guidelines, fpc
    • Using /usr/bin/python2 instead of /usr/bin/python in shebangs and Requires, guidelines, fpc

How To Test

We have implemented taskotron checks for Phase 1 changes. The checks are being run each time a Python package is build in Koji, and you may view the results in taskotron resultsdb or when you submit the update in Bodhi.

Dependencies

Each phase depends on the phase before it.

Documentation

Q&A

  • I don't care which Python version my package runs on. What do I do? I just want to say python and let the build system to decide what's that.

Unfortunately that is not possible. According to the Python packaging guidelines, if the software is Python 3 compatible, it must be packaged for Python 3. Thus, you should update your package to Python 3 immediately. If you need more instructions, a guide for porting Python-based RPMs is available.

  • I want to use the same specfile for both Fedora and EPEL. By forcing me to use python2- prefix on Fedora, you are making my life harder, because I cannot use it in EPEL. What do I do?

This can be done with the following code: 

%if 0%{?rhel}
%global py2_prefix python
%else
%global py2_prefix python2
%endif

...

BuildRequires: %{py2_prefix}-requests

If this would make your specfile unbearably ugly, please contact us at the python-devel mailing list. We can add a %py2_prefix macro to the build root for you, but we don't think it's necessary given common specfiles usually already have such if-else blocks. You can even do: 

BuildRequires:  python%{?fedora:2}-requests
Retrieved from "https://fedoraproject.org/w/index.php?title=FinalizingFedoraSwitchtoPython3&oldid=513314"