FinalizingFedoraSwitchtoPython3

Finalizing Fedora's Switch to Python 3

Summary

Currently, in Fedora package names, executables etc., python without a version number generally means Python 2. Given that the upstream support for Python 2 will end soon, we want to prepare Fedora for a transition to Python 3 as a default. Before switching python to refer to Python 3, a lot of preparatory work needs to be done and this page explains our plan to achieve this.

Wait, wasn't Python 3 as Default already done in Fedora 23? Yes it was, but part of that change proposal was that /usr/bin/python and unversioned package names starting with python will still mean Python 2, in compliance with upstream PEP 394. This change is about what happens once PEP 394 is updated (and we will drive that update if needed):

• /usr/bin/python will mean Python 3
• dnf install python will mean Python 3
• dnf install python-foo will mean Python 3 version of foo
• pip, virtualenv, flask, pycodestyle, flake8, sphinx-build and others like so will all run the Python 3 version, the Python 2 version will need to be run using the -2 (or 2) suffix

Owner

• Name: Iryna Shcherbina
• Name: Miro Hrončok
• Name: Petr Viktorin
• Email: python-maint at redhat.com

Current status

• Last updated: 2017-06-27

Currently (in Fedora 27):

• /usr/bin/python means Python 2
• dnf install python means Python 2
• dnf install python-foo means Python 2 version of foo
• pip, virtualenv, flask, pycodestyle, flake8, sphinx-build and others like so (almost) all run the Python 2 version, the Python 3 version needs to be run using the -3 (or 3) suffix

/usr/bin/python

/usr/bin/python is a symbolic link to /usr/bin/python2. This also means that packages that require /usr/bin/python (usually due to shebangs) de facto require Python 2.

The python package

Since Fedora 26, there is no python package, only python3 and python2. python is virtually provided from python2.

Packages with Python modules

According to the guidelines, built (binary) RPM packages with python in their names MUST use the python2- or python3- prefix. python- prefix (or -python suffix) is forbidden. The name with the python- prefix shall be provided from the python2- package using the %python_provide macro. This macro shall be used in both Python 2 and 3 subpackages, while it is currently a no-op on Python 3, but allows an easy switch in the future.

However, it wasn't always so and plenty of packages do not follow those guidelines yet.

The current situation in Fedora is as wollows ("correct" means "follows current packaging guidelines"):

• Correct package supporting Python 2 and 3 builds these binary RPMs:
  • python2-foo (provides python-foo as a virtual provide via %python_provides)
  • python3-foo (uses %python_provides, which is currently no-op)

• "Misnamed" package supporting py2/3 builds these binary RPMs:
  • python-foo (the py2 version; may provide python2-foo virtually, but usually doesn't)
  • python3-foo

• Correct py2-only packages are named:
  • python2-foo (provides python-foo as a virtual provide via %python_provides)

• Correct py3-only packages are named:
  • python3-foo (uses %python_provides, which is currently no-op)

• "Misnamed" py2-only package might be named:
  • python-foo, or
  • pyfoo, or
  • foo-python, etc.

• There are (close to) no "misnamed" py3-only packages

We are tracking statistics about this in the Fedora Python 3 Porting Database. There is close to a thousand packages that violate the naming guidelines by the time of Fedora 26.

Detailed Description

While the default version of Python in Fedora is Python 3, the python command still invokes a Python 2 interpreter, and whenever one installs a python-foo package, they get a Python 2 version of this package. Similarly, when a Python developer wants to run a familiar tool, such as virtualenv, twine, flake8, pytest and others, the Python 2 version is run. This has all been decided in long discussions around the Python 3 as Default Change, where the main argument was PEP 394 that says: for the time being, all distributions should ensure that python refers to [...] python2.

However, Python 2 is slowly approaching its end of existence and will no longer be supported upstream after 2020. Therefore we would like to start the transition to a state where python means python3. PEP 394 says: there will eventually come a time where the third party ecosystem surrounding Python 3 is sufficiently mature for this recommendation to be updated to suggest that the python symlink refer to python3 rather than python2. We want to be ready for this update, and furthermore we want to make this update happen as soon as possible, bot not later than 2020.

This requires a lot of preparatory work before we do the actual switch. We need to make sure that nothing uses python to imply Python 2 neither in binary RPM names nor in dependencies' names or shebangs, but rather explicitly specifies a Python version by using either python2 or python3. The state we want to achieve in Fedora to be ready for the switch, is where python-foo package names are only virtually provided with the %python_provide macro, so we can switch the macro to mean no-op on Python 2 and create the virtual provide on Python 3 packages.

Transition Steps

Phase 1: Eliminate the python- prefix from the Python 2 package names and /usr/bin/python from shebangs

• Targeted release for completion: Fedora 30
• Description: Ensure that all python-foo names are virtual provides declarations, with the actual RPMs themselves being named python2-foo. python-foo names are also not to be used to declare the dependencies of a Python 2 package. Ensure that no scripts we ship in Fedora use the /usr/bin/python shebang and no packages require /usr/bin/python. All scripts shall explicitly use /usr/bin/python2 or /usr/bin/python3 and all packages shall explicitly require those as well. This is a preparatory phase to clean up Fedora's Python packages and ensure consistency not only for new Python packages, but also for existing ones. This allows us to proceed with phase 2

• Steps:
  • Identify the packages with naming and shebangs issues. Those include packages with misnamed binary RPMs and dependencies.
  • Inform the package maintainers about the issues. Gather feedback and encourage fixing the packages.
  • Start mass bug filing slowly: open e.g. 20 Bugzillas and learn from the reaction.
• Start mass bug filing slowly: open e.g. 20 Bugzillas and learn from the reaction.

Phase 2: Switch python to refer to python3

• Targeted release: Fedora 32
• Description: Make python mean Python 3 in Fedora. This would mean that installing a python-foo package will imply a Python 3 version on the package, and invoking python or e.g. pytest will run a Python 3 version of the interpreter/tool. The steps needed to achieve this will be done in combination with proposing changes to the upstream guidance in PEP 394.

• Steps:
  • Switch %python_provides macro to provide python-foo from python3-foo instead of python2-foo
  • Switch /usr/bin/python to point to /usr/bin/python3

Phase 3: Maybe get rid of Python 2

• Targeted release: Fedora 33+
• Description: Python 2 is no longer supported at this point and is a dead upstream. A discussion shall be made whether to remove it from Fedora entirely and when. The current maintainers of python2 will likely orphan it. However, we realize plenty of stuff in Fedora would still require it at this point, so we expect Python SIG to continue maintaining it for some time.

Benefit to Fedora

Fedora has long been a one of the leaders of the Make Python 3 the Default Python initiative, having Python 3 as the "default Python" since Fedora 23 including a Fedora Workstation without Python 2, tracking the status of Python 3 compatibility and actively getting involved in porting upstream projects, such as (but not limited to) Samba. However, there are other distros (such as Arch) where python means Python 3 for quite some time.

This transition plan simplifies the final transition to Python 3 when the upstream support for Python 2 ends.

Scope

• Proposal owners:

Proposal owners have to communicate the idea to the packagers and provide all the detailed information. They may step in to fix the packages if absolutely needed.

• Other developers:

Package maintainers affected by the naming policies will be asked to fix their packages via mailing lists and Bugzilla.

• Policies and guidelines:
  • Using python2/python3 instead of python in binary RPM package name guidelines, fpc
  • Using python2/python3 instead of python in (Build)Requires guidelines, fpc
  • Using /usr/bin/python2 instead of /usr/bin/python in shebangs and Requires TBD

How To Test

We have implemented taskotron checks for Phase 1 changes. The checks are being run each time a Python package is build in Koji, and you may view the results in taskotron resultsdb or when you submit the update in Bodhi.

Dependencies

Only those Python 2 packages which use python- prefix in dependency declarations are expected to break. However, when the change is completed, no packages like this are expected to exist.

Documentation

Discussion on Fedora devel mailing list

PEP 394