From Fedora Project Wiki


Python means Python 3

Summary

In package and command names, "Python" will mean "Python 3".

Users installing and running Python or Python packages without specifying a version will get Python 3.

Running python will run python3. Running pytest will run the Python 3 version of pytest, and similarly for pydoc, pylint, etc.

dnf install python will install python3, and similarly for other python-* provides, e.g. dnf install python-requests will install python3-requests.

This is the final preparation for the retirement of Python 2 in Fedora 32 done in line with the recent upstream recommendations in PEP 394.

Owner

Current status

Detailed Description

Motivation

The final upstream release of Python 2 is planned for January 2020. No further fixes will be made upstream. Most of Fedora 31's lifetime is after that date. Python 2 will be maintained only by its Fedora maintainers.

In preparation for removing Python 2 from Fedora entirely, we will make the unqualified name "Python" refer to the fully supported version.

This is in line with upstream changes to PEP 394 recommendations.

This completes the work started with Fedora 23's "Python 3 as Default" change, with only removing Python 2 left to do.

What is being changed

The following changes will be made both in the Python packaging guidelines and the actual packages.

Package provides

Before: Packages with Python 2 modules used to provide the unversioned python- name. Users could do dnf install python-requests and the python2-requests package would be installed.

After: Packages with Python 3 modules will provide the unversioned python- name. Users can do dnf install python-requests and the python3-requests package will be installed.

This applies to the python3/python2 package as well as packages with Python modules. dnf install python will install python3.

The vast majority of packages will be updated by changing the %python-provide macro, with no changes in the individual spec files. The change owners will change the macro before the mass rebuild.

Python command

Before: The /usr/bin/python command was a symbolic link to /usr/bin/python2 living in the python-unversioned-command subpackage of python2. python2 recommended python-unversioned-command, so most users got the command by default when python2 was installed.

After: The /usr/bin/python command will be a symbolic link to /usr/bin/python3 living in the python-unversioned-command subpackage of python3. python3 will recommend python-unversioned-command, so most users will get the command by default when python3 is installed (and it is installed by default).


Other commands

Similarly, the Python-specific commands will switch from Python 2 to Python 3. This includes at least the following commands:

  • pip
  • python-config
  • wheel
  • idle
  • pydoc
  • pytest
  • nosetest
  • pycodestyle
  • pylint
  • epylint
  • pyreverse
  • symilar
  • unit2
  • msgfmt.py
  • pygettext.py
  • flask
  • ipython
  • f2py
  • ipdb
  • easy_install
  • ...

This list is incomplete, there is a script to discover all such commands.

Note that applications like pygmentize or cython, whose behavior doesn't depend on the Python version, are not affected. (By current guidelines, they should be already using Python 3 if possible.)

Changes needed by Python package maintainers

This section of the change covers action needed from Python package maintainers. Most of the packages need no change, but there are several exceptions.


Packages with ambiguous names

Tracked at: https://fedora.portingdb.xyz/namingpolicy/

If your package with a Python 2 module or plugin is named with unversioned python (such as for example claws-mail-plugins-python or PyQt4), it needs to be removed or renamed to have python2 in the name (such as, for example, claws-mail-plugins-python2 or python2-PyQt4).

If your package is an application that happens to be written in Python 2 (such as calibre), no renaming is needed (applications don't need the python-, python2- or python3- prefix).

Packages with ambiguous requires

Tracked at: https://fedora.portingdb.xyz/namingpolicy/

If your package depends on (Requires, BuildRequires, Recommends, etc.) a Python package with unversioned Python name (such as for example: systemd-python, python-setuptools, PyQt5, pycairo), it will now get resolved to the Python 3 version. Such dependencies need to be updated to specific Python 2 or Python 3 names (such as for example: pythonX-systemd, pythonX-setuptools, pythonX-qt5, pythonX-cairo where X is either 2 or 3).


Packages with ambiguous provides

Most of the backwards compatibility python-* provides are handled by the %python_provide macro and the change owners will change the macro behavior, so you don't have to worry about those. However, sometimes manual provides were added. If your package has some manual backwards compatibility provides for Python 2 packages, those need to be moved to the Python 3 packages or removed.

For example, before this change, the python2-m2crypto package provided and obsoleted m2crypto. Instead, the python3-m2crypto package should do that after this change.


Packages with missing %python_provide

The %python_provide macro used to do nothing for Python 3 packages. As such, it was often forgotten and only used with the Python 2 packages. Packagers of Python 3 packages should make sure to use the %python_provide macro according to the guidelines: 

%package -n python3-%{srcname}
...
%{?python_provide:%python_provide python3-%{srcname}}

Packages with Python versioned commands and tools

Some packages have two versions of the provided commands. For example, the python2-pytest package has /usr/bin/pytest-2 and /usr/bin/pytest-2.7, the python3-pytest package has /usr/bin/pytest-3 and /usr/bin/pytest-3.X. The unversioned /usr/bin/pytest command used to be a symbolic link to /usr/bin/pytest-2. Now it needs to be changed to /usr/bin/pytest-3 and moved to the python3-pytest package.

For some packages, such duplication makes no sense, because the user sees no difference. For example, there should be just one /usr/bin/pygmentize -- the user doesn't care if it runs on Python 3, Python 2 or if it is written in Rust. This is not a new rule, but if your package is not following it, now is a good time to make sure the tool uses Python 3.


Packages that need unversioned Python to be Python 2

Since Fedora 29, a specific workaround is needed to use the python command as Python 2. This workaround will now bring Python 3. If that does not work for your package, you'll need to fix it (patch it) or retire it from Fedora.

If you only need the python command to mean Python 2 during package build, you can do something like this: 

mkdir tmp_path
ln -s %{__python2} tmp_path/python
PATH=$(pwd)/tmp_path:$PATH make ...

However, if your package uses the python command during runtime, this ugly workaround won't work.


Alternative proposals

Based on the initial discussion on the devel mailing list, here are some alternative proposals and the reasons why the change owners decided to not follow them.

Alternative proposal: python means Python 2 forever

One obvious way to handle things is to never do this Change at all and keep things as they are. This has an obvious advantage: It is ultimately backwards compatible. However, we disagree with this approach for several reasons. Namely:

"python" meaning "Python 2" unfortunately brings in false expectations, such as that Python 2 is "the default Python", "the system Python", "the preferred Python" or "better Python". Python 2 will lose upstream support 2 months after the Fedora 31 release and we don't want to encourage such expectations. Fedora users need to be prepared that Python 2 is the old, legacy and mostly unsupported option.

Given the removal of most of the Python 2 packages from Fedora 32, this would also mean that if the user tries to dnf install python-foo, most of the time there would be no match.

In the Python world, there are two groups of users: the ones that expect "python" to mean Python 3 and the ones that expect it to be Python 2. Unfortunately the groups don't talk to each other a lot, so the "conservatives" might think that nobody expects "python" to mean Python 3. However, most of the Python ecosystem now does expect Python 3, be it the data analysts, scientific community, users of Microsoft Windows or users of Linux distributions such as Arch, Gentoo or OpenMandriva. Even the situation on macOS is changing.

Alternative proposal: No more unversioned python

Another way of handling things is to simply not have any /usr/bin/python at all. This has an obvious advantage: nobody will accidentally run Python 2 code with Python 3. It also has an obvious disadvantage: A lot of things that would have otherwise worked on Python 3 would just not work at all.

It also brings a lot of questions. Will there be also no pip command, etc.? What will happen if user tries to dnf install python, should it fail?

It is completely fine to drop the python command from highly controlled environments, such as the Fedora build system. However we don't consider it OK to take it away from the users.

Note that in RHEL 8, there is no unversioned "python" command by default: it has to be explicitly set. However, that decision was made with the long life expectancy of RHEL in mind and to make the system usable both for the more conservative customers that expect Python 2 as default, as well as a growing list of customers that expect Python 3 as default. We consider Fedora much more "leading edge" than RHEL in this manner.

It was suggested that we need to to remove /usr/bin/python to see what breaks. This is why we don't agree:

  • From the Fedora perspective: We've already almost eliminated (1, 2) the usage of /usr/bin/python from our packages, except packages that use it as a buildtime dependency when "any python would do" (e.g. gcc) and except packages that FTBFS since Fedora 29.
  • From the perspective of third party and user scripts: Users can still make #!/usr/bin/env python work (using virtualenvs or symlinks), so they avoid the breakage regardless of what /usr/bin/python points to or if it is present. (See the User Experience section of this proposal.) We would get unhappy users, but not much feedback on what breaks.
  • From the perspective of pip-installed packages: In our experience, the pip-installed packages generally handle shebangs well (e.g. they set the shebang to the interpreter that runs pip). Of course, there might be some packages where this is not true, but the enormous popularity of Python virtual environments means it's been battle tested for years.

Alternative proposal: Do this together with the Python 2 retirement in F32

One more idea is to delay this change until Fedora 32, where most of the Python 2 stack is removed. We've decided to do this one release sooner to send a stronger message to the users (and contributors) that they need to prepare for Python 2 packages being removed soon.

While "rebranding" everything to Python 3 might be controversial, it doesn't remove any functionality and it gives the users one more release to fully adapt to a world without Python 2 packages.


Benefit to Fedora

The name "Python" will not refer to software that will be unmaintained upstream for most of Fedora 31's lifetime and retired from Fedora 32.

Scope

  • Proposal owners:
    • Changes in python3, python2 packages:
    • Changes in other commands:
      • switch pytest, nosetests, ipython, pip... to python3 (see Detailed Description for the actual list)
    • Changes in python-rpm-macros:
      • make %python_provide provide python-foo for python3-foo instead of for python2-foo
    • Let the mass rebuild change the provides based on the %python_provide change
    • Examine the failures and provide help to package maintainers
    • File bugs for remaining packages that provide python-x or x-python for Python 2


  • Other developers (Python package maintainers, see Detailed Description):
    • Everybody: Make sure that you require and use python2 or python3 packages explicitly
    • If your package provides python-* or *-python for a Python 2 package by other means than the %python_provide macro, move that to the Python 3 package (or remove entirely if not applicable)
    • Fix or retire your package if it requires the unversioned python command to be Python 2
  • Release engineering: #8482
  • Trademark approval: not needed

Upgrade/compatibility impact

Custom scripts with python shebangs will invoke Python 3 by default, whereas previosuly they invoked Python 2 by default. See the User Experience section for details.

How To Test

For your favorite Python 3 package, check that it can be installed with the ambiguous Python name. For example, check that dnf install python-pip installs pyhon3-pip instead of pyhon2-pip.

For your favorite Python command, check that it invokes the Python 3 version. For example, check that python runs Python 3.

User Experience

Users who run python directly will get Python 3. Users who run pip directly will get pip for Python 3. Users who run pytest directly will get Python 3 pytest for Python 3. ...

Scripts with ambiguous Python shebangs (#!/usr/bin/python or #!/usr/bin/env python) will be executed by Python 3 by default.

If users need the python command or the #!/usr/bin/env python shebang to run Python 2, they can easily do that by: 

ln -s /usr/bin/python2 ~/.local/bin/python

Similarly, sysadmins can do that system-wide: 

ln -s /usr/bin/python2 /usr/local/bin/python


If users don't want the python command at all, they can dnf remove python-unversioned-command.


Dependencies

Most packages that provide Python version-specific functionality will be affected. However, the Change owners include proven packagers and they maintain python-rpm-macros, so most will not need packager attention.

We depend on the Fedora mass rebuild to adjust macro-generated package provides.

There are currently (2019-07-03) 62 packages left with ambiguous Python requires. Those need to be adapted. Lot of them unfortunately Fail to Build From Source and will be retired before the Fedora 31 branching.

Contingency Plan

  • Contingency mechanism: revert the changes, mass rebuild packages with the original %python_provide macro
  • Contingency deadline: beta freeze
  • Blocks release? No
  • Blocks product? No

Documentation

  • Updated Packaging guidelines
  • This Change page

Release Notes

TBD. Check User Experience section.

Retrieved from "https://fedoraproject.org/w/index.php?title=Changes/Python_means_Python3&oldid=548106"