Python: Add -P to default shebangs

Summary

The -P flag will be added to the Python shebang macros (%{py3_shbang_opts}, %{py3_shebang_flags}, ...). Packages that adhere to those macros will change their Python shebangs from #! /usr/bin/python3 -s to #! /usr/bin/python3 -sP and as a result, will no longer have the directory of the script (such as /usr/bin) in sys.path. An opt-out mechanism exists.

Owner

Name: Miro Hrončok, Karolina Surma, Victor Stinner
Email: python-maint@redhat.com

Current status

Targeted release: Fedora Linux 37
Last updated: 2022-07-20
devel thread
FESCo issue: #2795
Tracker bug: #2090866
Release notes tracker: #842

Detailed Description

All Python 3 shebang RPM macros will be changed to contain one more flag: -P. Previously, they contained -s, now they will contain -sP.

From the documentation for the -P option:

Don’t prepend a potentially unsafe path to sys.path:

python -m module command line: Don’t prepend the current working directory.
python script.py command line: Don’t prepend the script’s directory. If it’s a symbolic link, resolve symbolic links.
python -c code and python (REPL) command lines: Don’t prepend an empty string, which means the current working directory.

In shebangs, only the middle option (don’t prepend the script’s directory) is relevant.

Consider the following executbale script installed as /usr/bin/let-there-be-fun:

#! /usr/bin/python3 -s
import abc
...

When the script is directly executed (e.g. by running let-there-be-fun from the console), the script's directory (/usr/bin) is prepended to sys.path. Python tries to locate an importable abc module in /usr/bin first. This can cause real issues: python3-notebook: ImportError: bad magic number in six and bad magic number in six.

When the shebang includes -P:

#! /usr/bin/python3 -sP
import abc
...

...the script's directory (/usr/bin) is not prepended to sys.path. The change owners consider this approach safer for the majority of Fedora's RPM packages.

By default, all standardly RPM-packaged Python packages with scripts in /usr/bin will gain the -P flag in their shebang, assuming the software is packaged in a way that respects the Python shebang RPM macros (see below for opt-out and explicit opt-in mechanisms). Due to the variety of ways such scripts can be created/packaged, there will likely be packages that will not be affected by the change automatically. (In other words, the change is applied on RPM macro level, no added mechanics to force the flag, such as BRP scripts, are planned as part of this change.)

List of RPM macros that will gain `-P`

%{py3_shbang_opts}
%{py3_shbang_opts_nodash}
%{py3_shebang_flags}
%{py_shbang_opts}
%{py_shbang_opts_nodash}
%{py_shebang_flags}

List of source RPM packages that will gain `-P` in some of the built package(s)

Queried on 2022-07-07 in https://copr.fedorainfracloud.org/coprs/g/python/python-safe-path/

GConf2
ansible-core
ansible-lint
antlr4-project
argparse-manpage
asciinema
astrometry
asv
auditwheel
autodownloader
autorandr
bandit
blender
bodhi-client
bodhi-server
borgbackup
borgmatic
btrbk
certbot
cffconvert
cfn-lint
clang
compiler-rt
container-workflow-tool
dblatex
doge
emacs-jedi
ephemeral-port-reserve
esptool
fedfind
fedora-easy-karma
fedora-review
fio
gajim
gi-docgen
git-pull-request
glances
global
gnofract4d
go2rpm
gtkpod
hatch
httpie
ilua
ini2toml
inksmoto
jack-mixer
jello
jrnl
libsvm
linode-cli
llvm
marshalparser
mediawiki
menulibre
mkdocs
monsterz
mu
mycli
mygnuhealth
netbox
nicotine+
nml
noggin
nox
ntpsec
nx-libs
obserware
ocrmypdf
octave-miscellaneous
offlineimap
oraculum
partio
pcsc-lite
pg_activity
pipx
playitagainsam
poetry
ptpython
pynag
pyp2spec
pyplane
pytest
python-Automat
python-PyLEMS
python-TestSlide
python-ZEO
python-ZODB
python-ajsonrpc
python-ansi2html
python-autopep8
python-b4
python-black
python-bluepy
python-bluepyopt
python-blurb
python-boutdata
python-build
python-cartopy
python-cffsubr
python-chardet
python-charset-normalizer
python-cheetah
python-compreffor
python-confluent-kafka
python-cookiecutter
python-cs
python-cssutils
python-cysignals
python-daphne
python-datadog
python-datalad
python-diff-cover
python-dipy
python-distro
python-django
python-dns-lexicon
python-docutils
python-doxytag2zealdb
python-ephyviewer
python-esphomeflasher
python-exabgp
python-ezdxf
python-fabric
python-fastavro
python-fiona
python-flake8
python-flit
python-fsleyes
python-fslpy
python-geotiler
python-ghp-import
python-git-changelog
python-glymur
python-gpxpy
python-hatchling
python-hdfs
python-hdmf
python-igor
python-ipyparallel
python-jenkins-job-builder
python-jira
python-josepy
python-jsondiff
python-jsonschema
python-jupyter-client
python-jupyter-console
python-jupyter-core
python-kafka
python-keyring
python-markdown-it-py
python-molecule
python-molmass
python-myrepos-utils
python-myst-parser
python-name-that-hash
python-nb2plots
python-nbformat
python-neurom
python-niaarm
python-nibabel
python-nitrate
python-nixio
python-nose2
python-notebook
python-nudatus
python-os-testr
python-patatt
python-pdfminer
python-pint
python-pipdeptree
python-port-for
python-pox
python-ppft
python-pudb
python-pulp
python-pybtex
python-pycec
python-pyct
python-pygments
python-pykwalify
python-pypng
python-pyside2
python-pytest-bdd
python-rasterio
python-relatorio
python-rq
python-samloader
python-sciunit
python-scss
python-shortuuid
python-shtab
python-sphinx
python-sphinx-autobuild
python-sphinx-theme-builder
python-spyking-circus
python-sshtunnel
python-stem
python-streamlink
python-sure
python-system-calls
python-tldextract
python-tmuxp
python-toml-adapt
python-towncrier
python-tox
python-tqdm
python-twine
python-twisted
python-userpath
python-versioningit
python-virtualenv-clone
python-watchgod
python-websocket-client
python-wloc
python-xmlschema
python-xnat
python-zbase32
python-zope-testrunner
pythran
rabbitvcs
rdopkg
roca-detect
rofimoji
salt-lint
scribus-generator
shybrid
src
sssd
stomppy
syslog-ng
terminator
thefuck
thonny
tldr
tuna
tuptime
urlscan
usd
vips
visidata
wavextract
weasyprint
yt-dlp
yubikey-manager

Components known to need opt-out

llvm (files in /usr/share/opt-viewer are executable Python scripts with shebangs that import each other)

Opting out

If the new behavior is not desirable to your package, you can undefine the %_py3_shebang_P macro, like this:

# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%undefine _py3_shebang_P

Alternatively, you can amend the shebang flag macro (e.g. with sed) to remove the P flag.

If you use the current Python packaging guidelines, e.g. %pyproject_wheel and %pyproject_install, use:

# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%global py3_shebang_flags %(echo %py3_shebang_flags | sed s/P//)

If you use the 201x-era Python packaging guidelines, e.g. %py3_build and %py3_install, use:

# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%global py3_shbang_opts %(echo %py3_shbang_opts | sed s/P//)

(The only difference is the name of the macro.)

Opting in

If you use the current Python packaging guidelines, e.g. %pyproject_wheel and %pyproject_install, the standard set of Python shebang flags is applied to all files with Python shebangs installed in /usr/bin/.

If you use the 201x-era Python packaging guidelines, e.g. %py3_build and %py3_install, the standard set of Python shebang flags might be applied to some files and not applied to others depending on the exact structure of the packaged software.

If you wish to explicitly apply the standard set of Python shebang flags on a certain file that is not handled automatically, use the %py3_shebang_fix macro.

What if the packager changes `%__python3` to an older version of Python

The -P flag was introduced in Python 3.11. When %__python3 is redefined to an older Python version, e.g. /usr/bin/python3.10, including the -P flag in shebangs would break the scripts. Hence, the flag will be included conditionally, presumably somehow like this:

%py3_shbang_opts -s%(%{__python3} -Ic "import sys; print('P' if hasattr(sys.flags, 'safe_path') else '')")

What if the admin/user changes `/usr/bin/python3` to an older version of Python

The -P flag was introduced in Python 3.11. When an admin/user changes /usr/bin/python3 to point to an older version of Python, e.g. /usr/bin/python3.10, including the -P flag in shebangs would break the scripts.

However, changing /usr/bin/python3 to a different Python would brick a Fedora system even now. So we don't consider that an issue. See an example that changes /usr/bin/python3 to Python 3.9 (don't try this at home):

[root@086a2804411a /]# head -n1 /usr/bin/dnf
#!/usr/bin/python3

[root@086a2804411a /]# dnf --version
4.12.0
...

[root@086a2804411a /]# sudo ln -sf /usr/bin/python3.9 /usr/bin/python3

[root@086a2804411a /]# dnf --version
Traceback (most recent call last):
  File "/usr/bin/dnf", line 61, in <module>
    from dnf.cli import main
ModuleNotFoundError: No module named 'dnf'

Risks

As with any other change, there is a risk that this will break things. The change owners plan to test the change extensively via Copr before they deploy the change in Rawhide. If things go badly, they are prepared to delay or cancel the change.

The 201x-era Python RPM macros set the shebang flags by a weird hack. As a result, it is not well-defined what scripts will be affected by this change. The change owners are aware that:

not all Python scripts in /usr/bin will have the -P flag automatically
some scripts not in /usr/bin might gain the -P flag as well

The first point is an acceptable gradual deployment of the default flag. The second point is not very dangerous because we don't except users to directly execute Python scripts via shebangs when such scripts are not in $PATH. If a problematic package is found, it can opt-out easily. If this causes too much friction, we will only change the flag used in the %pyproject_install macro, leaving packages with the "legacy" macros intact.

Feedback

Relevant upstream discussion: Should console_scripts entry points exclude the scripts directory from sys.path?
Adding -P to Python: GitHub issue, email thread
dnf maintainers response for doing this explicitly in dnf first

Generally, people seem to think that this is a good thing. They are afraid to change the default behavior of Python but welcome using this flag for system-installed scripts.

Feeedback from the Fedora's devel list

People asked why -P-like behavior isn't the default for Python instead. While the change owners agree that that would indeed be a saner default, Python has just now added the -P option. Changing the default behavior is not easy. That approach might be considered too disruptive to users. This proposal at least gets the security benefits for all system shipped stuff, without breaking anything the user has been using from non-packaged locations. (Adapted from a reply by Daniel P. Berrangé.)

Benefit to Fedora

Python programs in /usr/bin will be less fragile to other random files being present in /usr/bin. Real hard-to-debug issues like python3-notebook: ImportError: bad magic number in six and bad magic number in six will not happen.

Scope

Proposal owners:
- Test everything in Copr
- If everything works, change the flags either before the Python 3.11 rebuild or before the Fedora 37 Mass Rebuild
- Provide guidance to packagers, fix bugs if needed

Other developers:
- Observe their packages, find and report bugs, opt-out if needed
- Volunteerily opt-in by calling the %py3_shebang_fix macro and/or by converting their packages to %pyproject_install

Release engineering: #10784

Policies and guidelines: The new flag needs to be documented in the Python packaging guidelines: https://pagure.io/packaging-committee/pull-request/1191

Trademark approval: not needed for this Change

Alignment with Objectives: no

Upgrade/compatibility impact

No impact is anticipated.

How To Test

Low level: Examine the value of the changed RPM macros, it should contain the -P flag
Middle level: Examine the shebang lines of RPM-installed Python scripts in /usr/bin, it should contain the -P flag
High level: Tests that RPM-installed Python scripts still behave as expected but don't try to import stuff from /usr/bin

User Experience

Users of RPM-installed scripts should get a safer experience by default. Users of Python should not observe a difference, the behavior is not a new default: the -P flag needs to be explicitly used.

Dependencies

We need Changes/Python3.11 first.

Contingency Plan

Contingency mechanism: defer to F38; only add -P to shebangs in %pyproject_install to keep backward compatibility of the old macros; revert and rebuild
Contingency deadline: 1 week before the beta freeze
Blocks release? No

Documentation

This page
TBD Updated Python guidelines
https://docs.python.org/3.11/using/cmdline.html#cmdoption-P
https://docs.python.org/3.11/whatsnew/3.11.html#summary-release-highlights (Security improvements)

Release Notes

TBD

Search

Changes/PythonSafePath

Contents

Python: Add -P to default shebangs

Summary

Owner

Current status

Detailed Description

List of RPM macros that will gain `-P`

List of source RPM packages that will gain `-P` in some of the built package(s)

Components known to need opt-out

Opting out

Opting in

What if the packager changes `%__python3` to an older version of Python

What if the admin/user changes `/usr/bin/python3` to an older version of Python

Risks

Feedback

Feeedback from the Fedora's devel list

Benefit to Fedora

Scope

Upgrade/compatibility impact

How To Test

User Experience

Dependencies

Contingency Plan

Documentation

Release Notes

Search

Changes/PythonSafePath

Python: Add -P to default shebangs

Summary

Owner

Current status

Detailed Description

List of RPM macros that will gain -P

List of source RPM packages that will gain -P in some of the built package(s)

Components known to need opt-out

Opting out

Opting in

What if the packager changes %__python3 to an older version of Python

What if the admin/user changes /usr/bin/python3 to an older version of Python

Risks

Feedback

Feeedback from the Fedora's devel list

Benefit to Fedora

Scope

Upgrade/compatibility impact

How To Test

User Experience

Dependencies

Contingency Plan

Documentation

Release Notes

List of RPM macros that will gain `-P`

List of source RPM packages that will gain `-P` in some of the built package(s)

What if the packager changes `%__python3` to an older version of Python

What if the admin/user changes `/usr/bin/python3` to an older version of Python