Changes/PythonSafePath

Python: Add -P to default shebangs

Summary

The -P flag will be added to the Python shebang macros (%{py3_shbang_opts}, %{py3_shebang_flags}, ...). Packages that adhere to those macros will change their Python shebangs from #! /usr/bin/python3 -s to #! /usr/bin/python3 -sP and as a result, will no longer have the directory of the script (such as /usr/bin) in sys.path. An opt-out mechanism exists.

Owner

• Name: Miro Hrončok, Karolina Surma, Victor Stinner
• Email: python-maint@redhat.com

Current status

• Targeted release: Fedora Linux 37
• Last updated: 2022-07-20
• devel thread
• FESCo issue: #2795
• Tracker bug: #2090866
• Release notes tracker: #842

Detailed Description

All Python 3 shebang RPM macros will be changed to contain one more flag: -P. Previously, they contained -s, now they will contain -sP.

From the documentation for the -P option:

Don’t prepend a potentially unsafe path to sys.path:

• python -m module command line: Don’t prepend the current working directory.
• python script.py command line: Don’t prepend the script’s directory. If it’s a symbolic link, resolve symbolic links.
• python -c code and python (REPL) command lines: Don’t prepend an empty string, which means the current working directory.

In shebangs, only the middle option (don’t prepend the script’s directory) is relevant.

Consider the following executbale script installed as /usr/bin/let-there-be-fun:

#! /usr/bin/python3 -s
import abc
...

When the script is directly executed (e.g. by running let-there-be-fun from the console), the script's directory (/usr/bin) is prepended to sys.path. Python tries to locate an importable abc module in /usr/bin first. This can cause real issues: python3-notebook: ImportError: bad magic number in six and bad magic number in six.

When the shebang includes -P:

#! /usr/bin/python3 -sP
import abc
...

...the script's directory (/usr/bin) is not prepended to sys.path. The change owners consider this approach safer for the majority of Fedora's RPM packages.

By default, all standardly RPM-packaged Python packages with scripts in /usr/bin will gain the -P flag in their shebang, assuming the software is packaged in a way that respects the Python shebang RPM macros (see below for opt-out and explicit opt-in mechanisms). Due to the variety of ways such scripts can be created/packaged, there will likely be packages that will not be affected by the change automatically. (In other words, the change is applied on RPM macro level, no added mechanics to force the flag, such as BRP scripts, are planned as part of this change.)

List of RPM macros that will gain -P

• %{py3_shbang_opts}
• %{py3_shbang_opts_nodash}
• %{py3_shebang_flags}
• %{py_shbang_opts}
• %{py_shbang_opts_nodash}
• %{py_shebang_flags}

List of source RPM packages that will gain -P in some of the built package(s)

Queried on 2022-07-07 in https://copr.fedorainfracloud.org/coprs/g/python/python-safe-path/

• GConf2
• ansible-core
• ansible-lint
• antlr4-project
• argparse-manpage
• asciinema
• astrometry
• asv
• auditwheel
• autodownloader
• autorandr
• bandit
• blender
• bodhi-client
• bodhi-server
• borgbackup
• borgmatic
• btrbk
• certbot
• cffconvert
• cfn-lint
• clang
• compiler-rt
• container-workflow-tool
• dblatex
• doge
• emacs-jedi
• ephemeral-port-reserve
• esptool
• fedfind
• fedora-easy-karma
• fedora-review
• fio
• gajim
• gi-docgen
• git-pull-request
• glances
• global
• gnofract4d
• go2rpm
• gtkpod
• hatch
• httpie
• ilua
• ini2toml
• inksmoto
• jack-mixer
• jello
• jrnl
• libsvm
• linode-cli
• llvm
• marshalparser
• mediawiki
• menulibre
• mkdocs
• monsterz
• mu
• mycli
• mygnuhealth
• netbox
• nicotine+
• nml
• noggin
• nox
• ntpsec
• nx-libs
• obserware
• ocrmypdf
• octave-miscellaneous
• offlineimap
• oraculum
• partio
• pcsc-lite
• pg_activity
• pipx
• playitagainsam
• poetry
• ptpython
• pynag
• pyp2spec
• pyplane
• pytest
• python-Automat
• python-PyLEMS
• python-TestSlide
• python-ZEO
• python-ZODB
• python-ajsonrpc
• python-ansi2html
• python-autopep8
• python-b4
• python-black
• python-bluepy
• python-bluepyopt
• python-blurb
• python-boutdata
• python-build
• python-cartopy
• python-cffsubr
• python-chardet
• python-charset-normalizer
• python-cheetah
• python-compreffor
• python-confluent-kafka
• python-cookiecutter
• python-cs
• python-cssutils
• python-cysignals
• python-daphne
• python-datadog
• python-datalad
• python-diff-cover
• python-dipy
• python-distro
• python-django
• python-dns-lexicon
• python-docutils
• python-doxytag2zealdb
• python-ephyviewer
• python-esphomeflasher
• python-exabgp
• python-ezdxf
• python-fabric
• python-fastavro
• python-fiona
• python-flake8
• python-flit
• python-fsleyes
• python-fslpy
• python-geotiler
• python-ghp-import
• python-git-changelog
• python-glymur
• python-gpxpy
• python-hatchling
• python-hdfs
• python-hdmf
• python-igor
• python-ipyparallel
• python-jenkins-job-builder
• python-jira
• python-josepy
• python-jsondiff
• python-jsonschema
• python-jupyter-client
• python-jupyter-console
• python-jupyter-core
• python-kafka
• python-keyring
• python-markdown-it-py
• python-molecule
• python-molmass
• python-myrepos-utils
• python-myst-parser
• python-name-that-hash
• python-nb2plots
• python-nbformat
• python-neurom
• python-niaarm
• python-nibabel
• python-nitrate
• python-nixio
• python-nose2
• python-notebook
• python-nudatus
• python-os-testr
• python-patatt
• python-pdfminer
• python-pint
• python-pipdeptree
• python-port-for
• python-pox
• python-ppft
• python-pudb
• python-pulp
• python-pybtex
• python-pycec
• python-pyct
• python-pygments
• python-pykwalify
• python-pypng
• python-pyside2
• python-pytest-bdd
• python-rasterio
• python-relatorio
• python-rq
• python-samloader
• python-sciunit
• python-scss
• python-shortuuid
• python-shtab
• python-sphinx
• python-sphinx-autobuild
• python-sphinx-theme-builder
• python-spyking-circus
• python-sshtunnel
• python-stem
• python-streamlink
• python-sure
• python-system-calls
• python-tldextract
• python-tmuxp
• python-toml-adapt
• python-towncrier
• python-tox
• python-tqdm
• python-twine
• python-twisted
• python-userpath
• python-versioningit
• python-virtualenv-clone
• python-watchgod
• python-websocket-client
• python-wloc
• python-xmlschema
• python-xnat
• python-zbase32
• python-zope-testrunner
• pythran
• rabbitvcs
• rdopkg
• roca-detect
• rofimoji
• salt-lint
• scribus-generator
• shybrid
• src
• sssd
• stomppy
• syslog-ng
• terminator
• thefuck
• thonny
• tldr
• tuna
• tuptime
• urlscan
• usd
• vips
• visidata
• wavextract
• weasyprint
• yt-dlp
• yubikey-manager

Components known to need opt-out

• llvm (files in /usr/share/opt-viewer are executable Python scripts with shebangs that import each other)

Opting out

If the new behavior is not desirable to your package, you can undefine the %_py3_shebang_P macro, like this:

# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%undefine _py3_shebang_P

Alternatively, you can amend the shebang flag macro (e.g. with sed) to remove the P flag.

If you use the current Python packaging guidelines, e.g. %pyproject_wheel and %pyproject_install, use:

# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%global py3_shebang_flags %(echo %py3_shebang_flags | sed s/P//)

If you use the 201x-era Python packaging guidelines, e.g. %py3_build and %py3_install, use:

# Don't add -P to Python shebang
# This package only works when /usr/bin is in sys.path (use your own rationale here)
%global py3_shbang_opts %(echo %py3_shbang_opts | sed s/P//)

(The only difference is the name of the macro.)

Opting in

If you use the current Python packaging guidelines, e.g. %pyproject_wheel and %pyproject_install, the standard set of Python shebang flags is applied to all files with Python shebangs installed in /usr/bin/.

If you use the 201x-era Python packaging guidelines, e.g. %py3_build and %py3_install, the standard set of Python shebang flags might be applied to some files and not applied to others depending on the exact structure of the packaged software.

If you wish to explicitly apply the standard set of Python shebang flags on a certain file that is not handled automatically, use the %py3_shebang_fix macro.

What if the packager changes %__python3 to an older version of Python

The -P flag was introduced in Python 3.11. When %__python3 is redefined to an older Python version, e.g. /usr/bin/python3.10, including the -P flag in shebangs would break the scripts. Hence, the flag will be included conditionally, presumably somehow like this:

%py3_shbang_opts -s%(%{__python3} -Ic "import sys; print('P' if hasattr(sys.flags, 'safe_path') else '')")

What if the admin/user changes /usr/bin/python3 to an older version of Python

The -P flag was introduced in Python 3.11. When an admin/user changes /usr/bin/python3 to point to an older version of Python, e.g. /usr/bin/python3.10, including the -P flag in shebangs would break the scripts.

However, changing /usr/bin/python3 to a different Python would brick a Fedora system even now. So we don't consider that an issue. See an example that changes /usr/bin/python3 to Python 3.9 (don't try this at home):

[root@086a2804411a /]# head -n1 /usr/bin/dnf
#!/usr/bin/python3

[root@086a2804411a /]# dnf --version
4.12.0
...

[root@086a2804411a /]# sudo ln -sf /usr/bin/python3.9 /usr/bin/python3

[root@086a2804411a /]# dnf --version
Traceback (most recent call last):
  File "/usr/bin/dnf", line 61, in <module>
    from dnf.cli import main
ModuleNotFoundError: No module named 'dnf'

Risks

As with any other change, there is a risk that this will break things. The change owners plan to test the change extensively via Copr before they deploy the change in Rawhide. If things go badly, they are prepared to delay or cancel the change.

The 201x-era Python RPM macros set the shebang flags by a weird hack. As a result, it is not well-defined what scripts will be affected by this change. The change owners are aware that:

1. not all Python scripts in /usr/bin will have the -P flag automatically
2. some scripts not in /usr/bin might gain the -P flag as well

The first point is an acceptable gradual deployment of the default flag. The second point is not very dangerous because we don't except users to directly execute Python scripts via shebangs when such scripts are not in $PATH. If a problematic package is found, it can opt-out easily. If this causes too much friction, we will only change the flag used in the %pyproject_install macro, leaving packages with the "legacy" macros intact.

Feedback

• Relevant upstream discussion: Should console_scripts entry points exclude the scripts directory from sys.path?
• Adding -P to Python: GitHub issue, email thread
• dnf maintainers response for doing this explicitly in dnf first

Generally, people seem to think that this is a good thing. They are afraid to change the default behavior of Python but welcome using this flag for system-installed scripts.

Feeedback from the Fedora's devel list

People asked why -P-like behavior isn't the default for Python instead. While the change owners agree that that would indeed be a saner default, Python has just now added the -P option. Changing the default behavior is not easy. That approach might be considered too disruptive to users. This proposal at least gets the security benefits for all system shipped stuff, without breaking anything the user has been using from non-packaged locations. (Adapted from a reply by Daniel P. Berrangé.)

Benefit to Fedora

Python programs in /usr/bin will be less fragile to other random files being present in /usr/bin. Real hard-to-debug issues like python3-notebook: ImportError: bad magic number in six and bad magic number in six will not happen.

Scope

• Proposal owners:
  • Test everything in Copr
  • If everything works, change the flags either before the Python 3.11 rebuild or before the Fedora 37 Mass Rebuild
  • Provide guidance to packagers, fix bugs if needed

• Other developers:
  • Observe their packages, find and report bugs, opt-out if needed
  • Volunteerily opt-in by calling the %py3_shebang_fix macro and/or by converting their packages to %pyproject_install

• Release engineering: #10784

• Policies and guidelines: The new flag needs to be documented in the Python packaging guidelines: https://pagure.io/packaging-committee/pull-request/1191

• Trademark approval: not needed for this Change

• Alignment with Objectives: no

Upgrade/compatibility impact

No impact is anticipated.

How To Test

• Low level: Examine the value of the changed RPM macros, it should contain the -P flag
• Middle level: Examine the shebang lines of RPM-installed Python scripts in /usr/bin, it should contain the -P flag
• High level: Tests that RPM-installed Python scripts still behave as expected but don't try to import stuff from /usr/bin

User Experience

Users of RPM-installed scripts should get a safer experience by default. Users of Python should not observe a difference, the behavior is not a new default: the -P flag needs to be explicitly used.

Dependencies

We need Changes/Python3.11 first.

Contingency Plan

• Contingency mechanism: defer to F38; only add -P to shebangs in %pyproject_install to keep backward compatibility of the old macros; revert and rebuild
• Contingency deadline: 1 week before the beta freeze
• Blocks release? No

Documentation

• This page
• TBD Updated Python guidelines
• https://docs.python.org/3.11/using/cmdline.html#cmdoption-P
• https://docs.python.org/3.11/whatsnew/3.11.html#summary-release-highlights (Security improvements)

Release Notes

TBD