Build Python with -fno-semantic-interposition for better performance
This change was originally proposed for Fedora 32 as Changes/PythonStaticSpeedup, however based on community feedback, it has been significantly reduced.
Summary
We add the -fno-semantic-interposition compiler/linker flag when building Python interpreters, as it provides significant performance improvement, up to 27% depending on the workload. Users will no longer be able to use LD_PRELOAD to override a symbol from libpython, which we consider a good trade off for the speedup.
Owner
- Name: Charalampos Stratakis, Victor Stinner, Miro Hrončok
- Email: python-maint@redhat.com
- Shout-out: Jan Kratochvíl for first suggesting this instead of the original proposal, followed by Kevin Kofler
Current status
- Targeted release: Fedora 32
- Last updated: 2019-11-15
- Tracker bug: <will be assigned by the Wrangler>
- Release notes tracker: <will be assigned by the Wrangler>
Detailed Description
When we build the Python interpreter with the -fno-semantic-interposition compiler/linker flag, we can achieve a performance gain of 5% to 27% depending on the workload. Link time optimizations and profile guided optimizations also have a greater impact when python3 is built this way.
As a negative side effect, it disables the LD_PRELOAD feature: it's no longer possible to override symbols in libpython with LD_PRELOAD. Thanks to that, the compiler can avoid PLT indirection for function calls and can inline more function functions in libpython. We're describing function calls from libpython to libpython, something which is very common in Python: almost all function calls are calls from libpython to libpython.
If Fedora users need to use LD_PRELOAD to override symbols in libpython, the recommend way is to build a custom Python without -fno-semantic-interposition.
Affected Pythons
Primarily, we will change the interpreter in the 
python3
Impact on other Python packages (and generally software using Python) is not anticipated (other than the possible speedup).
We will also change the alternate Python interpreters where possible and useful, primarily the still upstream supported alternate versions of CPython, such as
Benefit to Fedora
Python's performance will increase significantly depending on the workload. Since many core components of the OS also depend on Python this could lead to an increase in their performance as well, however individual benchmarks will need to be conducted to verify the performance gain for those components.
pyperformance results, ignoring differences smaller than 5%:
+-------------------------+------------------+------------------------------+ | Benchmark | python38-3.8.0-1 | python38-3.8.0-666 | +=========================+==================+==============================+ | nbody | 238 ms | 174 ms: 1.36x faster (-27%) | +-------------------------+------------------+------------------------------+ | raytrace | 919 ms | 686 ms: 1.34x faster (-25%) | +-------------------------+------------------+------------------------------+ | scimark_lu | 285 ms | 215 ms: 1.33x faster (-25%) | +-------------------------+------------------+------------------------------+ | scimark_sparse_mat_mult | 8.20 ms | 6.20 ms: 1.32x faster (-24%) | +-------------------------+------------------+------------------------------+ | django_template | 204 ms | 156 ms: 1.31x faster (-24%) | +-------------------------+------------------+------------------------------+ | chaos | 203 ms | 156 ms: 1.30x faster (-23%) | +-------------------------+------------------+------------------------------+ | logging_simple | 15.6 us | 12.2 us: 1.28x faster (-22%) | +-------------------------+------------------+------------------------------+ | richards | 124 ms | 97.0 ms: 1.28x faster (-22%) | +-------------------------+------------------+------------------------------+ | scimark_fft | 652 ms | 511 ms: 1.27x faster (-22%) | +-------------------------+------------------+------------------------------+ | hexiom | 17.4 ms | 13.8 ms: 1.27x faster (-21%) | +-------------------------+------------------+------------------------------+ | logging_format | 17.1 us | 13.5 us: 1.27x faster (-21%) | +-------------------------+------------------+------------------------------+ | nqueens | 174 ms | 137 ms: 1.26x faster (-21%) | +-------------------------+------------------+------------------------------+ | crypto_pyaes | 201 ms | 160 ms: 1.26x faster (-20%) | +-------------------------+------------------+------------------------------+ | deltablue | 12.6 ms | 10.0 ms: 1.25x faster (-20%) | +-------------------------+------------------+------------------------------+ | unpickle_pure_python | 576 us | 463 us: 1.24x faster (-20%) | +-------------------------+------------------+------------------------------+ | pickle_pure_python | 799 us | 644 us: 1.24x faster (-19%) | +-------------------------+------------------+------------------------------+ | go | 449 ms | 362 ms: 1.24x faster (-19%) | +-------------------------+------------------+------------------------------+ | spectral_norm | 247 ms | 200 ms: 1.24x faster (-19%) | +-------------------------+------------------+------------------------------+ | scimark_monte_carlo | 185 ms | 151 ms: 1.23x faster (-19%) | +-------------------------+------------------+------------------------------+ | logging_silent | 340 ns | 276 ns: 1.23x faster (-19%) | +-------------------------+------------------+------------------------------+ | unpickle | 23.3 us | 19.1 us: 1.22x faster (-18%) | +-------------------------+------------------+------------------------------+ | float | 200 ms | 166 ms: 1.21x faster (-17%) | +-------------------------+------------------+------------------------------+ | mako | 26.6 ms | 22.0 ms: 1.21x faster (-17%) | +-------------------------+------------------+------------------------------+ | xml_etree_generate | 159 ms | 133 ms: 1.20x faster (-17%) | +-------------------------+------------------+------------------------------+ | xml_etree_process | 128 ms | 107 ms: 1.20x faster (-16%) | +-------------------------+------------------+------------------------------+ | fannkuch | 795 ms | 670 ms: 1.19x faster (-16%) | +-------------------------+------------------+------------------------------+ | chameleon | 15.7 ms | 13.3 ms: 1.18x faster (-15%) | +-------------------------+------------------+------------------------------+ | scimark_sor | 347 ms | 294 ms: 1.18x faster (-15%) | +-------------------------+------------------+------------------------------+ | pathlib | 35.7 ms | 30.2 ms: 1.18x faster (-15%) | +-------------------------+------------------+------------------------------+ | regex_compile | 301 ms | 255 ms: 1.18x faster (-15%) | +-------------------------+------------------+------------------------------+ | genshi_text | 48.3 ms | 41.2 ms: 1.17x faster (-15%) | +-------------------------+------------------+------------------------------+ | sympy_str | 459 ms | 394 ms: 1.17x faster (-14%) | +-------------------------+------------------+------------------------------+ | genshi_xml | 102 ms | 87.6 ms: 1.16x faster (-14%) | +-------------------------+------------------+------------------------------+ | 2to3 | 540 ms | 465 ms: 1.16x faster (-14%) | +-------------------------+------------------+------------------------------+ | sqlite_synth | 4.89 us | 4.25 us: 1.15x faster (-13%) | +-------------------------+------------------+------------------------------+ | sympy_expand | 704 ms | 613 ms: 1.15x faster (-13%) | +-------------------------+------------------+------------------------------+ | html5lib | 162 ms | 141 ms: 1.15x faster (-13%) | +-------------------------+------------------+------------------------------+ | sympy_integrate | 34.2 ms | 30.0 ms: 1.14x faster (-12%) | +-------------------------+------------------+------------------------------+ | dulwich_log | 121 ms | 107 ms: 1.13x faster (-11%) | +-------------------------+------------------+------------------------------+ | sympy_sum | 286 ms | 253 ms: 1.13x faster (-11%) | +-------------------------+------------------+------------------------------+ | xml_etree_iterparse | 170 ms | 152 ms: 1.12x faster (-11%) | +-------------------------+------------------+------------------------------+ | telco | 10.2 ms | 9.14 ms: 1.11x faster (-10%) | +-------------------------+------------------+------------------------------+ | meteor_contest | 171 ms | 154 ms: 1.11x faster (-10%) | +-------------------------+------------------+------------------------------+ | json_dumps | 20.0 ms | 18.0 ms: 1.11x faster (-10%) | +-------------------------+------------------+------------------------------+ | tornado_http | 425 ms | 384 ms: 1.11x faster (-10%) | +-------------------------+------------------+------------------------------+ | xml_etree_parse | 249 ms | 226 ms: 1.10x faster (-9%) | +-------------------------+------------------+------------------------------+ | sqlalchemy_imperative | 53.4 ms | 49.6 ms: 1.08x faster (-7%) | +-------------------------+------------------+------------------------------+ | python_startup | 13.7 ms | 12.7 ms: 1.07x faster (-7%) | +-------------------------+------------------+------------------------------+ | json_loads | 43.3 us | 40.7 us: 1.06x faster (-6%) | +-------------------------+------------------+------------------------------+ | python_startup_no_site | 9.29 ms | 8.75 ms: 1.06x faster (-6%) | +-------------------------+------------------+------------------------------+ | pickle_dict | 33.8 us | 32.0 us: 1.06x faster (-5%) | +-------------------------+------------------+------------------------------+ | sqlalchemy_declarative | 272 ms | 258 ms: 1.05x faster (-5%) | +-------------------------+------------------+------------------------------+
Scope
- Proposal owners:
- Review and merge the pull request with the implementation.
- Go through the Python C extension packages that are linked to libpython and test if things work correctly. A copr repository will be provided for testing.
- Other developers: Other developers are encouraged to test the new statically linked python3 and check if their package works as expected
- Release engineering: #8953 This change does not require a mass rebuild, however a rebuild of the affected packages will be required. The affected packages will be rebuilt in copr first.
- Policies and guidelines: The packaging guidelines will need to be updated to explicitly mention that C extensions should not be linked to libpython, and that the python3 binary is statically linked.
- Trademark approval: N/A (not needed for this Change)
Upgrade/compatibility impact
Affected package maintainers should verify that their packages work as expected and the only impact the end users should see is a performance increase for workloads relying on Python.
How To Test
Copr repo with instructions: https://copr.fedorainfracloud.org/coprs/g/python/Python3_statically_linked/
Package changes test
The change will bring the new libpython3 subpackage as a dependency of python3-devel.
Test that it's installed:
$ rpm -q libpython3
Test that it's uninstalled if python3-devel is removed:
$ dnf remove python3-devel
Test that python3-libs no longer includes the libpython shared library.
$ rpm -ql python3-libs | grep libpython3
Dynamic linker test
To check that the python3.8 program is not linked to libpython, ldd can be used. For example, Python 3.7 will still be linked to libpython:
$ ldd /usr/bin/python3.7|grep libpython libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0 (0x00007fbb57333000)
But python3.8 will no longer be linked to libpython:
$ ldd /usr/bin/python3.8|grep libpython
Performance test
The performance speedup can be measured using the official Python benchmark suite pyperformance: see Run benchmarks.
Namespace test
The following script can be used to verify that the change is in effect:
import ctypes
import sys
EMPTY_TUPLE_SINGLETON = ()
def get_empty_tuple(lib):
# Call PyTuple_New(0)
func = lib.PyTuple_New
func.argtypes = (ctypes.c_ssize_t,)
func.restype = ctypes.py_object
return func(0)
def test_lib(libname, lib):
obj = get_empty_tuple(lib)
if obj is EMPTY_TUPLE_SINGLETON:
print("%s: SAME namespace" % libname)
else:
print("%s: DIFFERENT namespace" % libname)
def test():
program = ctypes.pythonapi
if hasattr(sys, 'abiflags'):
abiflags = sys.abiflags
else:
# Python 2
abiflags = ''
ver = sys.version_info
filename = ('libpython%s.%s%s.so.1.0'
% (ver.major, ver.minor, abiflags))
libpython = ctypes.cdll.LoadLibrary(filename)
test_lib('program', program)
test_lib('libpython', libpython)
test()
Output before the change:
program: SAME namespace libpython: SAME namespace
Output after the change:
program: SAME namespace libpython: DIFFERENT namespace
User Experience
Python based workloads should see a performance gain of up to 27%.
Dependencies
While this specific change is not dependent on anything else, we would like to ensure that all the packages that link to libpython continue to work as expected.
Currently (30/10/2019) 118 packages on rawhide depend on libpython.
Result of the "repoquery --repo=rawhide --source --whatrequires 'libpython3.8.so.1.0()(64bit)' " command on Fedora Rawhide, x86_64:
- COPASI
- Io-language
- OpenImageIO
- YafaRay
- antimony
- blender
- boost
- calamares
- calibre
- cantor
- ceph
- clingo
- condor
- createrepo_c
- csound
- cvc4
- dionaea
- dmlite
- domoticz
- fontforge
- freecad
- gdb
- gdcm
- gdl
- getdp
- glade
- globus-net-manager
- glom
- gnucash
- gpaw
- hamlib
- hokuyoaist
- hugin
- insight
- kdevelop-python
- kicad
- kitty
- krita
- lammps
- ldns
- libCombine
- libarcus https://src.fedoraproject.org/rpms/libarcus/pull-request/8
- libarcus-lulzbot
- libbatch
- libcec
- libcomps
- libdnf
- libftdi
- libkml
- libkolabxml
- libldb
- libnuml
- libpeas
- libplist
- libreoffice
- librepo
- libsavitar
- libsbml
- libsedml
- libtalloc
- libyang
- libyui-bindings
- link-grammar
- lldb
- mathgl
- med
- mod_wsgi
- nautilus-python
- nbdkit
- nest
- netgen-mesher
- neuron
- nextpnr
- nordugrid-arc
- nwchem
- openbabel
- openscap
- opentrep
- openvdb
- pam_wrapper
- paraview
- perl-Inline-Python
- pidgin
- pitivi
- plplot
- postgresql
- pynac
- pyotherside
- pythia8
- python-gstreamer1
- python-jep
- python-qt5
python3- qgis
- qpid-dispatch
- qpid-proton
- rdkit
- renderdoc
- rmol
- root
- samba
- scidavis
- sigil
- swift-lang
- texworks
- thunarx-python
- trademgen
- trellis
- unbound
- uwsgi
- vdr-epg-daemon
- vigra
- vim
- vrpn
- vtk
- weechat
- znc
Packages in bold are the ones present in the default docker/podman "fedora:rawhide" image.
Contingency Plan
- Contingency mechanism: If issues appear that cannot be fixed in a timely manner the change can be easily reverted and will be considered again for the next fedora release. Also a proper upgrade path mechanism will be provided in case of reversion, since libpython.3.?.so will be a separate package with this change.
- Contingency deadline: Before the beta freeze of Fedora 32 (2020-02-25)
- Blocks release? Yes
- Blocks product? None
Documentation
The documentation will be reflected in the changes for the python packaging guidelines.