User:Cstratak/python3 static link against libpython

Build Python 3 to statically link with libpython3.8.a for better performance

Summary

Python 3 traditionally in Fedora was built with a shared library libpython3.?.so and the final binary was dynamically linked against that shared library. This change is about creating the static library and linking the final python3 binary against it, as it provides significant performance improvement, up to 15% depending on the workload. The static library will not be shipped. The shared library will continue to exist in a separate subpackage. In essence, python3 will no longer depend on libpython.

Owner

• Name: Charalampos Stratakis, Miro Hrončok, Victor Stinner
• Email: python-maint@redhat.com

Current status

• Targeted release: Fedora 32
• Last updated: 2019-10-24
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

When we compile the python3 package on Fedora (prior to this change), we create the libpython3.?.so shared library and the final python3 binary (/usr/bin/python3) is dynamically linked against it. However by building the libpython3.?.a static library and statically linking the final binary against it, we can achieve a performance gain of approximately 15% depending on the workload. Link time optimizations and profile guided optimizations also have a greater impact when python3 is linked statically.

Since Python 3.8, C extensions are no longer linked to libpython by default (for example, they need to utilize the --embed flag for python3-config to do so). During the Python 3.8 upgrade and rebuilds we've uncovered various cases of packages linking to libpython implicitly through various hacks within their buildsystems and fixed as many as possible. However, there are legitimate reasons to link to libpython and for those cases libpython should be provided so applications that embed Python can continue to do so.

This mirrors the Debian/Ubuntu way of building python, where they offer a statically linked binary and an additional libpython subpackage. The libpython subpackage will be created and python3-devel will depend on it, so packages that embed Python will keep working.

By applying this change, libpython's namespace will be separated from python's, so python C extension which are linked to libpython might experience side effects or break.

C extensions that are not linked to libpython and are loaded in Python or an application embedding Python, as well as C extensions linked to libpython in an application embedding Python will not be affected

Currently there is no upstream option to build the static library, as well as the shared one and statically link the final binary to it, so we have to rely on a downstream patch to achieve it. We plan to work with upstream to incorporate the changes there as well.

Benefit to Fedora

Python's performance will increase significantly depending on the workload. Since many core components of the OS also depend on python this could lead to an increase in their performance as well.

Benchmarks to be added here

Scope

• Proposal owners:
  • Review and merge the pull request with the implementation.
  • Go through the python C extension packages that are linked to libpython and test if things work correctly. Will provide a copr repository for testing.

• Other developers: Other developers are encouraged to test the new statically linked python3 to see if their package works as expected

• Release engineering: #Releng issue number Rebuild of all the affected packages will be required

• Policies and guidelines: The packaging guidelines will need to be updated to explicitly mention that C extensions should not be linked to libpython, and that the python3 binary is statically linked.

• Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

Affected package maintainers should verify that their packages work as expected and the only impact the end users should see is a performance increase for workloads relying on Python.

How To Test

The following script can be used to verify that the change is in effect:

import ctypes
import sys
 
EMPTY_TUPLE_SINGLETON = ()

def get_empty_tuple(lib):
    # Call PyTuple_New(0)
    func = lib.PyTuple_New
    func.argtypes = (ctypes.c_ssize_t,)
    func.restype = ctypes.py_object
    return func(0)
 
def test_lib(libname, lib):
    obj = get_empty_tuple(lib)
    if obj is EMPTY_TUPLE_SINGLETON:
        print("%s: SAME namespace" % libname)
    else:
        print("%s: DIFFERENT namespace" % libname)

def test():
    program = ctypes.pythonapi

    if hasattr(sys, 'abiflags'):
        abiflags = sys.abiflags
    else:
        # Python 2
        abiflags = ''
    ver = sys.version_info
    filename = ('libpython%s.%s%s.so.1.0'
                % (ver.major, ver.minor, abiflags))
    libpython = ctypes.cdll.LoadLibrary(filename)

    test_lib('program', program)
    test_lib('libpython', libpython)

test()

Output before the change:

program: SAME namespace
libpython: SAME namespace

Output after the change:

program: SAME namespace
libpython: DIFFERENT namespace

User Experience

Python based workloads should see a performance gain of up to 15%.

Dependencies

While this specific change is not dependent on anything else, we would like to ensure that all the packages that link to libpython continue to work as expected.

Currently 113 packages on rawhide depend on libpython.

Result of the "repoquery --repo=rawhide --source --whatrequires 'libpython3.8.so.1.0()(64bit)' " command on Fedora Rawhide:

• COPASI-4.27.217-1.fc32.src.rpm
• Io-language-20151111-0.e64ff9.fc32.19.src.rpm
• OpenImageIO-2.0.11-1.fc32.src.rpm
• YafaRay-3.3.0-24.20190819git0a182c1.fc32.src.rpm
• antimony-0.9.3-16.fc32.src.rpm
• boost-1.69.0-11.fc32.src.rpm
• calamares-3.2.11-4.fc32.src.rpm
• calibre-4.1.0-1.fc32.src.rpm
• cantor-19.08.1-1.fc32.src.rpm
• ceph-14.2.4-1.fc32.src.rpm
• clingo-5.4.0-1.fc32.src.rpm
• condor-8.8.4-2.fc32.src.rpm
• createrepo_c-0.15.1-1.fc32.src.rpm
• csound-6.13.0-5.fc32.src.rpm
• cvc4-1.7-6.fc32.src.rpm
• dmlite-1.13.99-2.fc32.src.rpm
• domoticz-4.11352-0.git20191006.1.fc32.src.rpm
• fontforge-20190801-3.fc32.src.rpm
• freecad-0.18.3-5.fc32.1.src.rpm
• gdb-8.3.50.20190924-27.fc32.src.rpm
• gdcm-3.0.1-3.fc32.src.rpm
• gdl-0.9.9-12.20190915git2870075.fc32.src.rpm
• getdp-3.2.0-5.fc32.src.rpm
• glade-3.22.1-6.fc32.src.rpm
• globus-net-manager-1.4-1.fc32.src.rpm
• glom-1.30.4-16.fc32.src.rpm
• gnucash-3.7-1.fc32.src.rpm
• gpaw-19.8.1-3.fc32.src.rpm
• hamlib-3.3-8.fc32.src.rpm
• hokuyoaist-3.0.2-29.fc32.src.rpm
• hugin-2019.0.0-5.fc32.src.rpm
• insight-8.2.50.20190118-6.fc32.src.rpm
• kicad-5.1.4-5.fc32.src.rpm
• lammps-20190807-2.fc32.src.rpm
• ldns-1.7.0-28.fc32.src.rpm
• libCombine-0.2.3-0.5.20190327gitd7c11a9.fc32.src.rpm
• libarcus-4.1.0-4.fc32.src.rpm
• libarcus-lulzbot-3.6.18-2.fc32.src.rpm
• libbatch-2.4.2-1.fc32.src.rpm
• libcomps-0.1.11-5.fc32.src.rpm
• libdnf-0.35.5-2.fc32.src.rpm
• libftdi-1.4-1.fc32.src.rpm
• libkml-1.3.0-23.fc32.src.rpm
• libkolabxml-1.1.6-13.fc32.src.rpm
• libldb-2.0.7-1.fc32.src.rpm
• libnuml-1.1.1-19.20190327gite61f6d5.fc32.src.rpm
• libpeas-1.24.0-1.fc32.src.rpm
• libplist-2.0.0-15.fc32.src.rpm
• libreoffice-6.3.2.2-1.fc32.src.rpm
• librepo-1.10.6-1.fc32.src.rpm
• libsavitar-4.1.0-3.fc32.src.rpm
• libsbml-5.18.0-9.fc32.src.rpm
• libsedml-0.4.4-7.fc32.src.rpm
• libtalloc-2.3.0-1.fc32.src.rpm
• libyang-0.16.105-4.fc32.src.rpm
• libyui-bindings-1.1.2-19.fc32.src.rpm
• link-grammar-5.7.0-1.fc32.src.rpm
• lldb-9.0.0-1.fc32.src.rpm
• mathgl-2.4.4-1.fc32.src.rpm
• med-4.0.0-5.fc32.src.rpm
• mod_wsgi-4.6.6-4.fc32.src.rpm
• nautilus-python-1.2.3-3.fc32.src.rpm
• nbdkit-1.15.4-1.fc32.src.rpm
• nest-2.18.0-6.fc32.src.rpm
• netgen-mesher-6.2.1810-4.fc32.src.rpm
• neuron-7.7.1-11.fc32.src.rpm
• nextpnr-0-0.7.20190821gitc192ba2.fc32.src.rpm
• nordugrid-arc-6.2.0-1.fc32.src.rpm
• nwchem-6.8.2-1.fc32.src.rpm
• openbabel-2.4.1-26.fc32.src.rpm
• openmeeg-2.4.1-7.fc32.src.rpm
• openscap-1.3.1-4.fc32.src.rpm
• opentrep-0.07.1-5.fc32.src.rpm
• openvdb-6.2.0-1.fc32.src.rpm
• pam_wrapper-1.0.7-4.fc32.src.rpm
• paraview-5.6.0-11.fc32.src.rpm
• perl-Inline-Python-0.56-10.fc32.src.rpm
• pidgin-2.13.0-16.fc32.src.rpm
• pitivi-0.999-6.fc32.src.rpm
• plplot-5.15.0-2.fc32.src.rpm
• postgresql-11.5-5.fc32.src.rpm
• pynac-0.7.24-4.fc32.src.rpm
• pyotherside-1.5.8-5.fc32.src.rpm
• pythia8-8.2.43-4.fc32.src.rpm
• python-caja-1.23.0-3.fc32.src.rpm
• python-gstreamer1-1.16.1-1.fc32.src.rpm
• python-jep-3.9.0-2.fc32.src.rpm
• python-qt5-5.13.1-1.fc32.src.rpm
• python3-3.8.0~rc1-1.fc32.src.rpm
• qgis-3.8.2-6.fc32.src.rpm
• qpid-dispatch-1.9.0-2.fc32.src.rpm
• qpid-proton-0.29.0-1.fc32.src.rpm
• rdkit-2019.03.3-4.fc32.src.rpm
• renderdoc-1.4-4.fc32.src.rpm
• rmol-1.00.3-4.fc32.src.rpm
• root-6.18.04-1.fc32.src.rpm
• samba-4.11.0-3.fc32.src.rpm
• scidavis-1.25.1-2.fc32.src.rpm
• sigil-0.9.14-3.fc32.src.rpm
• swift-lang-5.1.1-0.1.20191004git4242edd.fc32.src.rpm
• texworks-0.6.3-3.fc32.src.rpm
• thunarx-python-0.5.1-24.fc32.src.rpm
• trademgen-1.00.4-4.fc32.src.rpm
• trellis-1.0-0.5.20190806git7e97b5b.fc32.src.rpm
• unbound-1.9.3-2.fc32.src.rpm
• uwsgi-2.0.18-3.fc32.src.rpm
• vdr-epg-daemon-1.1.146-7.fc32.src.rpm
• vigra-1.11.1-17.fc32.src.rpm
• vim-8.1.2120-1.fc32.src.rpm
• vrpn-07.33-19.fc32.src.rpm
• vtk-8.2.0-10.fc32.src.rpm
• weechat-2.4-4.fc32.src.rpm
• znc-1.7.5-1.fc32.src.rpm

Contingency Plan

• Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
• Contingency deadline: N/A (not a System Wide Change)
• Blocks release? N/A (not a System Wide Change), Yes/No
• Blocks product? product

Documentation

N/A (not a System Wide Change)