User:Cstratak/python3 static link against libpython

Build Python 3 to statically link with libpython3.8.a for better performance

Summary

Python 3 traditionally in Fedora was built with a shared library libpython3.?.so and the final binary was dynimically linked against that shared library. This change is about creating the static library and linking the final python3 binary against it, as it provides significant performance improvement, up to 15% depending on the workload. The static library will not be shipped. The shared library will continue to exist in a separate subpackage. In essence, python3 will no longer depend on libpython.

Owner

• Name: Charalampos Stratakis, Miro Hrončok, Victor Stinner
• Email: python-maint@redhat.com

Current status

• Targeted release: Fedora 32
• Last updated: 2019-10-16
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

When we compile the python3 package on Fedora (prior to this change), we create the libpython3.?.so shared library and the final python3 binary (/usr/bin/python3) is dynamically linked against it. However by building the libpython3.?.a static library and statically linking the final binary against it, we can achieve a performance gain of approximately 15% depending on the workload. Link time optimizations and profile guided optimizations also have a greater impact when python3 is linked statically.

Since Python 3.8, C extensions are no longer linked to libpython by default (for example, they need to utilize the --embed flag for python3-config to do so). During the Python 3.8 upgrade and rebuilds we've uncovered various cases and fixed as many as possible. However, there are legitimate reasons to link to libpython and for those cases libpython should be provided so applications that embed Python can continue to do so.

This mirrors the Debian/Ubuntu way of building python, where they offer a statically linked binary and an additional libpython subpackage. The libpython subpackage will be created and python3-devel will depend on it, so packages that embed python will keep working.

Caveats: There is currently no upstream option to build the static library, as well as the shared one and statically link the final binary to it, so we have to rely on a downstream patch to achieve it.

Also libpython's namespace will be separated from python's, so packages that are python C extension and also embed python within their code might experience side effects or break, as with Python 3.8 there is a clear distinction between a C extension and embedding python.

Benefit to Fedora

Python's performance will increase significantly depending on the workload. Since many core components of the OS also depend on python this could lead to an increase in their performance as well.

Scope

• Proposal owners:
  • Review, merge and build the pull request with the implementation.
  • Go through the packages that embed python in their code and see if things work correctly. Will provide a copr repository to test.

• Other developers: Other developers are encouraged to test the new statically linked python3 to see if their package works as expected

• Release engineering: #Releng issue number (mass rebuild not needed, no releng impact anticipated)

• Policies and guidelines: No changes are required to the packaging guidelines or other documents

• Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

N/A (not a System Wide Change)

How To Test

N/A (not a System Wide Change)

Dependencies

N/A (not a System Wide Change)

Contingency Plan

• Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
• Contingency deadline: N/A (not a System Wide Change)
• Blocks release? N/A (not a System Wide Change), Yes/No
• Blocks product? product

Documentation

N/A (not a System Wide Change)