Build Python 3 to statically link with libpython3.8.a for better performance
Summary
Python 3 traditionally in Fedora was built with a shared library libpython3.?.so and the final binary was dynamically linked against that shared library. This change is about creating the static library and linking the final python3 binary against it, as it provides significant performance improvement, up to 15% depending on the workload. The static library will not be shipped. The shared library will continue to exist in a separate subpackage. In essence, python3 will no longer depend on libpython.
Owner
- Name: Charalampos Stratakis, Miro Hrončok, Victor Stinner
- Email: python-maint@redhat.com
Current status
- Targeted release: Fedora 32
- Last updated: 2019-10-22
- Tracker bug: <will be assigned by the Wrangler>
- Release notes tracker: <will be assigned by the Wrangler>
Detailed Description
When we compile the python3 package on Fedora (prior to this change), we create the libpython3.?.so shared library and the final python3 binary (/usr/bin/python3) is dynamically linked against it. However by building the libpython3.?.a static library and statically linking the final binary against it, we can achieve a performance gain of approximately 15% depending on the workload. Link time optimizations and profile guided optimizations also have a greater impact when python3 is linked statically.
Since Python 3.8, C extensions are no longer linked to libpython by default (for example, they need to utilize the --embed flag for python3-config to do so). During the Python 3.8 upgrade and rebuilds we've uncovered various cases of packages linking to libpython implicitly through various hacks within their buildsystems and fixed as many as possible. However, there are legitimate reasons to link to libpython and for those cases libpython should be provided so applications that embed Python can continue to do so.
This mirrors the Debian/Ubuntu way of building python, where they offer a statically linked binary and an additional libpython subpackage. The libpython subpackage will be created and python3-devel will depend on it, so packages that embed Python will keep working.
By applying this change, libpython's namespace will be separated from python's, so python C extension which are linked to libpython might experience side effects or break.
C extensions that are not linked to libpython and are loaded in Python or an application embedding Python, as well as C extensions linked to libpython in an application embedding Python will not be affected
Currently there is no upstream option to build the static library, as well as the shared one and statically link the final binary to it, so we have to rely on a downstream patch to achieve it. We plan to work with upstream to incorporate the changes there as well.
Benefit to Fedora
Python's performance will increase significantly depending on the workload. Since many core components of the OS also depend on python this could lead to an increase in their performance as well.
Scope
- Proposal owners:
- Review and merge the pull request with the implementation.
- Go through the packages that embed python in their code and see if things work correctly. Will provide a copr repository to test.
- Other developers: Other developers are encouraged to test the new statically linked python3 to see if their package works as expected
- Release engineering: #Releng issue number (mass rebuild not needed, no releng impact anticipated)
- Policies and guidelines: No changes are required to the packaging guidelines or other documents
- Trademark approval: N/A (not needed for this Change)
Upgrade/compatibility impact
N/A (not a System Wide Change)
How To Test
N/A (not a System Wide Change)
User Experience
Dependencies
N/A (not a System Wide Change)
Contingency Plan
- Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
- Contingency deadline: N/A (not a System Wide Change)
- Blocks release? N/A (not a System Wide Change), Yes/No
- Blocks product? product
Documentation
N/A (not a System Wide Change)