(→Scope) |
(→Scope) |
||
| Line 78: | Line 78: | ||
<!-- What work do the feature owners have to accomplish to complete the feature in time for release? Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?--> | <!-- What work do the feature owners have to accomplish to complete the feature in time for release? Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?--> | ||
** Review and merge the [https://src.fedoraproject.org/rpms/python3/pull-request/133 pull request with the implementation]. | ** Review and merge the [https://src.fedoraproject.org/rpms/python3/pull-request/133 pull request with the implementation]. | ||
** Go through the packages that | ** Go through the python C extension packages that are linked to libpython and test if things work correctly. Will provide a copr repository for testing. | ||
* Other developers: Other developers are encouraged to test the new statically linked python3 to see if their package works as expected <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | * Other developers: Other developers are encouraged to test the new statically linked python3 to see if their package works as expected <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
| Line 87: | Line 87: | ||
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication --> | The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing, and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication --> | ||
* Policies and guidelines: | * Policies and guidelines: The packaging guidelines will need to be updated to explicitly mention that C extensions should not be linked to libpython, and that the python3 binary is statically linked. <!-- REQUIRED FOR SYSTEM WIDE CHANGES --> | ||
<!-- Do the packaging guidelines or other documents need to be updated for this feature? If so, does it need to happen before or after the implementation is done? If a FPC ticket exists, add a link here. --> | <!-- Do the packaging guidelines or other documents need to be updated for this feature? If so, does it need to happen before or after the implementation is done? If a FPC ticket exists, add a link here. --> | ||
Revision as of 12:37, 22 October 2019
Build Python 3 to statically link with libpython3.8.a for better performance
Summary
Python 3 traditionally in Fedora was built with a shared library libpython3.?.so and the final binary was dynamically linked against that shared library. This change is about creating the static library and linking the final python3 binary against it, as it provides significant performance improvement, up to 15% depending on the workload. The static library will not be shipped. The shared library will continue to exist in a separate subpackage. In essence, python3 will no longer depend on libpython.
Owner
- Name: Charalampos Stratakis, Miro Hrončok, Victor Stinner
- Email: python-maint@redhat.com
Current status
- Targeted release: Fedora 32
- Last updated: 2019-10-22
- Tracker bug: <will be assigned by the Wrangler>
- Release notes tracker: <will be assigned by the Wrangler>
Detailed Description
When we compile the python3 package on Fedora (prior to this change), we create the libpython3.?.so shared library and the final python3 binary (/usr/bin/python3) is dynamically linked against it. However by building the libpython3.?.a static library and statically linking the final binary against it, we can achieve a performance gain of approximately 15% depending on the workload. Link time optimizations and profile guided optimizations also have a greater impact when python3 is linked statically.
Since Python 3.8, C extensions are no longer linked to libpython by default (for example, they need to utilize the --embed flag for python3-config to do so). During the Python 3.8 upgrade and rebuilds we've uncovered various cases of packages linking to libpython implicitly through various hacks within their buildsystems and fixed as many as possible. However, there are legitimate reasons to link to libpython and for those cases libpython should be provided so applications that embed Python can continue to do so.
This mirrors the Debian/Ubuntu way of building python, where they offer a statically linked binary and an additional libpython subpackage. The libpython subpackage will be created and python3-devel will depend on it, so packages that embed Python will keep working.
By applying this change, libpython's namespace will be separated from python's, so python C extension which are linked to libpython might experience side effects or break.
C extensions that are not linked to libpython and are loaded in Python or an application embedding Python, as well as C extensions linked to libpython in an application embedding Python will not be affected
Currently there is no upstream option to build the static library, as well as the shared one and statically link the final binary to it, so we have to rely on a downstream patch to achieve it. We plan to work with upstream to incorporate the changes there as well.
Benefit to Fedora
Python's performance will increase significantly depending on the workload. Since many core components of the OS also depend on python this could lead to an increase in their performance as well.
Scope
- Proposal owners:
- Review and merge the pull request with the implementation.
- Go through the python C extension packages that are linked to libpython and test if things work correctly. Will provide a copr repository for testing.
- Other developers: Other developers are encouraged to test the new statically linked python3 to see if their package works as expected
- Release engineering: #Releng issue number (mass rebuild not needed, no releng impact anticipated)
- Policies and guidelines: The packaging guidelines will need to be updated to explicitly mention that C extensions should not be linked to libpython, and that the python3 binary is statically linked.
- Trademark approval: N/A (not needed for this Change)
Upgrade/compatibility impact
N/A (not a System Wide Change)
How To Test
N/A (not a System Wide Change)
User Experience
Dependencies
N/A (not a System Wide Change)
Contingency Plan
- Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
- Contingency deadline: N/A (not a System Wide Change)
- Blocks release? N/A (not a System Wide Change), Yes/No
- Blocks product? product
Documentation
N/A (not a System Wide Change)