Further reduce Fedora-specific build flags in non-RPM Python extensions
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
Summary
Continuing the work started with https://fedoraproject.org/wiki/Changes/Python_Extension_Flags, this change is about further reducing the build and linker flags (CFLAGS, CXXFLAGS and LDFLAGS) saved internally in the Python interpreter for use by distutils and other build systems. Compiling non-RPM Python extension modules will carry only the compiler flags required for binary compatibility with the interpreter they were built against and not Fedora specific ones.
Practically that means the only Fedora derived flag will be -fexceptions and Python will apply its own upstream hardcoded ones, making the final flag set for a non-RPM compiled Python extension as follows:
-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions
The current main Python interpreter on Fedora 39 will be modified (Python 3.12) and Python 3.6-3.11 will follow.
Owner
- Name: Charalampos Stratakis
- Email: python-maint AT redhat.com
Current status
- Targeted release: Fedora Linux 39
- Last updated: 2023-06-15
- [<will be assigned by the Wrangler> devel thread]
- FESCo issue: <will be assigned by the Wrangler>
- Tracker bug: <will be assigned by the Wrangler>
- Release notes tracker: <will be assigned by the Wrangler>
Detailed Description
After implementing https://fedoraproject.org/wiki/Changes/Python_Extension_Flags we uncoupled some distro specific compilation and linker flags propagated to C extensions.
However with an ever increasing set of compiler flags being added and applied distro-wide, as compilers and security standards evolve (e.g. -D_FORTIFY_SOURCE=3) it becomes an increasingly complex job to vet each flag that might leak into user-built Python C extensions through the Python interpreter. Instead of removing only some flags and letting the rest follow through, we will be taking a more proactive approach by removing all the compiler and linker flags, except the ones that are required to maintain the binary compatibility with the Python interpreter the extensions were built against which is -fexceptions. We will also preserve the ones that Python hardcodes itself through the Makefile.
Similarly, when a user builds their own C programs, no compiler flags are applied by default and the user is free to making their own decision. Bringing the compilation of Python C extensions closer to that experience is the next logical step.
Currently a user-built Python C extension will be built with:
CFLAGS:
-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv
LDFLAGS:
'-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g -Wl,-z,relro -Wl,--as-needed -Wl,-z,now -Wl,--build-id=sha1 -g'
After this change:
CFLAGS:
-Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions -fexceptions -fexceptions
LDFLAGS: None
Feedback
The initial thread that inspired this change was https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/76RV7VLCOZRHIMTG4J3M4NMIBAD4LO76/#76RV7VLCOZRHIMTG4J3M4NMIBAD4LO76
Benefit to Fedora
Python developers will get more upstream-like experience when building Python extension modules and also closer to building vanilla C programs. Also new decisions made about the distro-wide compiler flags won't necessarily affect Python developers building their extension modules.
In addition any Python developer using Fedora will have the capability to build the extension on Fedora, test it and later ship it and build it on a CI or other systems that are not based on Fedora.
Scope
- Proposal owners:
- Other developers:
- Release engineering: #Releng issue number
- Policies and guidelines: N/A (not needed for this Change)
- Trademark approval: N/A (not needed for this Change)
- Alignment with Community Initiatives:
Upgrade/compatibility impact
Not anticipated. Extension modules (built for the same Python version) are compatible with the interpreter with or without the removed flags back and forth.
How To Test
For users (Python developers)
- build your favorite Python extension module in venv or outside venv with your favorite build system
- observe the used flags and check that the full set of flags are are not there as mentioned in the detailed description, report bugs for
python3.12 - check if the extension works as expected
For packagers (Fedora contributors)
- build your favorite RPM package with Python extension module
- observe the used flags and check that the full set of flags are there and not the reduced one, report bugs for that package otherwise (and block our tracking bug)
- check if the package works as expected
User Experience
See Benefit to Fedora above.
Dependencies
Contingency Plan
- Contingency mechanism: Change owners can revert the change at any point.
- Contingency deadline: final freeze (not a System Wide Change)
- Blocks release? No
Documentation
N/A (not a System Wide Change)