From Fedora Project Wiki
 
(33 intermediate revisions by 3 users not shown)
Line 6: Line 6:
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release.  
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release.  
Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->
Note that motivation for the change should be in the Benefit to Fedora section below, and this part should answer the question "What?" rather than "Why?". -->
The Python standard library bytecode cache files (e.g. <code>/usr/lib64/python3.9/.../__pycache__/*.pyc</code>) will be moved from the {{package|python3-libs}} package to three new optional subpackages (split by optimization level). The non-optimized bytecode cache will be recommended by {{package|python3-libs}} and installed by default but removable. The bytecode cache optimization level 1 and 2 will not be recommended (and hence will not be installed by default) but will be installable. The default SELinux policy will be adapted not to audit AVC denials when the bytecode cache is created by Python on runtime. This will '''save 8.89 MiB disk space on default installations''' or '''17.12 MiB on minimal installations''' (by opting-out from the recommended subpackage with non-optimized bytecode cache). When all three new packages are installed, the size will increase slightly over the status quo (by 4.5 MiB).
The Python standard library bytecode cache files (e.g. `/usr/lib64/python3.9/.../__pycache__/*.pyc`) will be moved from the {{package|python3-libs}} package to three new optional subpackages (split by optimization level). The non-optimized bytecode cache will be recommended by {{package|python3-libs}} and installed by default but removable. The bytecode cache optimization level 1 and 2 will not be recommended (and hence will not be installed by default) but will be installable. The default SELinux policy will be adapted not to audit AVC denials when the bytecode cache is created by Python at runtime. This will '''save 8.89 MiB disk space on default installations''' or '''17.12 MiB on minimal installations''' (by opting-out from the recommended subpackage with non-optimized bytecode cache). When all three new packages are installed, the size will increase slightly over the status quo (by 4.5 MiB).


== Owner ==
== Owner ==
Line 56: Line 56:
=== What is the Python bytecode cache ===
=== What is the Python bytecode cache ===


When Python code is interpreted, it is compiled to [https://docs.python.org/3/glossary.html#term-bytecode Python bytecode]. When a pure Python module is imported for the first time, the compiled bytecode is serialized and cached to a <code>.pyc</code> file located in the <code>__pycache__</code> directory next to the <code>.py</code> source. Subsequent imports use the cache directly, until it is invalidated (for example when the <code>.py</code> source is edited and its <code>mtime</code> stamp is bumped) -- at that point, the cache is updated. This behavior is explained in detail in [https://www.python.org/dev/peps/pep-3147/#python-behavior PEP 3147]. The invalidation is described in [https://www.python.org/dev/peps/pep-0552/ PEP 552].
When Python code is interpreted, it is compiled to [https://docs.python.org/3/glossary.html#term-bytecode Python bytecode]. When a pure Python module is imported for the first time, the compiled bytecode is serialized and cached to a `.pyc` file located in the `__pycache__` directory next to the `.py` source. Subsequent imports use the cache directly, until it is invalidated (for example when the `.py` source is edited and its `mtime` stamp is bumped) -- at that point, the cache is updated. This behavior is explained in detail in [https://www.python.org/dev/peps/pep-3147/#python-behavior PEP 3147]. The invalidation is described in [https://www.python.org/dev/peps/pep-0552/ PEP 552].


Python can operate in 3 different optimization levels: 0, 1 and 2. By default, the optimization level is 0. When invoked with the [https://docs.python.org/3/using/cmdline.html#cmdoption-o <code>-O</code> command line option] optimization is set to 1, similarly with [https://docs.python.org/3/using/cmdline.html#cmdoption-oo <code>-OO</code>] it is 2. Bytecode cache for different optimization levels is saved with different filenames as described in [https://www.python.org/dev/peps/pep-0488/ PEP 488].
Python can operate in 3 different optimization levels: 0, 1 and 2. By default, the optimization level is 0. When invoked with the [https://docs.python.org/3/using/cmdline.html#cmdoption-o `-O` command line option] optimization is set to 1, similarly with [https://docs.python.org/3/using/cmdline.html#cmdoption-oo `-OO`] it is 2. Bytecode cache for different optimization levels is saved with different filenames as described in [https://www.python.org/dev/peps/pep-0488/ PEP 488].


As an example, a Python module located at <code>/path/to/basename.py</code> will have bytecode cache files for CPython 3.9 stored as:
As an example, a Python module located at `/path/to/basename.py` will have bytecode cache files for CPython 3.9 stored as:


* <code>/path/to/__pycache__/basename.cpython-39.pyc</code> for the non-optimized bytecode
* `/path/to/__pycache__/basename.cpython-39.pyc` for the non-optimized bytecode
* <code>/path/to/__pycache__/basename.cpython-39.opt-1.pyc</code> for optimization level 1
* `/path/to/__pycache__/basename.cpython-39.opt-1.pyc` for optimization level 1
* <code>/path/to/__pycache__/basename.cpython-39.opt-2.pyc</code> for optimization level 2
* `/path/to/__pycache__/basename.cpython-39.opt-2.pyc` for optimization level 2


=== Python bytecode cache in RPM packages (status quo) ===
=== Python bytecode cache in RPM packages (status quo) ===


Pure Python modules shipped in RPM packages (and namely the ones shipped trough the {{package|python3-libs}} package) are located at paths not writable by regular user, under <code>/usr/lib(64)/python3.9/</code>, hence the bytecode cache is also located in such locations. To work around this problem, the bytecode cache is pre-compiled when RPM packages are built and {{package|python3-libs}} ships and owns the sources as well as the bytecode cache:
Pure Python modules shipped in RPM packages (and namely the ones shipped trough the {{package|python3-libs}} package) are located at paths not writable by regular user, under `/usr/lib(64)/python3.9/`, hence the bytecode cache is also located in such locations. To work around this problem, the bytecode cache is pre-compiled when RPM packages are built and {{package|python3-libs}} ships and owns the sources as well as the bytecode cache:


  $ rpm -ql python3-libs
  $ rpm -ql python3-libs
Line 94: Line 94:
  8440 /usr/lib64/python3.9/__pycache__/ast.cpython-39.opt-2.pyc
  8440 /usr/lib64/python3.9/__pycache__/ast.cpython-39.opt-2.pyc
  8441 /usr/lib64/python3.9/__pycache__/ast.cpython-39.pyc
  8441 /usr/lib64/python3.9/__pycache__/ast.cpython-39.pyc
=== What if the bytecode cache would not be packaged ===
When the bytecode cache is not packaged, several things happen:
# When non-root users run Python, the imported modules are never cached. As a result, the startup time of Python apps might be slightly larger than necessary until root runs them.
# When root runs Python, the imported modules are cached. As a result untracked `.pyc` files start to pop up in `/usr/lib(64)/python3.9/`. When the system is updated to a newer Python version, the untracked files remain on the filesystem until manually cleaned up.
# When root runs Python in SELinux restricted context, the imported modules are attempted to be cached but SELinux does not allow that. The result is same as (1) with a lot of noise from SELinux.
=== Packaging the bytecode cache into optional subpackages ===
To be able to [[#Size_impact|save quite some disk space]] without disrupting the user experience, we propose to ship the pre-compiled bytecode cache previously included in {{package|python3-libs}} as follows:
* Pre-compiled non-optimized bytecode cache files (`*.cpython-39.pyc`) will be packaged in {{package|python3-libs-bytecode-opt-0}}.
* Pre-compiled level 1 optimized bytecode cache files (`*.cpython-39.opt-1.pyc`) will be packaged in {{package|python3-libs-bytecode-opt-1}}.
* Pre-compiled level 2 optimized bytecode cache files (`*.cpython-39.opt-2.pyc`) will be packaged in {{package|python3-libs-bytecode-opt-2}}.
In order to properly own any runtime-generated bytecode-cache files, the {{package|python3-libs}} will list all files listed in the three abovementioned packages as `%ghost`.
{{package|python3-libs}} will not ''Require'' any of the bytecode cache packages, hence the packages will be (un)installable and fully optional.
Given that almost all Fedora Python packages invoke Python in the non-optimized mode¹, {{package|python3-libs}} will ''Recommend'' {{package|python3-libs-bytecode-opt-0}} and hence the package will be installed by default together with Python; the user experience will remain the same for the vast majority of users and use cases.
Furthermore, container images and other minimal systems maintainers may choose to exclude the {{package|python3-libs-bytecode-opt-0}} package to save more disk space if desired.
Note that by splitting the three optimization levels to different RPM packages, files can no longer be hardlinked between each other. This results in a slight size increase when all three optimization levels are installed. The change owners consider the need for all three subpackages to be present simultaneously on one size-sensitive system unlikely and hence consider this a fair trade.
¹ No real data was collected to support this claim. This hypothesis is made by the Python maintainers based on their own experience.
=== SELinux policy changes ===
In order to suppress the otherwise omnipresent AVC denial messages about Python failing to write the bytecode cache, the Python maintainers have teamed up with the Fedora's selinux-policy maintainers to suppress those. The implementation details about this are available at:
* https://github.com/fedora-selinux/selinux-policy/pull/404
When Python runs under the root user in SELinux restricted context, SELinux will still prevent it from writing the bytecode cache, but it will not clutter the audit log.
=== Size impact ===
Sizes calculated in `mock` on x86_64. Only {{package|python3-libs}} and the relevant bytecode cache packages were installed (i.e. no other Python packages). By `du -c /usr/lib64/python3.9` (converted to MiBs by dividing by 1024 and rounding to 2 decimal places).
{| class="wikitable" style="text-align: right;"
|-
! style="text-align: right;" | Situation !! Size of `/usr/lib(64)/python3.9` !! Difference in MiB !! Difference in %
|-
| Status quo (before this change) || 31.84 MiB <!-- 32608 KiB ---> || ||
|-
| Default (non-optimized cache only) || 22.96 MiB <!-- 23508 KiB ---> || -8.89 MiB || -27.91 %
|-
| No cache || 14.72 MiB <!-- 15076 KiB ---> || -17.12 MiB || -53.77 %
|-
| Non-optimized cache and optimization level 1 || 29.71 MiB <!-- 30424 KiB ---> || -2.13 MiB || -6.70 %
|-
| All optimization levels (same files as status quo) || 36.35 MiB <!-- 37220 KiB ---> || +4.50 MiB || +14.14 %
|}
=== Speed impact ===
The presence or absence of the bytecode cache only impacts the speed of imports. It is most common that the imports happen while an application starts. Once the application is running, there is no speed difference.
A totally inappropriate and unscientific experiment:
$ du -a /usr/lib64/python3.9/ | grep py$ | sort -n -r | head -n 1
224 /usr/lib64/python3.9/_pydecimal.py
With caches:
$ time python3 -c 'import importlib as i, _pydecimal as p; [i.reload(p) for _ in range(10000)]'
real 0m13.986s
user 0m13.554s
sys 0m0.365s
$ time python3 -O -c 'import importlib as i, _pydecimal as p; [i.reload(p) for _ in range(10000)]'
real 0m13.594s
user 0m13.186s
sys 0m0.337s
$ time python3 -OO -c 'import importlib as i, _pydecimal as p; [i.reload(p) for _ in range(10000)]'
real 0m13.225s
user 0m12.855s
sys 0m0.290s
Without caches:
$ time python3 -c 'import importlib as i, _pydecimal as p; [i.reload(p) for _ in range(10000)]'
real 4m20.554s
user 4m14.600s
sys 0m4.850s
$ time python3 -O -c 'import importlib as i, _pydecimal as p; [i.reload(p) for _ in range(10000)]'
real 4m14.291s
user 4m9.333s
sys 0m3.721s
$ time python3 -OO -c 'import importlib as i, _pydecimal as p; [i.reload(p) for _ in range(10000)]'
real 4m14.816s
user 4m11.035s
sys 0m2.400s
This suggests that an application that does 10000 module imports (with rather large 224 KiB modules) would be slowed down on start by 4 minutes. Obviously, such measurements depend on many aspects and doing 10000 imports is rather far-fetched -- there are only ~550 pure Python modules in {{package|python3-libs}}, hence even if they all are imported, the slowdown should not exceed couple seconds. However, it is indisputable that importing modules without the cache is significantly slower.
Deployments negatively impacted by this are advised to either install the appropriate bytecode subpackage or pre-compile the relevant modules ahead of time (e.g. when building a container image).
=== Rejected ideas ===
In this section, we briefly describe ideas that were presented by others or considered by the change owners, but rejected.
Many Python minimization proposals were previously described in great detail in [https://github.com/hroncok/python-minimization/blob/master/document.md Python minimization in Fedora] and [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/LACP3PFQPUO6BQQLYYJDFF4CR3DHWRSQ/ discussed on the devel mailing list].
==== Stop shipping mandatory `.py` sources, ship only `.pyc` cache ====
This is described as [https://github.com/hroncok/python-minimization/blob/master/document.md#solution-7-stop-shipping-mandatory-source-files-ship-pyc-instead Solution 7] in the abovementioned document.
It is possible to ship bytecode cache installed e.g. as `/usr/lib64/python3.9/ast.pyc` instead of the appropriate Python source (e.g. `/usr/lib64/python3.9/ast.py` in our example). Such solution could save additional 3.1 MiB when no sources and no other optimization levels would be shipped. In addition, it would also not suffer from any import slowdowns regardless of the optimization level Python was invoked with. Our analysis however shows significant drawbacks when shipping `.pyc` files only:
* Without the source codes, some Python tracebacks are less informational.
* Without the source codes, many IDEs and other Python developer tools might be confused (e.g. code completion or the IPython `??` syntax to show a function's code).
* Sysadmins and ops are notoriously known to edit Python source files (including the standard library) on production 🎩.
* The shipped `.pyc` files would need to be compiled with a certain optimization level (presumably 0, without loss of generality). Python invoked with `-O` or `-OO` would still execute such `.pyc` file regardless -- in special circumstances, this can lead to slight behavior nuances that would be very hard to debug.
This can be somehow worked around by offering the possibility to install the sources (or even recommend them by default). However:
* With `module.py` present, the `module.pyc` files is always totally ignored.
* The `module.py` would then still need to ship various optimization level bytecode caches in the `__pycache__` directory, possibly duplicating `module.pyc` and `__pycache__/module.cpython-39.pyc` (there is no way to hardlink those on RPM level, since they are in different directory).
* Just by installing the optional sources on production in order to be able to debug problems, Python invoked in different optimization level than the one used to pre-compile the shipped `module.pyc` would suddenly start executing different bytecode, essentially producing a haisenbug waiting to happen.
The change owners consider shipping individual well-tested, large, modules with machine-generated sources as `.pyc` only to save space a reasonable compromise (and it was [https://src.fedoraproject.org/rpms/python3.9/pull-request/16 already done] with `pydoc_data.topics` and several `encodings` submodules). However doing it with the entire standard library would provide a very non-standard user experience and might possibly blow up at many places.
==== Make Python not attempt to write bytecode cache into `/usr/lib(64)/python3.9` ====
Originally, when panning this change, the idea was to prevent Python to write the bytecode cache to `/usr/lib(64)/python3.9/...` on imports by a special marker file (e.g. `/usr/lib(64)/python3.9/nocache` or similar). Such marker would mean "this directory (possibly recursively) contains bytecode maintained by non-Python tooling". Python would use the cache if present, but it would not even attempt to write the cache if it is missing or outdated. Such change of behavior would need to be introduced in Python upstream.
When drafting this change for upstream, we have failed to provide sufficient reasoning to introduce such new behavior. The only problems with writing the cache identified by the change owners were:
* files not owned by any RPM package (solved by `%ghost`)
* SELinux AVC denials noise (solved by adapting `selinux-policy`)
Hence, this idea was rejected.
==== Keep the hardlink based filesize optimizations by RPM trickery ====
When all three bytecode packages are installed, the disk usage is slightly bigger than before this change because files in different packages cannot be hardlinked on RPM level. In theory, it might be possible to compensate this by various means. For example:
* Hardlink the files in `%posttrans` scriplet.
* On build time, detect what files are identical, hardlink such files and include all their instances in all relevant packages (e.g. all three bytecode packages would contain all three hardlinked versions of `__pycache__/abc.cpython-39*.pyc`, but only one version of `__pycache__/ast.cpython-39*.pyc`).
Such solutions are considered not worth it by the change owners, because they would make the user experience and/or the packaging unnecessary complicated. As said previously, the change owners consider the need for all three subpackages to be present simultaneously on one size-sensitive system unlikely.
=== Not realized ideas ===
In this section, we briefly describe ideas that were presented by others or considered by the change owners, but were not realized (e.g. for capacity reasons). Such ideas may be realized later.
==== Store bytecode cache in `/var/cache` and/or `~/.cache` ====
The change owners ''feel'' that runtime created cache should not be stored in `/usr/lib(64)` but rather `/var/cache` and/or `~/.cache`. Since Python 3.8 it is possible to set [https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX the `PYTHONPYCACHEPREFIX` environment variable] to modify the location of the bytecode cache.
Changes could be proposed to Python upstream to default for cache in cache-specific filesystem paths (with paths priorities etc.), but this idea is not being pursued at this time.
==== Apply this change to all Python RPM packages ====
The motivation and implementation of this change proposal can be in theory extended to ''all'' Python RPM packages in Fedora. However, since this is a tad tedious from the packaging perspective, an automation that would significantly free the packagers away from the technical details would need to be created. If this is desired, the idea can be pursued later (possibly once this change is battle tested).


== Feedback ==
== Feedback ==


<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->
<!-- Summarize the feedback from the community and address why you chose not to accept proposed alternatives. This section is optional for all change proposals but is strongly suggested. Incorporating feedback here as it is raised gives FESCo a clearer view of your proposal and leaves a good record for the future. If you get no feedback, that is useful to note in this section as well. For innovative or possibly controversial ideas, consider collecting feedback before you file the change proposal. -->
Many Python minimization proposals were previously described in great detail in [https://github.com/hroncok/python-minimization/blob/master/document.md Python minimization in Fedora] and [https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/LACP3PFQPUO6BQQLYYJDFF4CR3DHWRSQ/ discussed on the devel mailing list]. This proposal is also based on feedback received there.


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 128: Line 297:
     https://fedoraproject.org/wiki/Changes/perl5.26 (major upgrade to a popular software stack, visible to users of that stack)
     https://fedoraproject.org/wiki/Changes/perl5.26 (major upgrade to a popular software stack, visible to users of that stack)
-->
-->
[[File:PythonFloppy.jpg|thumb|right|77 3.5" floppy disks could fit the entire Python installation, courtesy of Harold Miller]]
* In the default scenario, 8.89 MiB disk space saved.
* In the minimal scenario, 17.12 MiB disk space saved.
* Bandwidth saved on dnf upgrades as well (both on clients and mirrors).


== Scope ==
== Scope ==
* Proposal owners:
* Proposal owners:
** Update the Python package, see https://src.fedoraproject.org/rpms/python3.9/pull-request/29
** Update selinux-policy, see https://github.com/fedora-selinux/selinux-policy/pull/404
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


Line 136: Line 312:
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Release engineering: [https://pagure.io/releng/issues #Releng issue number] (a check of an impact with Release Engineering is needed) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Release engineering: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.  
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
Line 146: Line 322:
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->


* Alignment with Objectives:  
* Alignment with Objectives: [https://docs.fedoraproject.org/en-US/minimization/ Fedora Minimization Objective]
<!-- Does your proposal align with the current Fedora Objectives: https://docs.fedoraproject.org/en-US/project/objectives/ ? It's okay if it doesn't, but it's something to consider -->
<!-- Does your proposal align with the current Fedora Objectives: https://docs.fedoraproject.org/en-US/project/objectives/ ? It's okay if it doesn't, but it's something to consider -->


Line 153: Line 329:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
Fedora installations upgrading to this change (either rawhide users or on distro upgrades to Fedora 34) will be impacted slightly. Users will end up with {{package|python3-libs}} and {{package|python3-libs-bytecode-opt-0}} installed, but all the previously included bytecode cache files for optimization levels 1 and 2 will remain present unless manually removed or until Python is upgraded in Fedora to 3.10+. Such files will remain outdated and hence most likely become invalid and useless over time.
 
Hence the change owners have prepared a one-time-off scriptlet that would clean the files on the first upgrade to the new configuration to prevent this from happening. Such scriptlet can be removed in Fedora 36 or sooner when Fedora upgrades to Python 3.10+.


== How To Test ==
== How To Test ==
Line 171: Line 349:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
TBD


== User Experience ==
== User Experience ==
Line 184: Line 362:
  - Green has been scientifically proven to be the most relaxing color. The move to a default background color of green with green text will result in Fedora users being the most relaxed users of any operating system.
  - Green has been scientifically proven to be the most relaxing color. The move to a default background color of green with green text will result in Fedora users being the most relaxed users of any operating system.
-->
-->
Users will get {{package|python3-libs-bytecode-opt-0}} by default and in most cases should not notice any difference except a smaller filesystem footprint.
Users executing Python with `-O` or `-OO` may notice slight slowdown of Python or Python applications startup and may need to install {{package|python3-libs-bytecode-opt-1}}/{{package|python3-libs-bytecode-opt-2}} as needed.
Users of minimal systems where the maintainers removed {{package|python3-libs-bytecode-opt-0}} may notice slight slowdown of Python or Python applications startup and may need to install {{package|python3-libs-bytecode-opt-0}} if they are concerned. Alternatively, they may bytecompile only the relevant parts of the Python standard library ahead of time (e.g. when building derived container images). Users who prefer the bytecode not to be there may want to build their container images with [https://docs.python.org/3/using/cmdline.html#envvar-PYTHONDONTWRITEBYTECODE `PYTHONDONTWRITEBYTECODE` environment variable] set.
When the {{package|python3-libs-bytecode-opt-0}}/{{package|python3-libs-bytecode-opt-1}}/{{package|python3-libs-bytecode-opt-2}} package is installed, it will override any runtime-generated bytecode cache.
When the {{package|python3-libs-bytecode-opt-0}}/{{package|python3-libs-bytecode-opt-1}}/{{package|python3-libs-bytecode-opt-2}} package is uninstalled, RPM would leave all the files on disk because they are also owned by {{package|python3-libs}} -- we consider this a bad UX and hence we will add `%postun` scriptlets to remove the files.


== Dependencies ==
== Dependencies ==
Line 194: Line 381:


<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
<!-- If you cannot complete your feature by the final development freeze, what is the backup plan?  This might be as simple as "Revert the shipped configuration".  Or it might not (e.g. rebuilding a number of dependent packages).  If you feature is not completed in time we want to assure others that other parts of Fedora will not be in jeopardy.  -->
* Contingency mechanism: (What to do?  Who will do it?) N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency mechanism: Abort, abort! <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
<!-- When is the last time the contingency mechanism can be put in place?  This will typically be the beta freeze. -->
* Contingency deadline: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Contingency deadline: before the beta freeze <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
<!-- Does finishing this feature block the release, or can we ship with the feature in incomplete state? -->
* Blocks release? N/A (not a System Wide Change), Yes/No <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks release? Not a chance <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Blocks product? product <!-- Applicable for Changes that blocks specific product release/Fedora.next -->
* Blocks product? It plocks a broduct <!-- Applicable for Changes that blocks specific product release/Fedora.next -->


== Documentation ==
== Documentation ==
Line 205: Line 392:


<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
N/A (not a System Wide Change)
TBD


== Release Notes ==
== Release Notes ==
Line 213: Line 400:
Release Notes are not required for initial draft of the Change Proposal but has to be completed by the Change Freeze.  
Release Notes are not required for initial draft of the Change Proposal but has to be completed by the Change Freeze.  
-->
-->
TBD

Latest revision as of 18:35, 8 September 2020


Python: Optional Bytecode Cache

Summary

The Python standard library bytecode cache files (e.g. /usr/lib64/python3.9/.../__pycache__/*.pyc) will be moved from the python3-libs package to three new optional subpackages (split by optimization level). The non-optimized bytecode cache will be recommended by python3-libs and installed by default but removable. The bytecode cache optimization level 1 and 2 will not be recommended (and hence will not be installed by default) but will be installable. The default SELinux policy will be adapted not to audit AVC denials when the bytecode cache is created by Python at runtime. This will save 8.89 MiB disk space on default installations or 17.12 MiB on minimal installations (by opting-out from the recommended subpackage with non-optimized bytecode cache). When all three new packages are installed, the size will increase slightly over the status quo (by 4.5 MiB).

Owner

Current status

  • Targeted release: Fedora 34
  • Last updated: 2020-09-08
  • FESCo issue: <will be assigned by the Wrangler>
  • Tracker bug: <will be assigned by the Wrangler>
  • Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

What is the Python bytecode cache

When Python code is interpreted, it is compiled to Python bytecode. When a pure Python module is imported for the first time, the compiled bytecode is serialized and cached to a .pyc file located in the __pycache__ directory next to the .py source. Subsequent imports use the cache directly, until it is invalidated (for example when the .py source is edited and its mtime stamp is bumped) -- at that point, the cache is updated. This behavior is explained in detail in PEP 3147. The invalidation is described in PEP 552.

Python can operate in 3 different optimization levels: 0, 1 and 2. By default, the optimization level is 0. When invoked with the -O command line option optimization is set to 1, similarly with -OO it is 2. Bytecode cache for different optimization levels is saved with different filenames as described in PEP 488.

As an example, a Python module located at /path/to/basename.py will have bytecode cache files for CPython 3.9 stored as:

  • /path/to/__pycache__/basename.cpython-39.pyc for the non-optimized bytecode
  • /path/to/__pycache__/basename.cpython-39.opt-1.pyc for optimization level 1
  • /path/to/__pycache__/basename.cpython-39.opt-2.pyc for optimization level 2

Python bytecode cache in RPM packages (status quo)

Pure Python modules shipped in RPM packages (and namely the ones shipped trough the python3-libs package) are located at paths not writable by regular user, under /usr/lib(64)/python3.9/, hence the bytecode cache is also located in such locations. To work around this problem, the bytecode cache is pre-compiled when RPM packages are built and python3-libs ships and owns the sources as well as the bytecode cache: 

$ rpm -ql python3-libs
...
/usr/lib64/python3.9/__pycache__/ast.cpython-39.opt-1.pyc
/usr/lib64/python3.9/__pycache__/ast.cpython-39.opt-2.pyc
/usr/lib64/python3.9/__pycache__/ast.cpython-39.pyc
...
/usr/lib64/python3.9/ast.py
...

As a result, the package is quite big, essentially shipping all pure Python modules 4 times.

Depending of the module content, its bytecode cache files might be identical across optimization levels. For such cases, the files are hardlinked to reduce the bloat: 

$ ls -1i /usr/lib64/python3.9/collections/__pycache__/abc.*pyc
8634 /usr/lib64/python3.9/collections/__pycache__/abc.cpython-39.opt-1.pyc
8634 /usr/lib64/python3.9/collections/__pycache__/abc.cpython-39.opt-2.pyc
8634 /usr/lib64/python3.9/collections/__pycache__/abc.cpython-39.pyc

This is however not possible for all the modules from python3-libs: 

$ ls -1i /usr/lib64/python3.9/__pycache__/ast.*pyc
8438 /usr/lib64/python3.9/__pycache__/ast.cpython-39.opt-1.pyc
8440 /usr/lib64/python3.9/__pycache__/ast.cpython-39.opt-2.pyc
8441 /usr/lib64/python3.9/__pycache__/ast.cpython-39.pyc

What if the bytecode cache would not be packaged

When the bytecode cache is not packaged, several things happen:

  1. When non-root users run Python, the imported modules are never cached. As a result, the startup time of Python apps might be slightly larger than necessary until root runs them.
  2. When root runs Python, the imported modules are cached. As a result untracked .pyc files start to pop up in /usr/lib(64)/python3.9/. When the system is updated to a newer Python version, the untracked files remain on the filesystem until manually cleaned up.
  3. When root runs Python in SELinux restricted context, the imported modules are attempted to be cached but SELinux does not allow that. The result is same as (1) with a lot of noise from SELinux.

Packaging the bytecode cache into optional subpackages

To be able to save quite some disk space without disrupting the user experience, we propose to ship the pre-compiled bytecode cache previously included in python3-libs as follows:

In order to properly own any runtime-generated bytecode-cache files, the python3-libs will list all files listed in the three abovementioned packages as %ghost.

python3-libs will not Require any of the bytecode cache packages, hence the packages will be (un)installable and fully optional.

Given that almost all Fedora Python packages invoke Python in the non-optimized mode¹, python3-libs will Recommend python3-libs-bytecode-opt-0 and hence the package will be installed by default together with Python; the user experience will remain the same for the vast majority of users and use cases.

Furthermore, container images and other minimal systems maintainers may choose to exclude the python3-libs-bytecode-opt-0 package to save more disk space if desired.

Note that by splitting the three optimization levels to different RPM packages, files can no longer be hardlinked between each other. This results in a slight size increase when all three optimization levels are installed. The change owners consider the need for all three subpackages to be present simultaneously on one size-sensitive system unlikely and hence consider this a fair trade.

¹ No real data was collected to support this claim. This hypothesis is made by the Python maintainers based on their own experience.

SELinux policy changes

In order to suppress the otherwise omnipresent AVC denial messages about Python failing to write the bytecode cache, the Python maintainers have teamed up with the Fedora's selinux-policy maintainers to suppress those. The implementation details about this are available at:

When Python runs under the root user in SELinux restricted context, SELinux will still prevent it from writing the bytecode cache, but it will not clutter the audit log.

Size impact

Sizes calculated in mock on x86_64. Only python3-libs and the relevant bytecode cache packages were installed (i.e. no other Python packages). By du -c /usr/lib64/python3.9 (converted to MiBs by dividing by 1024 and rounding to 2 decimal places).

Situation Size of /usr/lib(64)/python3.9 Difference in MiB Difference in %
Status quo (before this change) 31.84 MiB
Default (non-optimized cache only) 22.96 MiB -8.89 MiB -27.91 %
No cache 14.72 MiB -17.12 MiB -53.77 %
Non-optimized cache and optimization level 1 29.71 MiB -2.13 MiB -6.70 %
All optimization levels (same files as status quo) 36.35 MiB +4.50 MiB +14.14 %

Speed impact

The presence or absence of the bytecode cache only impacts the speed of imports. It is most common that the imports happen while an application starts. Once the application is running, there is no speed difference.

A totally inappropriate and unscientific experiment: 

$ du -a /usr/lib64/python3.9/ | grep py$ | sort -n -r | head -n 1
224	/usr/lib64/python3.9/_pydecimal.py

With caches: 

$ time python3 -c 'import importlib as i, _pydecimal as p; [i.reload(p) for _ in range(10000)]'

real	0m13.986s
user	0m13.554s
sys	0m0.365s

$ time python3 -O -c 'import importlib as i, _pydecimal as p; [i.reload(p) for _ in range(10000)]'

real	0m13.594s
user	0m13.186s
sys	0m0.337s

$ time python3 -OO -c 'import importlib as i, _pydecimal as p; [i.reload(p) for _ in range(10000)]'

real	0m13.225s
user	0m12.855s
sys	0m0.290s


Without caches: 

$ time python3 -c 'import importlib as i, _pydecimal as p; [i.reload(p) for _ in range(10000)]'

real	4m20.554s
user	4m14.600s
sys	0m4.850s

$ time python3 -O -c 'import importlib as i, _pydecimal as p; [i.reload(p) for _ in range(10000)]'

real	4m14.291s
user	4m9.333s
sys	0m3.721s

$ time python3 -OO -c 'import importlib as i, _pydecimal as p; [i.reload(p) for _ in range(10000)]'

real	4m14.816s
user	4m11.035s
sys	0m2.400s

This suggests that an application that does 10000 module imports (with rather large 224 KiB modules) would be slowed down on start by 4 minutes. Obviously, such measurements depend on many aspects and doing 10000 imports is rather far-fetched -- there are only ~550 pure Python modules in python3-libs, hence even if they all are imported, the slowdown should not exceed couple seconds. However, it is indisputable that importing modules without the cache is significantly slower.

Deployments negatively impacted by this are advised to either install the appropriate bytecode subpackage or pre-compile the relevant modules ahead of time (e.g. when building a container image).

Rejected ideas

In this section, we briefly describe ideas that were presented by others or considered by the change owners, but rejected.

Many Python minimization proposals were previously described in great detail in Python minimization in Fedora and discussed on the devel mailing list.

Stop shipping mandatory .py sources, ship only .pyc cache

This is described as Solution 7 in the abovementioned document.

It is possible to ship bytecode cache installed e.g. as /usr/lib64/python3.9/ast.pyc instead of the appropriate Python source (e.g. /usr/lib64/python3.9/ast.py in our example). Such solution could save additional 3.1 MiB when no sources and no other optimization levels would be shipped. In addition, it would also not suffer from any import slowdowns regardless of the optimization level Python was invoked with. Our analysis however shows significant drawbacks when shipping .pyc files only:

  • Without the source codes, some Python tracebacks are less informational.
  • Without the source codes, many IDEs and other Python developer tools might be confused (e.g. code completion or the IPython ?? syntax to show a function's code).
  • Sysadmins and ops are notoriously known to edit Python source files (including the standard library) on production 🎩.
  • The shipped .pyc files would need to be compiled with a certain optimization level (presumably 0, without loss of generality). Python invoked with -O or -OO would still execute such .pyc file regardless -- in special circumstances, this can lead to slight behavior nuances that would be very hard to debug.

This can be somehow worked around by offering the possibility to install the sources (or even recommend them by default). However:

  • With module.py present, the module.pyc files is always totally ignored.
  • The module.py would then still need to ship various optimization level bytecode caches in the __pycache__ directory, possibly duplicating module.pyc and __pycache__/module.cpython-39.pyc (there is no way to hardlink those on RPM level, since they are in different directory).
  • Just by installing the optional sources on production in order to be able to debug problems, Python invoked in different optimization level than the one used to pre-compile the shipped module.pyc would suddenly start executing different bytecode, essentially producing a haisenbug waiting to happen.

The change owners consider shipping individual well-tested, large, modules with machine-generated sources as .pyc only to save space a reasonable compromise (and it was already done with pydoc_data.topics and several encodings submodules). However doing it with the entire standard library would provide a very non-standard user experience and might possibly blow up at many places.

Make Python not attempt to write bytecode cache into /usr/lib(64)/python3.9

Originally, when panning this change, the idea was to prevent Python to write the bytecode cache to /usr/lib(64)/python3.9/... on imports by a special marker file (e.g. /usr/lib(64)/python3.9/nocache or similar). Such marker would mean "this directory (possibly recursively) contains bytecode maintained by non-Python tooling". Python would use the cache if present, but it would not even attempt to write the cache if it is missing or outdated. Such change of behavior would need to be introduced in Python upstream.

When drafting this change for upstream, we have failed to provide sufficient reasoning to introduce such new behavior. The only problems with writing the cache identified by the change owners were:

  • files not owned by any RPM package (solved by %ghost)
  • SELinux AVC denials noise (solved by adapting selinux-policy)

Hence, this idea was rejected.

Keep the hardlink based filesize optimizations by RPM trickery

When all three bytecode packages are installed, the disk usage is slightly bigger than before this change because files in different packages cannot be hardlinked on RPM level. In theory, it might be possible to compensate this by various means. For example:

  • Hardlink the files in %posttrans scriplet.
  • On build time, detect what files are identical, hardlink such files and include all their instances in all relevant packages (e.g. all three bytecode packages would contain all three hardlinked versions of __pycache__/abc.cpython-39*.pyc, but only one version of __pycache__/ast.cpython-39*.pyc).

Such solutions are considered not worth it by the change owners, because they would make the user experience and/or the packaging unnecessary complicated. As said previously, the change owners consider the need for all three subpackages to be present simultaneously on one size-sensitive system unlikely.

Not realized ideas

In this section, we briefly describe ideas that were presented by others or considered by the change owners, but were not realized (e.g. for capacity reasons). Such ideas may be realized later.

Store bytecode cache in /var/cache and/or ~/.cache

The change owners feel that runtime created cache should not be stored in /usr/lib(64) but rather /var/cache and/or ~/.cache. Since Python 3.8 it is possible to set the PYTHONPYCACHEPREFIX environment variable to modify the location of the bytecode cache.

Changes could be proposed to Python upstream to default for cache in cache-specific filesystem paths (with paths priorities etc.), but this idea is not being pursued at this time.

Apply this change to all Python RPM packages

The motivation and implementation of this change proposal can be in theory extended to all Python RPM packages in Fedora. However, since this is a tad tedious from the packaging perspective, an automation that would significantly free the packagers away from the technical details would need to be created. If this is desired, the idea can be pursued later (possibly once this change is battle tested).

Feedback

Many Python minimization proposals were previously described in great detail in Python minimization in Fedora and discussed on the devel mailing list. This proposal is also based on feedback received there.

Benefit to Fedora

77 3.5" floppy disks could fit the entire Python installation, courtesy of Harold Miller
  • In the default scenario, 8.89 MiB disk space saved.
  • In the minimal scenario, 17.12 MiB disk space saved.
  • Bandwidth saved on dnf upgrades as well (both on clients and mirrors).

Scope

  • Other developers: N/A (not a System Wide Change)
  • Release engineering: N/A (not a System Wide Change)
  • Policies and guidelines: N/A (not a System Wide Change)
  • Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

Fedora installations upgrading to this change (either rawhide users or on distro upgrades to Fedora 34) will be impacted slightly. Users will end up with python3-libs and python3-libs-bytecode-opt-0 installed, but all the previously included bytecode cache files for optimization levels 1 and 2 will remain present unless manually removed or until Python is upgraded in Fedora to 3.10+. Such files will remain outdated and hence most likely become invalid and useless over time.

Hence the change owners have prepared a one-time-off scriptlet that would clean the files on the first upgrade to the new configuration to prevent this from happening. Such scriptlet can be removed in Fedora 36 or sooner when Fedora upgrades to Python 3.10+.

How To Test

TBD

User Experience

Users will get python3-libs-bytecode-opt-0 by default and in most cases should not notice any difference except a smaller filesystem footprint.

Users executing Python with -O or -OO may notice slight slowdown of Python or Python applications startup and may need to install python3-libs-bytecode-opt-1/python3-libs-bytecode-opt-2 as needed.

Users of minimal systems where the maintainers removed python3-libs-bytecode-opt-0 may notice slight slowdown of Python or Python applications startup and may need to install python3-libs-bytecode-opt-0 if they are concerned. Alternatively, they may bytecompile only the relevant parts of the Python standard library ahead of time (e.g. when building derived container images). Users who prefer the bytecode not to be there may want to build their container images with PYTHONDONTWRITEBYTECODE environment variable set.

When the python3-libs-bytecode-opt-0/python3-libs-bytecode-opt-1/python3-libs-bytecode-opt-2 package is installed, it will override any runtime-generated bytecode cache.

When the python3-libs-bytecode-opt-0/python3-libs-bytecode-opt-1/python3-libs-bytecode-opt-2 package is uninstalled, RPM would leave all the files on disk because they are also owned by python3-libs -- we consider this a bad UX and hence we will add %postun scriptlets to remove the files.

Dependencies

N/A (not a System Wide Change)

Contingency Plan

  • Contingency mechanism: Abort, abort!
  • Contingency deadline: before the beta freeze
  • Blocks release? Not a chance
  • Blocks product? It plocks a broduct

Documentation

TBD

Release Notes

TBD

Retrieved from "https://fedoraproject.org/w/index.php?title=Changes/Python_Optional_Bytecode_Cache&oldid=587551"