From Fedora Project Wiki
mNo edit summary
(scope)
Line 169: Line 169:


== Scope ==
== Scope ==
* Proposal owners:
* Proposal owners: make it work technically, propose the new guidelines
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Other developers: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Other developers: may opt-in for the new behavior or explicitly stick with the old one (not a System Wide Change, they don't have to do anything) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


Line 181: Line 181:
<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings -->
<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings -->


* Policies and guidelines: N/A (not a System Wide Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
* Policies and guidelines: will be changed as described in description <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->


* Trademark approval: N/A (not needed for this Change)
* Trademark approval: not needed
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->
<!-- If your Change may require trademark approval (for example, if it is a new Spin), file a ticket ( https://fedorahosted.org/council/ ) requesting trademark approval from the Fedora Council. This approval will be done via the Council's consensus-based process. -->



Revision as of 10:21, 2 February 2018

Incomplete
This page is incomplete!


No more automagic Python bytecompilation

Summary

The current way of automatic Python byte compiling of files outside of Python specific directories is too magical and error prone. It is built on false premises that might have been logical years ago. We will provide a way to opt-out of it and adjust the guidelines to prefer the new way of handling the bytecompilation of such files. Later the old behavior will be opt-in only or will cease to exist.

Owner

  • Name: Miro Hrončok
  • Email: mhroncok@redhat.com
  • Release notes owner:

Current status

  • Targeted release: Fedora 29
  • Last updated: 2018-02-02
  • Tracker bug: <will be assigned by the Wrangler>

Detailed Description

Status quo

As for Fedora 28, all *.py files outside of the /usr/lib(64)?/pythonX.Y/ directories are bytecompiled by %{__python} (/usr/bin/python). This is overly magical and assumes several things (not always right):

  • all files named *.py are Python modules that need to be bytecompiled
  • when a file is not in /usr/lib(64)?/pythonX.Y/ it is intended for the /usr/bin/python interpreter
    • That is currently Python 2, but may be removed or changed to Python 3 at any point in the future.
    • This is only the default behavior, it can be changed by redefining %__python to:
      • python3: that currently happens in various packages and is documented in the guidelines as a way to do it
      • python2: nobody does that, because the magic "just works" for this use case - relying on the fact that /usr/bin/python is currently python2
        • that assumption is forbidden by the Python packaging guidelines, yet here it is heavily used

See Packaging:Python Appendix for more information (this links to a specific revision so the link makes sense once this change is implemented and the guidelines are changed).

The current behavior is magical. Mistakes are made. Things are done or not done based on the presence of /usr/bin/python. See a simple example of a package that builds fine without /usr/bin/python in the buildroot but fails when it's there.

Name:           reproducer
Version:        0.1
Release:        1%{?dist}
Summary:        Reproducer for a bytecompile script issue
License:        MIT
BuildArch:      noarch

%description
This package will build fine if /usr/bin/python is *not* in the buildroot.

%prep
echo "Poland" > country-name.pl
echo "Paraguay" > country-name.py
echo "Saint Helena" > country-name.sh
echo "Serbia" > country-name.rs

%build

%install
mkdir -p %{buildroot}%{_datadir}/%{name}
cp country-name.* %{buildroot}%{_datadir}/%{name}

%files
%dir %{_datadir}/%{name}/
%{_datadir}/%{name}/country-name.??


How we are changing it

For the time being, we keep the old behavior working.

A opt-out mechanism for this automagic compilation will be provided (such as %?disable_automagic_pybytecompile). This will only opt-out from the compilation of files outside of /usr/lib(64)?/pythonX.Y/. Speaking code, this will disable the script from this point forward.

Guidelines will be adjusted to say the following:

  • if you have *.py files outside of the /usr/lib(64)?/pythonX.Y/, you MUST disable the automagic and compile them explicitly
  • explicit compilation is done by the %py_byte_compile macro

Example for package that has both Python versions:

# Turn off the brp-python-bytecompile automagic
%?disable_automagic_pybytecompile

# Buildrequire both python2 and python3
BuildRequires: python2-devel python3-devel

%install
# Installs a python2 private module into %{buildroot}%{_datadir}/mypackage/foo
# and installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
make install DESTDIR=%{buildroot}

# Manually invoke the python byte compile macro for each path that needs byte
# compilation.
%py_byte_compile %{__python2} %{buildroot}%{_datadir}/mypackage/foo
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar

Note that unlike the current example in the guidelines linked above, this does not disable the compilation of files in /usr/lib(64)?/pythonX.Y/.

Example for Python 3 only:

# Turn off the brp-python-bytecompile automagic
%?disable_automagic_pybytecompile

BuildRequires: python3-devel

%install
# Installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
make install DESTDIR=%{buildroot}

# Manually invoke the python byte compile macro for each path that needs byte
# compilation.
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar

The Python 2 only example is analogical.

Currently, %py_byte_compile lives in python3-devel, we'll move it to some generic package (such as python-rpm-macros ).

Analogically, we'll also provide %?enable_automagic_pybytecompile for packagers to explicitly say they rely on the current behavior. Later (i.e. not in Fedora 29, but approximately when /usr/bin/python stops being python2), we'll make the old behavior opt-in (or disbale it entirely if no package uses %?enable_automagic_pybytecompile).

Benefit to Fedora

More explicit specfiles when it comes to Python byte compilation. This will ease the change once we decide /usr/bin/python is no longer python2. The new guidelines will be less error prone. Note that we'd prefer to switch to the new behavior right now, but we keeping it opt-in not to break the ~500 packages that use it.


Scope

  • Proposal owners: make it work technically, propose the new guidelines
  • Other developers: may opt-in for the new behavior or explicitly stick with the old one (not a System Wide Change, they don't have to do anything)
  • Policies and guidelines: will be changed as described in description
  • Trademark approval: not needed

Upgrade/compatibility impact

N/A (not a System Wide Change)

How To Test

N/A (not a System Wide Change)

User Experience

N/A (not a System Wide Change)

Dependencies

N/A (not a System Wide Change)

Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
  • Contingency deadline: N/A (not a System Wide Change)
  • Blocks release? N/A (not a System Wide Change), Yes/No
  • Blocks product? product

Documentation

N/A (not a System Wide Change)

Release Notes