User:Bkabrda/EPEL7 Python3

EPEL 7 in Python 3 Plan Draft

Why Python 3 in EPEL 7, Why not SCLs

Due to Fedora plans for migration to Python 3 [1], it is becoming necessary to have Python 3 accessible in some form in EPEL 7. This will allow upstream projects to move to Python 3 (dropping Python 2 support) and still be usable through EPEL repositories. Some key stakeholders in this regard are Fedora Infra and DNF.

Another important reason is further promotion of Python 3 and raising awareness for enterprise-level Python users.

Because of key stakeholders like Fedora Infra and DNF, SCLs as a solution to this problem were ruled out. SCLs are currently not allowed in EPEL (and it seems they won't be allowed in near future) and since these projects need/strongly prefer to be in EPEL itself, SCLs are not a viable alternative for them.

Some Facts

• Python 3 is actively developed by upstream and new minor versions are released approximately every year. Latest Python 3 minor is 3.4.
• Python minor versions are not ABI compatible, rebuilds are needed (both because of libpython soname change and because of bytecode magic numbers).
• Python has a good support for parallel installable stacks, even for minor versions of one major version (e.g. 3.3 and 3.4 can easily be installable alongside).
• EPEL policies try to discourage updates that break ABI and require rebuilds of other packages - due to the above fact, I consider Python minor version update a "major version update" as specified in EPEL update guidelines [2]. And as the guidelines say, this should be avoided if at all possible.

Proposal

• Python 3 is packaged as python3X parallel-installable package (X is minor version, e.g. "4" ATM) that can coexist with other parallel installable python3Y stacks.
• One of these stacks is always considered the "default" in the sense that it owns /usr/bin/python3.
• All binary RPMs of extension packages must be named python3X-foo. They must follow Fedora's python3 packaging guidelines with the exception that name contains both major and minor Python version (in Fedora it's only major). See below for packaging instructions.
• Most of the time, there is only one python3X stack in EPEL. During transitional periods, after 3X+1 is released, there are two.
• In a situation when python3X is in EPEL and 3.X+1 is released upstream, the following happens:
  • python3X+1 package is created for EPEL ASAP and all extension packages are built also for this new python3X+1 stack.
  • When all packages are rebuilt for python3X+1, the old python3X stack is retired after certain period. This period gives everyone enough time to rebuild their packages while being as short as possible. There is intentionally no hard limit here, we will approach this case by case.
  • "/usr/bin/python3" belongs to the "stable" python3X stack. Switching /usr/bin/python3 from python3X to python3X+1 happens shortly before the end of transitional period (== before obsoleting python3X) and it is announced on epel-devel.
  • Usage of "/usr/bin/python3" is discouraged in favour of using /usr/bin/python3.X explicitly.

Packaging Parallel python3X stacks

• Automatic dependency generators and bytecompilation hooks work fine.
• RPM naming:
  • As mentioned above, binary RPMs are named python3X for Python itself and python3X-foo for extension packages.
• SRPM naming, dist-git:
  • We're using Fedora-like dist-git repos:
    • If a python3-foo Fedora repo exists, it must be used.
    • Otherwise if python-foo Fedora repo exists, it must be used.
    • Otherwise the package doesn't exist in Fedora and packager should decide before review which of the above he'll choose to have created, according to Fedora Python Packaging Guidelines.
  • SRPM names are same as dist-git repo names. This means that we can have python-foo dist-git repo and python-foo SRPM which produces just python34-foo RPM, while not producing python-foo RPM.

Specfiles, Macros, Packaging Process

I experimented with several possibilities and this one seems to be the best:

%global with_python3 1
%if 0%{?rhel} > 6
%global python3_pkgversion_nodots 34
%global with_python3_other 1
%global python3_other_pkgversion_nodots 35
%else
%global python3_pkgversion_nodots 3
%endif

Name:           python-X
Version:        1
Release:        1%{?dist}
Summary:        X
Source:         X

License:        MIT
URL:            http://X
%if 0%{?with_python3}
# BR: python%{python3_pkgversion_nodots}-devel
# and other BR
%endif

%if 0%{?with_python3_other}
# BR: python%{python3_other_pkgversion_nodots}-devel
# and other BR
%endif

%description
X.

%if 0%{?with_python3}
%package -n python%{python3_pkgversion_nodots}-X
Summary: python%{python3_pkgversion_nodots} build of X

%description -n python%{python3_pkgversion_nodots}-X
python%{python3_pkgversion_nodots} build of X.
%endif

%if 0%{?with_python3_other}
%package -n python%{python3_other_pkgversion_nodots}-X
Summary: python%{python3_other_pkgversion_nodots} build of X

%description -n python%{python3_other_pkgversion_nodots}-X
python%{python3_other_pkgversion_nodots} build of X.
%endif

%prep
%setup -q -n X-%{version}

%if 0%{?with_python3}
# do what you do for Fedora's Python 3 stack
%endif

%if 0%{?with_python3_other}
# do the same as above, just with pytohn3_other macro set
%endif

%build
# do it like in Fedora, add with_python3_other

%install
# do it like in Fedora, add with_python3_other

%if 0%{?with_python3}
%files -n python%{python3_pkgversion_nodots}-X
%endif

%if 0%{?with_python3_other}
%files -n python%{python3_other_pkgversion_nodots}-X
%endif

%changelog

Explanation

For start, packagers will be able to merge specfile from Fedora and use it - all macros like %__python3 or %python3_sitelib will be provided by the current python3X-devel. It'll even be possible to use the same specfile for Fedora and EPEL with minimal effort. When merging specfiles from Fedora, packagers will need to:

• do initial Fedora import
• add conditionalized defintions of python3_pkgversion_nodots, with_other_python3 and python3_other_pkgversion_nodots (see the top of the specfile)
• change Fedora's python3-X subpacke name to python%{python3_pkgversion_nodots}-X
• add defintion and %prep, %build and %install steps of python%{python3_other_pkgversion_nodots}-X subpackage. which looks exactly the same as python%{python3_pkgversion_nodots}-X subpackage, it just uses different macros

Changing stacks when a new python3X+1 build is introduced or python3X is retired:

• during periods when only a single Python 3 runtime is in EPEL 7, packages will be built with %with_python3 enabled and with %with_python3_other disabled
• when a new python3X+1 is built in EPEL - let's say that there is python34 and python35 has just been introduced:
  • %with_python3_other is enabled in all packages, %python3_pkgversion_nodots is still set to 34, %python3_other_pkgversion_nodots is still set to 35
  • mass rebuild is run to build as much of the new stack possible automatically
  • at a certain point at time, annoucement is made that python34 is to be retired and python35 is to be *the* one - at this point, python34 and python35 will be rebuilt; python34-devel will provide the "other" macros from now on, while python35 will provide the "default" macros (note that all the specfiles still build just fine as they are)
  • some time after python34 and python35 are rebuilt, %python3_pkgversion_nodots is set to "35", %python3_other_pkgversion_nodots is set to "36" and %with_python3_other is disabled
  • no rebuild is needed at this point, but all packages build just python35-X subpackages
  • now the whole python34 stack is disabled

All the painful parts in the above process can be done automatically (enabling/disabling "with" macros, changing values of "pkgversion" macros, mass rebuilding).

subpackages for Python 2 already in RHEL
Some packages imported from Fedora, for example setuptools, already have their Python 2 versions built in RHEL and these must not be replaced/updated by EPEL packages. This means that for these packages, the packager will either need to remove the python 2 build parts from the specfile or he'll need to conditionalize them to be only built in Fedora.

[1] http://fedoraproject.org/wiki/Changes/Python_3_as_Default [2] https://fedoraproject.org/wiki/EPEL/GuidelinesAndPolicies#A_major_version_update