User:Bkabrda/EPEL7 Python3

EPEL 7 in Python 3 Plan Draft

Why Python 3 in EPEL 7, Why not SCLs

Due to Fedora plans for migration to Python 3 [1], it is becoming necessary to have Python 3 accessible in some form in EPEL 7. This will allow upstream projects to move to Python 3 (dropping Python 2 support) and still be usable through EPEL repositories. Some key stakeholders in this regard are Fedora Infra and DNF.

Another important reason is further promotion of Python 3 and raising awareness for enterprise-level Python users.

Because of key stakeholders like Fedora Infra and DNF, SCLs as a solution to this problem were ruled out. SCLs are currently not allowed in EPEL (and it seems they won't be allowed in near future) and since these projects need/strongly prefer to be in EPEL itself, SCLs are not a viable alternative for them.

Some Facts

• Python 3 is actively developed by upstream and new minor versions are released approximately every year. Latest Python 3 minor is 3.4.
• Python minor versions are not ABI compatible, rebuilds are needed (both because of libpython soname change and because of bytecode magic numbers).
• Python has a good support for parallel installable stacks, even for minor versions of one major version (e.g. 3.3 and 3.4 can easily be installable alongside).
• EPEL policies try to discourage updates that break ABI and require rebuilds of other packages - due to the above fact, I consider Python minor version update a "major version update" as specified in EPEL update guidelines [2]. And as the guidelines say, this should be avoided if at all possible.

Proposal

• We will package Python 3 as python3X parallel-installable package (X is minor version, e.g. "4" ATM).
• All extension packages will be named python3X-foo. They will follow Fedora's python3 packaging guidelines with the exception that name contains both major and minor Python version (in Fedora it's only major).
• The whole python3X stack will be parallel installable with any other python3Y stack.
• In a situation when python3X is in EPEL and 3.X+1 is released upstream, the following should happen:
  • python3X+1 package will be created for EPEL ASAP and all extension packages should be built also for this new python3X+1 stack. (While the python3X stack would stay untouched the whole time.)
  • When all packages are rebuilt for python3X+1, the old python3X stack will be retired after some amount of time (TODO: how long?, TODO: would this align with the EPIC proposal somehow?).
  • "/usr/bin/python3" will belong to the "stable" python3X stack, hence it should be only changed to point to the new stack once the old is obsoleted. Usage of "/usr/bin/python3" should be discouraged in favour of using /usr/bin/python3.X explicitly (TODO: at which point should "/usr/bin/python3" be switched from python3X to python3X+1?)

Packaging Parallel python3X stacks

• Automatic dependency generators and bytecompilation hooks should do fine.
• SRPM/RPM naming:
  • As mentioned above, binary RPMs will be named python3X for Python itself and python3X-foo for extension packages.
  • Source RPMs should also be named python3X-foo.srpm, although we could create python3-foo.srpm and build python3X-foo.rpm and python3X+1-foo.rpm out of it. Doing the latter is pretty similar to what we're doing in Fedora with parallel python2/python3 stacks and may be easier to maintain. (We could even create just python-foo.srpm, but that might be confusing, since the same SRPMs could exist in RHEL/CentOS 7, which are used for the Python 2 RHEL 7 stack.)
• Dist-git possibilities:
  • Having python3X-foo dist-git repo for every package. Easy and obvious solution, but packages would have to be re-reviewed and new repos created with every Python 3 minor release. This would align with the first SRPM variant (python3X-foo SRPMS).
  • Using current dist-git repos, e.g. branch epel-7 in python-foo (or python3-foo, if such Fedora dist-git exists). With this approach, we'd create python3-foo.srpm and build python3X and possibly python3X+1 RPMs out of it. The disadvantage is that some packages would be in python-foo and others in python3-foo dist git repos, which could get confusing. Also, their packaging could in time diverge a lot, so it might not make sense to do this.
  • Or we could do the above, but create special epel7-python3-foo dist-git repos to keep things more organized and separated from standard Python repos (I'd prefer this).

Specfiles, Macros, Packaging Process

I experimented with several possibilities and this one seems to be the best:

%global with_python3 1
%if 0%{?rhel} > 6
%global python3_pkgversion_nodots 34
%global with_python3_other 1
%global python3_other_pkgversion_nodots 35
%else
%global python3_pkgversion_nodots 3
%endif

Name:           python-X
Version:        1
Release:        1%{?dist}
Summary:        X
Source:         X

License:        MIT
URL:            http://X
%if 0%{?with_python3}
# BR: python%{python3_pkgversion_nodots}-devel
# and other BR
%endif

%if 0%{?with_python3_other}
# BR: python%{python3_other_pkgversion_nodots}-devel
# and other BR
%endif

%description
X.

%if 0%{?with_python3}
%package -n python%{python3_pkgversion_nodots}-X
Summary: python%{python3_pkgversion_nodots} build of X

%description -n python%{python3_pkgversion_nodots}-X
python%{python3_pkgversion_nodots} build of X.
%endif

%if 0%{?with_python3_other}
%package -n python%{python3_other_pkgversion_nodots}-X
Summary: python%{python3_other_pkgversion_nodots} build of X

%description -n python%{python3_other_pkgversion_nodots}-X
python%{python3_other_pkgversion_nodots} build of X.
%endif

%prep
%setup -q -n X-%{version}

%if 0%{?with_python3}
# do what you do for Fedora's Python 3 stack
%endif

%if 0%{?with_python3_other}
# do the same as above, just with pytohn3_other macro set
%endif

%build
# do it like in Fedora, add with_python3_other

%install
# do it like in Fedora, add with_python3_other

%if 0%{?with_python3}
%files -n python%{python3_pkgversion_nodots}-X
%endif

%if 0%{?with_python3_other}
%files -n python%{python3_other_pkgversion_nodots}-X
%endif

%changelog

Explanation

For start, packagers will be able to merge specfile from Fedora and use it - all macros like %__python3 or %python3_sitelib will be provided by the current python3X-devel. It'll even be possible to use the same specfile for Fedora and EPEL with minimal effort. When merging specfiles from Fedora, packagers will need to:

• do initial Fedora import
• add conditionalized defintions of python3_pkgversion_nodots, with_other_python3 and python3_other_pkgversion_nodots (see the top of the specfile)
• change Fedora's python3-X subpacke name to python%{python3_pkgversion_nodots}-X
• add defintion and %prep, %build and %install steps of python%{python3_other_pkgversion_nodots}-X subpackage. which looks exactly the same as python%{python3_pkgversion_nodots}-X subpackage, it just uses different macros

Changing stacks when a new python3X+1 build is introduced or python3X is retired:

• during periods when only a single Python 3 runtime is in EPEL 7, packages will be built with %with_python3 enabled and with %with_python3_other disabled
• when a new python3X+1 is built in EPEL - let's say that there is python34 and python35 has just been introduced:
  • %with_python3_other is enabled in all packages, %python3_pkgversion_nodots is still set to 34, %python3_other_pkgversion_nodots is still set to 35
  • mass rebuild is run to build as much of the new stack possible automatically
  • at a certain point at time, annoucement is made that python34 is to be retired and python35 is to be *the* one - at this point, python34 and python35 will be rebuilt; python34-devel will provide the "other" macros from now on, while python35 will provide the "default" macros (note that all the specfiles still build just fine as they are)
  • some time after python34 and python35 are rebuilt, %python3_pkgversion_nodots is set to "35", %python3_other_pkgversion_nodots is set to "36" and %with_python3_other is disabled
  • no rebuild is needed at this point, but all packages build just python35-X subpackages
  • now the whole python34 stack is disabled

All the painful parts in the above process can be done automatically (enabling/disabling "with" macros, changing values of "pkgversion" macros, mass rebuilding).

subpackages for Python 2 already in RHEL
Some packages imported from Fedora, for example setuptools, already have their Python 2 versions built in RHEL and these must not be replaced/updated by EPEL packages. This means that for these packages, the packager will either need to remove the python 2 build parts from the specfile or he'll need to conditionalize them to be only built in Fedora.

[1] http://fedoraproject.org/wiki/Changes/Python_3_as_Default [2] https://fedoraproject.org/wiki/EPEL/GuidelinesAndPolicies#A_major_version_update