User:Rkuska/Py2Py3Packaging

Prelude

This page summarizes a proposal of changes to Python packaging guidelines, that should occur with switch to Python 3 as a default, as proposed in [1] (will be referred to as Change).

For further reference we need to *distinguish* between python *modules*, python *applications* and python *applications exceptions*.

• APPLICATIONS

Application foo is not meant to be used within others python libraries via import foo and Both python3 and python2 versions of foo provides same functionality and therefore only one version is needed. This also includes scripts. DevAssistant is an *application* - We invoke DA and we don't care if it is python2 and python3 based, both will fulfill our task.

• APPLICATIONS EXCEPTIONS

On the other hand, pip isn't application even hough it s not meant to be used via import statement because both python3 and python2 versions provides different functionality (python-pip installs python2 packages and python3-pip installs python3 packages), therefore it is a *exception*.

• MODULES

Modules are meant to be used via import statement in other python libraries and therefore both python2 and python3 (and any others future majors) versions are needed.

Applications exceptions and modules share the same packaging guidelines, bare applications differs in naming policy and also macros usage.

Changes in Package Naming and SRPM <-> RPM Relations

Current State

python-* packages (built with Python 2) are installed by default (from LiveCD, as dependencies of applications that require some Python packages) and by yum install python-foo. Python 3 packages are named python3-* and are installed by yum install python3-foo (this of course doesn't apply for applications as stated above).

From the packaging point of view, this is achieved by either

• Having one SRPM python-foo that produces python-foo and python3-foo binary RPMs.

or

• Having two SRPMs, python-foo (produces python-foo binary RPM) and python3-foo SRPM (produces python3-foo binary RPM).

and for applications:

• Having SRPM foo that produces foo binary RPM where python major version dependency is decided by packager unless it is not crucial package shipped on liveCD and than it is built by python2.

Future State Implied by the Change

After the switch, python3-* packages are going to be installed by default (from LiveCD, as dependencies of applications that require some Python packages).

%{__python} macro will point to /usr/bin/python3 [6].

Proposal for Further Changes

• A brief discussion already happened at [2] and [5].
• SRPMs that produce apllications binary RPM will stay intact by naming changes
• SRPMs that produce binary RPMs for more Python runtimes should keep current names (python-foo).
• SRPMs that produce binary RPM for just one runtime should be named pythonX-foo.
• Every binary RPM (except for applications) must be named pythonX-foo, where X is the major version of the Python runtime that it uses.
• For the time being (and maybe forever), every python2-foo package must have a virtual provide for its previous name. For most of the packages, this will be Provides: python-foo, but some of the packages haven't had the python- prefix historically, so these will just have Provides: foo (for example Provides: PyYAML).
• From packaging perspective, the above also means that all the python2-foo packages will need to have a proper Obsoletes: ... tag.
• From user's perspective this means:
  • python-* packages will be renamed to python2-*, but will still keep the provide of python-* (so yum install python-foo will still work).
  • Packages that didn't have the python- prefix, e.g. pyfoo, will newly be named python2-pyfoo, but they will still provide the old name (so again, yum install pyfoo will still work).
• Propose new macro for easier building of applications %{py_major_default} which value in f23 will be '3'
• Propose bin files naming policy
  • Application foo provides unversioned bin file(s)
  • Modules and applications exceptions provides bin file and versioned bin files in format:
    • foo - python2 version of bin file, shipped only with python2-foo
  • Unless upstream isn't using any specific versioning:
    • foo-X - where X is a major version of python with which was binary built, this is provided by all subpackages
    • foo-X.Y - where Y is a minor version of python with which was binary built , this is provided by all subpackages
    • foo-(X|X.Y)-vZ - where Z is version of compat package for compat packages, this is provided by all subpackages

Why?

• This concept will easily scale to more Python runtimes (PyPy, Jython), as already proposed previously by Tom Spur. Please note, that scaling to other runtimes is not part of this proposal and should be discussed further in another proposal/ML thread.
• Currently, upstream recommendation [3] is to point /usr/bin/python to Python 2, but as the PEP notes, it will be reviewed and it is anticipated that in time it will be updated to recommend pointing /usr/bin/python to /usr/bin/python3. When this time comes, we should also move the provides suggested above (Provides: python-foo) from python2-* packages to python3-* packages to keep things like yum install /usr/bin/python python-foo consistent.
• Having python2-* vs. python3-* packages is a good way of explicitly distinguishing packages from the two stacks. This is connected with the reason above - we should recommend explicit usage of /usr/bin/python{2,3} and yum install python{2,3}-foo, and we should discourage use of implicit /usr/bin/python and python-* so that nothing breaks for users when these are changed in any way. (We already started to advertise usage of versioned /usr/bin/python{2,3} binary by deprecating %__python in current guidelines and recommending %__python{2,3} instead.)

Alternatives

• Just keep what we have - doesn't seem to scale to other Python runtimes; doesn't allow explicit naming (python2-foo vs. python3-foo) while keeping the possibility to move Provides: python-foo to python3-foo in the future.
• Only allow split SRPMs - seems to be too much maintenance work.

Example Specfile

Example of what a specfile (for modules and applications exceptions) would look like follows. Everything works in the same way as it does now with building python3-* subpackages; the same approach is applied to python2-* subpackages.

This specfile will produce two binary RPMs, python2-six and python3-six. Any of them can be disabled by setting with_python{2,3} to 0:

%global with_python2 1
%global with_python3 1

# this macro is defined here only for testing purposes, it would
# be defined in macros.python2 provided by python2-devel
%global py2dir %{_builddir}/python2-%{name}-%{version}-%{release}

Name:           python-six
Version:        1.4.1
Release:        1%{?dist}
Summary:        Python 2 and 3 compatibility utilities

Group:          Development/Languages
License:        MIT
URL:            http://pypi.python.org/pypi/six/
Source0:        http://pypi.python.org/packages/source/s/six/six-%{version}.tar.gz

BuildArch:      noarch
%if 0%{?with_python2}
BuildRequires:  python2-devel
# For use by selftests:
BuildRequires:  python2-pytest
BuildRequires:  python2-tkinter
%endif

%if 0%{?with_python3}
BuildRequires:  python3-devel
# For use by selftests:
BuildRequires:  python3-pytest
BuildRequires:  python3-tkinter
%endif

Provide descriptions for both packages (and for the main package, since rpmbuild enforces that).

%description
python-six provides simple utilities for wrapping over differences between
Python 2 and Python 3.

%if 0%{?with_python2}
%package -n python2-six
Summary:        Python 2 and 3 compatibility utilities
Group:          Development/Languages
Provides:       python-six = %{version}-%{release}
Obsoletes:      python-six < 1.4.1-1

%description -n python2-six
python-six provides simple utilities for wrapping over differences between
Python 2 and Python 3.
%endif

%if 0%{?with_python3}
%package -n python3-six
Summary:        Python 2 and 3 compatibility utilities
Group:          Development/Languages

%description -n python3-six
python-six provides simple utilities for wrapping over differences between
Python 2 and Python 3.

This is the Python 3 build of the module.
%endif

%prep, %build, %install and %check sections look pretty much the same, each has to explictly switch the directory to the respective py{2,3}dir. I believe we could come up with some macros that would make this easier and more readable [7].

%prep
%setup -q -n six-%{version}
# possibly apply patches here

%if 0%{?with_python2}
rm -rf %{py2dir}
cp -a . %{py2dir}
%endif

%if 0%{?with_python3}
rm -rf %{py3dir}
cp -a . %{py3dir}
%endif


%build
%if 0%{?with_python2}
pushd %{py2dir}
%{__python2} setup.py build
popd
%endif

%if 0%{?with_python3}
pushd %{py3dir}
%{__python3} setup.py build
popd
%endif


%install
%if 0%{?with_python3}
pushd %{py3dir}
%{__python3} setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT
popd
%endif

%if 0%{?with_python2}
pushd %{py2dir}
%{__python2} setup.py install -O1 --skip-build --root $RPM_BUILD_ROOT
popd
%endif

There is no default %files section, only %files sections for python{2,3}-six packages.

%if 0%{?with_python2}
%files -n python2-six
%doc LICENSE README documentation/index.rst
%{python2_sitelib}/*
%endif

%if 0%{?with_python3}
%files -n python3-six
%doc LICENSE README documentation/index.rst
%{python3_sitelib}/*
%endif

As application are using only one version of python and this version should be the default python as defined by distro (python3 in f23) packagers may use the unversioned macros with %{py_major_default} so they can use the same specfile across different fedora versions and also epel.

Name:           fooapplication
Version:        0.6.10
Release:        2%{?dist}
Summary:        Foo this

Group:          Applications/System
License:        GPLv2
URL:            http://pypi.python.org/pypi/%{srcname}
Source0:        http://pypi.python.org/packages/source/d/%{srcname}/%{srcname}-%{version}.tar.gz

BuildArch:      noarch
BuildRequires:  python%{?py_default_major}-devel

In fedora23 this will be a python3-devel and in fedora<23 and epel this will be a python-devel.

%build
CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py build

%install
%{__python} setup.py install --skip-build --root %{buildroot}

In fedora23 this will be built and install by /usr/bin/python3 and in fedora<23 and epel with /usr/bin/python2. Same logic applies for %{python_site*} macros. This way packagers may keep the same specfile across different versions. Of course this is just a possibility for packagers (to make their life easier) it is not a 'must follow' instruction. Also, important to mention is that %{py_default_major} macro should be part of the buildroot macros.

[1] https://fedoraproject.org/wiki/Changes/Python_3_as_Default

[2] https://lists.fedoraproject.org/pipermail/devel/2013-July/186822.html

[3] http://www.python.org/dev/peps/pep-0394/

[4] http://www.python.org/dev/peps/pep-0394/#future-changes-to-this-recommendation

[5] https://lists.fedoraproject.org/pipermail/python-devel/2015-April/000693.html

[6] https://fedorahosted.org/fpc/ticket/498

[7] https://fedorahosted.org/fpc/ticket/281