User:Marbu/PackagingPythonGuidelines

This is not an official wikipage for Python packaging in Fedora. It's just a draft of proposed changes. For more details, see: FPC Ticket #598

Python Version Support

In Fedora we have multiple Python runtimes, one for each supported major Python release. At this point that's one for python2.x and one for python3.x. If a piece of software supports python3, it must be packaged for python3. If it supports python2 as well, it may be packaged for python2. If it supports only python2 then (obviously) it must not be packaged for python3.

Multiple Python Runtimes

Each runtime corresponds to a binary of the form /usr/bin/python$MAJOR.$MINOR. One of these python runtimes is the "system runtime" which is what is run when invoking /usr/bin/python. On Fedora 22, for example, this is a link to /usr/bin/python2 (which itself is a link to /usr/bin/python2.7).

However, packages in Fedora should not depend on where /usr/bin/python happens to point but instead should call the proper executable for the needed python major version directly, either /usr/bin/python2 or /usr/bin/python3 as appropriate.

All python runtimes have a virtual provide for python(abi) = $MAJOR-$MINOR. For example, the python-3.4 runtime rpm has:

 $ rpm -q --provides python3 |grep -i abi
 python(abi) = 3.4

python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with. This is done automatically for files below /usr/lib[^/]*/python${PYVER}

Mirroring policy for regular packages, the Python-version-specific subpackages of your package most not be removed in a release branch of Fedora.

BuildRequires

Packages building for python2 will need BuildRequires: python2-devel. Packages building for python3 will need BuildRequires: python3-devel. Packages building for for both will need build dependencies on both.

Provides

Using a fictional module named "example", the subpackage containing the python2 version must provide python2-example. This is of course always the case if the subpackage is named python2-example (as in the examples below). If the subpackage has some other name then then Provides: python2-example must be added explicitly (but see the %python_provide macro below).

The python3 subpackage must provide python3-example. However, as the naming guidelines mandate that the python3 subpackage be named python3-example, this will happen automatically.

The %python_provide macro

In addition, the subpackage corresponding to the current system runtime must provide Provides: python-example. However, the system version of python in Fedora will almost certainly change at some point in the future.

To avoid having to alter every python module package when this change happens, and to allow the same spec to build for releases with different system versions of python, a convenient %python_provide macro is provided. When given (as an argument) the name of the current subpackage, it will evaluate to either the proper Provide: line or nothing at all as appropriate for the version of the system runtime. Thus if the package follows the suggested subpackage naming scheme as shown in the below example, placing this macro in the definition of each subpackage will guarantee that the necessary Provide: line appears where it should, and does not appear where it shouldn't.

Finally, if the %python_provide is called with python-example (no python major version in the name) then it will evaluate to either Provides: python2-example or Provides: python3-example depending the version of the current system runtime.

Macros

In Fedora 22 and newer, as well as RHEL7, the following macros are defined for you:

Macro	Normal Definition	Notes
__python	%{__python2}	Default Python interpreter. Currently links to Python2 interpreter
__python2	/usr/bin/python2	Python 2 interpreter. Also the current default python interpreter
__python3	/usr/bin/python3	Python 3 interpreter
python_provide	(Lua script)	Given a package name, evaluates to either Provides: python-example or nothing at all depending on the version of the system runtime. See here for an example.
python_sitelib	%{python2_sitelib}	Where pure python modules are installed for the default Python implementation
python_sitearch	%{python2_sitearch}	Where python extension modules that are compiled C are installed for the default Python implementation
python2_sitelib	/usr/lib/python2.X/site-packages	Where pure python2 modules are installed
python2_sitearch	/usr/lib64/python2.X/site-packages on x86_64 /usr/lib/python2.X/site-packages on x86	Where python2 extension modules that are compiled C are installed
python3_sitelib	/usr/lib/python3.X/site-packages	Where pure python3 modules are installed
python3_sitearch	/usr/lib64/python3.X/site-packages on x86_64 /usr/lib/python3.X/site-packages on x86	Where python3 extension modules that are compiled C are installed
py_byte_compile	(script)	Defined in python3-devel. See the byte compiling section for usage
python3_version	3.X	Defined in python3-devel. Useful when running programs with Python version in filename, such as nosetest-%{python3_version}
python3_version_nodots	3X	Defined in python3-devel since Fedora 21 / Python 3.4. Useful when listing files explicitly in %files section , such as %{python3_sitelib}/foo/*.cpython-%{python3_version_nodots}.pyo

The generic %{_python} macros
The unversioned macros, %{__python}, %{python_sitelib}, and %{python_sitearch} are generic macros that will always point to the default implementation. You may only use them with applications that need to choose to use the system's default version of python. (Currently this is the python2 interpreter.)

This is future proofing for the time when things will be switched over to python3 by default instead of python2.

You should use %{__python2}, %{python2_sitelib}, and %{python2_sitearch} to explicitly reference the python2 interpreter instead.

During %install or when listing %files you can use the python2_sitearch and python2_sitelib macros to specify where the installed modules are to be found. For instance:

%files
# A pure python2 module
%{python2_sitelib}/foomodule/
# A compiled python2 extension module
%{python2_sitearch}/barmodule/
# A compiled python3 extension module
%{python3_sitearch}/bazmodule/

Use of the macros has several benefits:

• It ensures that the packages are installed correctly on multilib architectures.
• Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if python2_sitelib moves into %{_datadir}).

Files to include

When packaging python modules, several types of files are included:

• *.py source files because they are used when generating tracebacks.
• *.pyc and *.pyo byte compiled files (and, if present, the enclosing __pycache__ directory in most cases).
  • Python will try to create them at runtime if they don't exist which leads to spurious SELinux AVC denials in the logs.
  • If the system administrator invokes python with -OO, .pyos will be created with no docstrings. This can break some programs.
• *.egg-info files or directories. If these are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at runtime.

The source files must be included in the same package as the byte compiled versions.

Byte compiling

Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run. These files are saved in files with the extension of .pyc (compiled python) or .pyo (optimized compiled python). With current versions of python 3, these files will be located inside a directory named __pycache__. Older versions of python will simply place them alongside the .py files.

The .pyc and .pyo files contain byte code that is portable across OSes. If you do not include them in your packages, python will try (and generally fail) to create them when the user runs the program. If the system administrator runs the program, then the files will be successfully written, causing stray .pyc and .pyo files which will not be removed when the package is removed. To prevent that the byte compiled files need to be compiled and included in the %files section. Normally, byte compilation is done for you by the brp-python-bytecompile script. This script runs after the %install section of the spec file has been processed and byte compiles any .py files that it finds (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the %{buildroot} in them).

You must include in your package the .pyc and .pyo files. If the build process creates a __pycache__ directory in a subdirectory of  %{python3_sitearch} or %{python3_sitelib}, you must also include all items in the __pycache__ directory. (You should not include the directories %{python3_sitearch}/__pycache__ or %{python3_sitelib}/__pycache__ because they are already owned by the python3-libs package.)

All that you need to do is include the files in the %files section (replacing %{python3_sitelib} with the appropriate macro for your package):

%files
%{python3_sitelib}/foo/

or, if the python code installs directly into %{python3_sitelib}:

%files
%{python3_sitelib}/foo.py
%{python3_sitelib}/__pycache__/*

Avoid INSTALLED_FILES
python's distutils has an INSTALLED_FILES feature that lists which files are installed when you run %py_install. Do not use it for packaging as that will not list the directories which need to be specified in the %files section as well. Using globs in the %files section is simpler and safer.

Including egg info
When you run %py_install in any current Fedora, distutils generates a .egg-info file with metadata about the python module that is installed. These files need to be included as well. (See Packaging:Python_Eggs )

Optimization

Fedora packages running with python < 3.5 (including any version of python 2) must not invoke python with the -OO option or set the environment variable PYTHONOPTIMIZE to 2 or greater. (Using -O or PYTHONOPTIMIZE less than 2 is fine, though unnecessary.)

Similarly, any .pyo shipped in a Fedora package for python < 3.5 must not have been byte compiled using optimization level 2 or higher.

Manual byte compilation

For more details on the internals of byte compilation, please see the appendix.

Common SRPM vs split SRPMs

Many times when you package a python module you will want to create a module for python2 and a module for python3. Both versions should be built from the same SRPM. An exception to this would be if the two versions are distributed as separate archives and do not follow the same release schedule.

Use of the %python_provide macro
When building more than once from the same spec file, you must not have a %files section. Instead you must build one subpackage for each python runtime and use the %python_provide macro as shown in the example specfile below.

Example common spec file

The following is a very simple spec file for a module building for both python2 and python3. It builds both versions in the same directory; this is possible because the build products for different versions of Python usually do not conflict.

There are cases where it is not possible to build in a single directory. Most commonly this happens when the sources are modified during the build process to convert them from python2 to python3 using the the 2to3 tool. In that case, please see the appendix.

As you can see in the %install section below, the order in which you do the python2 versus python3 install can sometimes matter. You need to be aware of when the install is writing to the same file in both packages (in this example, a script in %{_bindir} and make sure that you're getting the version you expect.

%global srcname example
%global sum An example python module

Name:           python-%{srcname}
Version:        1.2.3
Release:        1%{?dist}
Summary:        %{sum}

License:        MIT
URL:            http://pypi.python.org/pypi/%{srcname}
Source0:        http://pypi.python.org/packages/source/e/%{srcname}/%{srcname}-%{version}.tar.gz

BuildArch:      noarch
BuildRequires:  python2-devel python3-devel

%description
An python module which provides a convenient example.

%package -n python2-%{srcname}
Summary:        %{sum}
%{?python_provide:%python_provide python2-%{srcname}}

%description -n python2-%{srcname}
An python module which provides a convenient example.


%package -n python3-%{srcname}
Summary:        %{sum}
%{?python_provide:%python_provide python3-%{srcname}}

%description -n python3-%{srcname}
An python module which provides a convenient example.


%prep
%autosetup -n %{srcname}-%{version}

%build
%py2_build
%py3_build

%install
# Must do the python2 install first because the scripts in /usr/bin are
# overwritten with every setup.py install, and in general we want the
# python3 version to be the default.
%py2_install
%py3_install

%check
%{__python2} setup.py test
%{__python3} setup.py test

# Note that there is no %%files section for the unversioned python module if we are building for several python runtimes
%files -n python2-%{srcname}
%license COPYING
%doc README.rst
%{python2_sitelib}/*
%{_bindir}/sample-exec-2.7

%files -n python3-%{srcname}
%license COPYING
%doc README.rst
%{python3_sitelib}/*
%{_bindir}/sample-exec
%{_bindir}/sample-exec-3.4

%changelog

Avoiding collisions between the python 2 and python 3 stacks

The python 2 and python 3 stacks are intended to be fully-installable in parallel. When generalizing the package for both python 2 and python 3, it is important to ensure that two different built packages do not attempt to place different payloads into the same path.

Executables in /usr/bin

Many existing python packages install executables into /usr/bin.

For example if we have a console_scripts in a setup.py shared between python 2 and python 3 builds: these will spit out files in /usr/bin/, and these will collide.

For example python-coverage has a setup.py that contains:

    entry_points = {
        'console_scripts': [
            'coverage = coverage:main',
            ]
        },

which thus generates a /usr/bin/coverage executable (this is a python script that runs another python script whilst generating code-coverage information on the latter).

Similarly for the 'scripts' clause; see e.g. python-pygments: Pygments-1.1.1/setup.py has:

    scripts = ['pygmentize'],

which generates a /usr/bin/pygmentize (this is a python script that leverages the pygments syntax-highlighting module, giving a simple command-line interface for generating syntax-highlighted files)

If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only one version of the executable should be packaged. On releases up to and including F21, this was the python 2 implementation. Python3 should be used in F22 and later if supported by upstream. Be sure to test the new implementation. Transitioning from python2 to python3 is left to individual package maintainers except for packages in Fedora's critical path. For these, we want to port to python3 versions in the same Fedora release if possible.

Examples of this:

• /usr/bin/pygmentize ought to generate the same output regardless of whether it's implemented via Python 2 or Python 3, so only one version needs to be shipped.

If the executables provide different functionality for Python 2 and Python 3, then both versions should be packaged.

Examples of this:

• /usr/bin/coverage runs a python script, augmenting the interpreter with code-coverage information. Given that the interpreter itself is the thing being worked with, it's reasonable to package both versions of the executable.
• /usr/bin/bpython augments the interpreter with a "curses" interface. Again, it's reasonable to package both versions of this.
• /usr/bin/easy_install installs a module into one of the Python runtimes: we need a version for each runtime.

As an exception, for the packages that are part of a python runtime itself, we plan to package both versions of the executables, so that e.g. both the python 2 and python 3 versions of 2to3 are packaged.

Naming

Many executables already contain a "-MAJOR.MINOR" suffix, for example /usr/bin/easy_install-3.4. These obviously can be used as-is, as they won't conflict.

For other executables, the general rule is:

• If only one executable is to be shipped, then it owns its own slot and should use /usr/bin/python3 from Fedora 22 on.
• If executables are to be shipped for both python 2 and python 3:
  • Both python 2 and python 3 variants must provide symlinks with a '-X' and '-X.Y' suffix (python runtime major version, or python runtime major.minor version), unless upstream already provides appropriately versioned executables without the dash.
  • The unversioned executable must be the python2 version.
  • For example, the python3 version of "coverage" must ship executables /usr/bin/coverage-3 and /usr/bin/coverage-3.4 (assuming python3 is currently version 3.4), while the python2 version must provide /usr/bin/coverage, /usr/bin/coverage-2 and /usr/bin/coverage-2.7 (assuming python2 version 2.7).
  • For compatibility packages, the Python version is appended *after* the specific package version, for example /usr/bin/coverage-v1.2-3 and /usr/bin/coverage-v1.2-3.4 for python3-coverage1.2 compat package.

See this thread and a newer thread [1] for discussions of this.

Packaging eggs

Please see the Python eggs guidelines for information specific to Python eggs.

Reviewer checklist

The following briefly summarizes the guidelines for reviewers to go over:

• Must: If you build for more than one python runtime you must use the %python_provide macro.
• Must: If you build for a single python runtime you must add %python_provide python-$module so that the current default python is provided from the unversioned python package.
• Must: Python modules must be built from source. They cannot simply drop an egg from upstream into the proper directory. (See prebuilt binaries Guidelines for details)
• Must: Python modules must not download any dependencies during the build process.
• Must: When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
• Must: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
• Should: A package which is used by another package via an egg interface should provide egg info.

Filtering Requires: and Provides:

RPM's dependency generator can often throw in additional dependencies and will often think packages provide functionality contrary to reality. To fix this, the dependency generator needs to be overriden so that the additional dependencies can be filtered out. See Packaging:AutoProvidesAndRequiresFiltering for details.