User:Zbyszek/PythonPackagingDraft

Python Version Support

In Fedora we have multiple python runtimes, one for each supported major release. At this point that's one for python2.x and one for python3.x. If a piece of software supports python3, it must be packaged for python3. If it supports python2 as well, it may be packaged for python2. If it supports only python2 then it must not be packaged for python3.

Multiple Python Runtimes

Each runtime corresponds to a binary of the form /usr/bin/python$MAJOR.$MINOR One of these python runtimes is the "system runtime" which is what is run when invoking /usr/bin/python. On Fedora 21 this is a link to /usr/bin/python2 (which itself is a link to /usr/bin/python2.7).

However, packages in Fedora should not depend on where /usr/bin/python happens to point but instead should call the proper executable for the needed python major version directly, either /usr/bin/python2 or /usr/bin/python3 as appropriate.

All python runtimes have a virtual provide for python(abi) = $MAJOR-$MINOR. For example, the python-3.4 runtime rpm has:

 $ rpm -q --provides python3 |grep -i abi
 python(abi) = 3.4

python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with. This is done automatically for files below /usr/lib[^/]*/python${PYVER}

BuildRequires

To build a package containing python2 files, you need to have

BuildRequires: python2-devel

Similarly, when building a package which ships python3 files, you need

BuildRequires: python3-devel

A package that has both python2 and python3 files will need to BuildRequire both.

Provides

Python2 packages need to provide a python2 version of their name. So, for example, a python package of the "foo" module needs to include Provides: python2-foo.

Macros

In RHEL 6 and older, python2 packages that install python modules need to define __python2, python2_sitelib, and python2_sitearch macros. This is not needed in current Fedora or with python3 modules as the macros are defined by rpm and the python3-devel package. To define those conditionally you can use this:

%if 0%{?rhel} && 0%{?rhel} <= 6
%{!?__python2: %global __python2 /usr/bin/python2}
%{!?python2_sitelib: %global python2_sitelib %(%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")}
%{!?python2_sitearch: %global python2_sitearch %(%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")}
%endif

Note that the use of %{!? [...]} does allow this to work without the check for rhel versions but putting the conditional in documents when we can remove the entire stanza from the spec file.

In Fedora, the following macros are defined for you:

Macro	Normal Definition	Notes
__python	%{__python2}	Default Python interpreter. Currently links to Python2 interpreter
__python2	/usr/bin/python2	Python 2 interpreter. Also the current default python interpreter
__python3	/usr/bin/python3	Python 3 interpreter
python_sitelib	%{python2_sitelib}	Where pure python modules are installed for the default Python implementation
python_sitearch	%{python2_sitearch}	Where python extension modules that are compiled C are installed for the default Python implementation
python2_sitelib	/usr/lib/python2.X/site-packages	Where pure python2 modules are installed
python2_sitearch	/usr/lib64/python2.X/site-packages on x86_64 /usr/lib/python2.X/site-packages on x86	Where python2 extension modules that are compiled C are installed
python3_sitelib	/usr/lib/python3.X/site-packages	Where pure python3 modules are installed
python3_sitearch	/usr/lib64/python3.X/site-packages on x86_64 /usr/lib/python3.X/site-packages on x86	Where python3 extension modules that are compiled C are installed
py_byte_compile	(script)	Defined in python3-devel. See the [#Bytecompiling_with_the_correct_python_version bytecompiling] section for usage
python3_version	3.X	Defined in python3-devel. Useful when running programs with Python version in filename, such as nosetest-%{python3_version}
python3_version_nodots	3X	Defined in python3-devel since Fedora 21 / Python 3.4. Useful when listing files explicitly in %files section , such as %{python3_sitelib}/foo/*.cpython-%{python3_version_nodots}.pyo

The generic %{_python} macros
The unversioned macros, %{__python}, %{python_sitelib}, and %{python_sitearch} are generic macros that will always point to the default implementation. You may only use them with applications that need to choose to use the system's default version of python. (Currently this is the python2 interpreter.)

This is future proofing for the time when things will be switched over to python3 by default instead of python2.

You should use %{__python2}, %{python2_sitelib}, and %{python2_sitearch} to explicitly reference the python2 interpreter instead.

During %install or when listing %files you can use the python2_sitearch and python2_sitelib macros to specify where the installed modules are to be found. For instance:

%files
# A pure python2 module
%{python2_sitelib}/foomodule/
# A compiled python2 extension module
%{python2_sitearch}/barmodule/
# A compiled python3 extension module
%{python3_sitearch}/bazmodule/

Using the macros has several benefits.

1. It ensures that the packages are installed correctly on multilib architectures.
2. Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if python2_sitelib moves into %{_datadir})

Files to include

When installing python modules we include several different types of files.

• *.py source files because they are used when generating tracebacks
• *.pyc and *.pyo byte compiled files (and, if present, the enclosing __pycache__ directory in most cases)
• python will try to create them at runtime if they don't exist which leads to spurious SELinux AVC denials in the logs
• If the system administrator invokes python with -OO, .pyos will be created with no docstrings. This can break some programs.
• *.egg-info files or directories. If these are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at runtime.

Source files

Source files (*.py) must be included in the same packages as the byte-compiled versions of them.

Byte compiling

Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run. These files are saved in files with the extension of .pyc (compiled python) or .pyo (optimized compiled python). With current versions of python 3, these files will be located inside a directory named __pycache__. Older versions of python will simply place them alongside the .py files.

The .pyc and .pyo files contain byte code that is portable across OSes. If you do not include them in your packages, python will try (and generally fail) to create them when the user runs the program. If the system administrator runs the program, then the files will be successfully written, causing stray .pyc and .pyo files which will not be removed when the package is removed. To prevent that the byte compiled files need to be compiled and included in the %files section. Normally, byte compilation is done for you by the brp-python-bytecompile script. This script runs after the %install section of the spec file has been processed and byte compiles any .py files that it finds (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the %{buildroot} in them).

You must include in your package the .pyc and .pyo files. If the build process creates a __pycache__ directory in a subdirectory of  %{python3_sitearch} or %{python3_sitelib}, you must also include all items in the __pycache__ directory. (You should not include the directories %{python3_sitearch}/__pycache__ or %{python3_sitelib}/__pycache__ because they are already owned by the python3-libs package.)

All that you need to do is include the files in the %files section (replacing %{python3_sitelib} with the appropriate macro for your package):

%files
%{python3_sitelib}/foo/

or, if the python code installs directly into %{python3_sitelib}:

%files
%{python3_sitelib}/foo.py
%{python3_sitelib}/__pycache__/*

Avoid INSTALLED_FILES
python's distutils has an INSTALLED_FILES feature that lists which files are installed when you run python setup.py install. Do not use it for packaging as that will not list the directories which need to be specified in the %files section as well. Using globs in the %files section is simpler and safer.

Including egg info
When you run %{__python2} setup.py install in any current Fedora, distutils generates a .egg-info file with metadata about the python module that is installed. These files need to be included as well. (See Packaging eggs and setuptools concerns )

Bytecompiling with the correct python version

When byte compiling a .py file, python embeds a magic number in the byte compiled files that correspond to the runtime. Files in %{python?_sitelib} and %{python?_sitearch} must correspond to the runtime for which they were built. For instance, a pure python module compiled for the 3.4 runtime needs to be below %{_usr}/lib/python3.4/site-packages

The brp-python-bytecompile script tries to figure this out for you. The script determines which interpreter to use when byte compiling the module by following these steps:

1. what directory is the module installed in? If it's /usr/lib{,64}/pythonX.Y, then pythonX.Y is used to byte compile the module. If pythonX.Y is not installed, then an error is returned and the rpm build process will exit on an error so remember to BuildRequire the proper python package.
2. the script interpreter defined in %{__python} is used to compile the modules. This defaults to the latest python2 version on Fedora. If you need to compile this module for python3, set it to /usr/bin/python3 instead:

   %global __python %{__python3}
   

   Doing this is useful when you have a python3 application that's installing a private module into its own directory. For instance, if the foobar application installs a module for use only by the command line application in %{_datadir}/foobar. Since these files are not in one of the python3 library paths (ie. /usr/lib/python3.1) you have to override %{__python} to tell brp-python-bytecompile to use the python3 interpreter for byte compiling.

These settings are enough to properly byte compile any package that builds python modules in %{python?_sitelib} or %{python?_sitearch} or builds for only a single python interpreter. However, if the application you're packaging needs to build with both python2 and python3 and install into a private module directory (perhaps because it provides one utility written in python2 and a second utility written in python3) then you need to do this manually. Here's a sample spec file snippet that shows what to do:

# Turn off the brp-python-bytecompile script
%global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompile[[:space:]].*$!!g')
# Buildrequire both python2 and python3
BuildRequires: python2-devel python3-devel
[...]

%install
# Installs a python2 private module into %{buildroot}%{_datadir}/mypackage/foo
# and installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar
make install DESTDIR=%{buildroot}

# Manually invoke the python byte compile macro for each path that needs byte
# compilation.
%py_byte_compile %{__python2} %{buildroot}%{_datadir}/mypackage/foo
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar

The %py_byte_compile macro takes two arguments. The first is the python interpreter to use for byte compiling. The second is a file or directory to byte compile. If the second argument is a directory, the macro will recursively byte compile any *.py file in the directory.

No %{} for py_byte_compile
RPM macros can only take arguments when they do not have curly braces around them. Therefore, py_byte_compile won't work correctly if you write: %{py_byte_compile} %{__python2}

Common SRPM vs split SRPMs

Many times when you package a python module you will want to create a module for python2 and a module for python3. Both version should be built from the same SRPM. The exception to this would be if the two versions are distributed as separate tarballs and do not follow the same release schedule.

Python Bindings
python bindings are sometimes built as part of the C library's build. The ideal for these is to patch the code so it will build against both python2 and python3. Most build systems support a seperate build directory. In that case in the build phase configure one build subdirectory to build against python2, another to build against python3. Only in case this is not possible, copy the sources to a second build directory in %prep. Example: the build of rpm itself emits an rpm-python subpackage (see Red Hat Bug 531543.)

Building more than once

One way that's currently very common is for the build scripts to create either a python2 or python3 module based on which interpreter is used to run the setup.py script.

Example spec file

%if 0%{?fedora}
%global with_python3 1
%else
%{!?__python2: %global __python2 /usr/bin/python2}
%{!?python2_sitelib: %global python2_sitelib %(%{__python2} -c "from distutils.sysconfig import get_python_lib; print (get_python_lib())")}
%endif

%global srcname traitlets

At the top of our spec file we have the standard define for python2_sitelib on older RHEL releases. We also define with_python3 which we'll use to conditionalize the build whenever we have a section that is only useful when building a python3 module. Using with_python3 makes it easy to use the same spec for older releases of Fedora and EPEL.

Leave python2 and python3 modules enabled in releases
Once support for a certain version of has been added to a package, it must be left enabled. End users could be using the python2 or python3 subpackage. If you turn the subpackage build on and off it will cause the package to unexpectedly disappear from the repos. You should only turn off with_python3 as a debugging measure within scratch builds, for releases that do not support python 3, or when moving a python3 module into its own, independent package.

Name:           python-%{srcname}
Version:        4.0.0
Release:        1%{?dist}
Summary:        A lightweight derivative of Enthought Traits
License:        BSD
URL:            https://github.com/ipython/%{srcname}
Source0:        https://github.com/ipython/%{srcname}/archive/%{version}.tar.gz#/%{srcname}-%{version}.tar.gz

BuildArch:      noarch
BuildRequires:  python2-devel
Provides:	python2-%{srcname} = %{version}-%{release}

When we build the python3 module in addition to the python2 module we need both python2-devel and python3-devel.

%description
A lightweight pure-Python derivative of Enthought Traits, used for
configuring Python objects.

%if 0%{?with_python3}
%package -n python3-%{srcname}
Summary:        A lightweight derivative of Enthought Traits
BuildRequires:  python3-devel

%description -n python3-%{srcname}
A lightweight pure-Python derivative of Enthought Traits, used for
configuring Python objects.

%endif # with_python3

Here we define the python3 subpackage. Note that we use %package -n to name the module appropriately.

%prep
%autosetup -n %{srcname}-%{version}

Building both versions in the same directory is possible because setuptools uses different build directories for multiple python versions and build architectures (e.g. build/lib.linux-x86_64-2.7/ and build/lib.linux-x86_64-3.4/). In addition, starting with python3, the version of the interpreter is included in the file name (.cpython-34m.so, .cpython-34.pyc), so build products do not conflict. Python 2.x does not use this, but because we are building for only one version of Python 2 and one version of Python 3, the conflict is avoided anyway. Sometimes it is not possible to build both version from the same directory. Most often this happens when sources are modified during build to "translate" them from python2 to python3 using 2to3. In that case it is necessary to use a copy of the source directory to build both versions. See Using separate build directories below.

%build
%{__python2} setup.py build
%if 0%{?with_python3}
%{__python3} setup.py build
%endif

%install
%{__python2} setup.py install --skip-build --root %{buildroot}
%if 0%{?with_python3}
%{__python3} setup.py install --skip-build --root %{buildroot}
%endif

%check
%{__python2} setup.py check
%if 0%{?with_python3}
%{__python3} setup.py check
%endif

You'll notice that the %build, %install, and %check sections follow a common pattern. They do the normal steps for building the python2 module but then they switch to python3 and run the same steps for python3. Creating the new sections is generally pretty easy. Copy the code for python2 and convert all macro references:

• %{__python2} becomes %{__python3}
• %{python2_sitelib} becomes %{python3_sitelib}
• %{python2_sitearch} becomes %{python3_sitearch}

Order can be important
In the %install section, the order in which you do the python2 versus python3 install matters when the install is writing to the same file in both packages (usually this matters only for scripts in %{_bindir}). Starting with Fedora 22, python3 binaries should be used by default, so install for python2 first, and then for python3.

NOTE:
When compiling a module, proper build flags should be used.

Add

export CFLAGS="$RPM_OPT_FLAGS"

at the beginning of %build section for non-noarch modules.

Finally, contents of the rpms should be listed in the %files section:

%files
%doc README
%license COPYING
%{python2_sitelib}/*

%if 0%{?with_python3}
%files -n python3-%{srcname}
%doc README
%license COPYING
%{python3_sitelib}/*
%endif

NOTE:
In this case the README and COPYING files are identical between the two subpackages. It provides no benefit to the user to have both /usr/share/doc/python-trailets/README and /usr/share/doc/python-trailets/README: when looking for documentation the user is likely to check both files, only to realize that they are identical. Therefore is is often better to use a single documentation directory shared between the two subpackages, and a single license directory shared between the two subpackages. We can do this by adding:

%global _docdir_fmt %{name}

Using separate build directories

Sometimes is it impossible to build both versions from the same source directory. Most often this happens when sources are "translated" to python3 in the source directory and made incompatible with python2 in the process. This used to be fairly common, but is fortunately much rarer now. Some things to look for are:

1. Sources are not Python 3 compatible (print without parentheses is used, old module names like ConfigParser are imported),
2. six module is not used,
3. 2to3 is run in setup.py without creating a separate build directory.

The method in building from the same code is to make the two separate builds as independent as possible. To do that, we copy the source tree to %{py3dir} so that the python2 sources are entirely independent from the python3 sources. Some things to watch out for:

• Make sure that you are copying the correct code. The example is copying the code from within the top directory of the untarred source. If the %prep has changed directory you will need to change back to the tarball location.
• Patching the source code is done before copying to python3. Since you have both a python2 and a python3 directory you might be tempted to patch each one separately. Resist! Upstream for your package has chosen to distribute a single source tree that builds for both python2 and python3. For your patches to get into upstream, you need to write patches that work with both as well.}}

rpmbuild resets the directory at the end of each phase, so you don't need to restore the directory at the end of %prep.

%prep
%autosetup
find -name '*.txt' | xargs chmod -x    # adjust sources *before* copying

%if 0%{?with_python3}
cp -p . ${py3dir}
pushd ${py3dir}
find . -name '*.py' | xargs sed -i '1s|^#!python|#!%{__python3}|'
popd
%endif # with_python3

find . -name '*.py' | xargs sed -i '1s|^#!.*|#!%{__python2}|'

%build
pushd python2
%{__python2} setup.py build
popd

%if 0%{?with_python3}
pushd python3
%{__python3} setup.py build
popd
%endif # with_python3

%install
# Must do the python2 install first because the scripts in /usr/bin are
# overwritten with every setup.py install (and we want the python3 version
# to be the default for now).
%{__python2} setup.py install --skip-build --root %{buildroot}

%if 0%{?with_python3}
pushd %{py3dir}
%{__python3} setup.py install --skip-build --root %{buildroot}
popd
%endif # with_python3

%check
pushd python2
%{__python2} setup.py test
popd

%if 0%{?with_python3}
pushd python3
%{__python3} setup.py test
popd
%endif # with_python3

You'll notice that the %build, %install, and %check sections follow a common pattern. They do the normal steps for building the python2 module but then they switch to the other source directory and run the same steps for python3. Creating the new sections is generally pretty easy. First copy the existing code. Then wrap it with a pushd/popd to python3. The usage of pushd/popd commands will ensure that the directories are logged. Finally, convert all macro references (see above for macro list).

Running 2to3 from the spec file

Sometimes, upstream hasn't integrated running 2to3 on the code into their build scripts but they support making a python3 module from it if you manually run 2to3 on the source. This is the case when it's documented on the upstream's website, in a file in the tarball, or even when email with the module's author has instructions for building a python3 module from the python2 source and the authors are willing to support the result. In these cases it's usually just a matter of the upstream not having written the build script that can turn the python2 source into python3. When this happens you can run 2to3 from the spec file. Once you have it working, you can also help upstream integrate it into their build scripts which will benefit everyone in the long term.

You should usually follow upstream's directions on how to run 2to3 and build the python3 module in these cases but there's a few things you should check to make sure upstream is doing it correctly.

• If the 2to3 program is invoked instead of using the lib2to3 library functions, make sure it's invoked with --write --nobackups. --write is needed to make 2to3 actually change the files. --nobackups avoids leaving foo.py.bak files in the module directories that then make it into the final package payload.
• Be sure to run 2to3 on the correct directory. When you run 2to3 you need to run it on the whole tree. A common mistake here for distutils packages has been to run it on the directory below setup.py, missing the setup.py file itself. This leads to errors when python3 tries to execute setup.py
• If you need to run 2to3 to fix code, use 2to3 or /usr/bin/2to3. At the moment, this program is coming from the python-tools rpm. Using 2to3 means that you'll be using a name that is supported upstream and across distros rather than /usr/bin/python3-2to3 which we have renamed in Fedora to avoid filesystem conflicts. This also makes it easier for us to test and eventually change from using the python2 2to3 to the python3 2to3. We just need to change the python3 package to provide the /usr/bin/2to3 program instead of python and all of our python packages will start using that version instead.
• If 2to3 runs into a problem, please file a Fedora bug. Please try to isolate a minimal test case that reproduces the problem when doing so.

Avoiding collisions between the python 2 and python 3 stacks

The python 2 and python 3 stacks are intended to be fully-installable in parallel. When generalizing the package for both python 2 and python 3, it is important to ensure that two different built packages do not attempt to place different payloads into the same path.

Executables in /usr/bin

The problem

Many existing python packages install executables into /usr/bin.

For example if we have a console_scripts in a setup.py shared between python 2 and python 3 builds: these will spit out files in /usr/bin/, and these will collide.

For example python-coverage has a setup.py that contains:

    entry_points = {
        'console_scripts': [
            'coverage = coverage:main',
            ]
        },

which thus generates a /usr/bin/coverage executable (this is a python script that runs another python script whilst generating code-coverage information on the latter).

Similarly for the 'scripts' clause; see e.g. python-pygments: Pygments-1.1.1/setup.py has:

    scripts = ['pygmentize'],

which generates a /usr/bin/pygmentize (this is a python script that leverages the pygments syntax-highlighting module, giving a simple command-line interface for generating syntax-highlighted files)

Guidelines

If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only one version of the executable should be packaged. On releases up to and including F21, this was the python 2 implementation. Python3 should be used in F22 and later if supported by upstream. Be sure to test the new implementation. Transitioning from python2 to python3 is left to individual package maintainers except for packages in Fedora's critical path. For these, we want to port to python3 versions in the same Fedora release if possible.

Examples of this:

• /usr/bin/pygmentize ought to generate the same output regardless of whether it's implemented via Python 2 or Python 3, so only one version needs to be shipped.

If the executables provide different functionality for Python 2 and Python 3, then both versions should be packaged.

Examples of this:

• /usr/bin/coverage runs a python script, augmenting the interpreter with code-coverage information. Given that the interpreter itself is the thing being worked with, it's reasonable to package both versions of the executable.
• /usr/bin/bpython augments the interpreter with a "curses" interface. Again, it's reasonable to package both versions of this.
• /usr/bin/easy_install installs a module into one of the Python runtimes: we need a version for each runtime.

As an exception, for the rpms that are part of a python runtime itself, we plan to package both versions of the executables, so that e.g. both the python 2 and python 3 versions of 2to3 are packaged.

Naming

Many executables already contain a "-MAJOR.MINOR" suffix, for example /usr/bin/easy_install-3.4. These obviously can be used as-is, as they won't conflict.

For other executables, the general rule is:

• If only one executable is to be shipped, then it owns its own slot and should use /usr/bin/python3 from Fedora 22 on.
• If executables are to be shipped for both python 2 and python 3:
  • Both python 2 and python 3 variants must provide symlinks with a '-X' and '-X.Y' suffix (python runtime major version, or python runtime major.minor version), unless upstream already provides appropriately versioned executables without the dash.
  • The unversioned executable must be the python 2 version.
  • For example, the python 2 version of "coverage" must ship executables /usr/bin/coverage, /usr/bin/coverage-2 and /usr/bin/coverage-2.7, while the python 3 version must provide /usr/bin/coverage-3 and /usr/bin/coverage-3.4 (assuming python3 is Python 3.4).
  • For compatibility packages, the Python version is appended *after* the specific package version, for example /usr/bin/coverage-v1.2-3 and /usr/bin/coverage-v1.2-3.4 for python3-coverage1.2 compat package.

See this thread and a newer thread [1] for discussions of this.

Packaging eggs and setuptools concerns

Eggs can mean several different things because they can be placed on disk in several formats:

• A module and a file with a .egg-info extension that contains the metadata. Created by distutils in Fedora 9 and above.
• As a module and a directory with a .egg-info extension that contains the metadata. Created using setuptools and also the invocation of setup.py in our examples below.
• As a directory with a .egg extension that contains the module and egg metadata. Created when we use easy_install -m to allow installing multiple versions of a module.
• As a single zip file with a .egg extension that contains the module and the egg metadata.

In Fedora packages, these will be installed to %{python2_sitelib} or %{python2_sitearch} directories. We do not install the single zip file version of eggs in Fedora but the three other formats are used.

How to package

The following are a summary of the guidelines for reviewers to go over when a python module is packaged. The complete policy includes examples and rationale for the way we do things.

• Must: Python eggs must be built from source. They cannot simply drop an egg from upstream into the proper directory. (See prebuilt binaries Guidelines for details)
• Must: Python eggs must not download any dependencies during the build process.
• Must: When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
• Must: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
• Should: A package which is used by another package via an egg interface should provide egg info.

Filtering Requires: and Provides:

RPM's dependency generator can often throw in additional dependencies and will often think packages provide functionality contrary to reality. To fix this, the dependency generator needs to be overriden so that the additional dependencies can be filtered out. See Packaging:AutoProvidesAndRequiresFiltering for details.