(s/python_site/python2_site/) |
(Updates from https://fedorahosted.org/fpc/ticket/404) |
||
Line 133: | Line 133: | ||
{{admon/warning|Avoid INSTALLED_FILES|python's distutils has an <code>INSTALLED_FILES</code> feature that lists which files are installed when you run <code>python setup.py install</code>. Do not use it for packaging as that will not list the directories which need to be specified in the <code>%files</code> section as well. Using globs in the <code>%files</code> section is simpler and safer.}} | {{admon/warning|Avoid INSTALLED_FILES|python's distutils has an <code>INSTALLED_FILES</code> feature that lists which files are installed when you run <code>python setup.py install</code>. Do not use it for packaging as that will not list the directories which need to be specified in the <code>%files</code> section as well. Using globs in the <code>%files</code> section is simpler and safer.}} | ||
{{admon/warning|Including egg info|When you run <code>%{ | {{admon/warning|Including egg info|When you run <code>%{__python2} setup.py install</code> in any current Fedora, distutils generates a <code>.egg-info</code> file with metadata about the python module that is installed. These files need to be included as well. (See [https://fedoraproject.org/wiki/Packaging:Python#Packaging_eggs_and_setuptools_concerns Packaging eggs and setuptools concerns] )}} | ||
=== Bytecompiling with the correct python version === | === Bytecompiling with the correct python version === | ||
Line 170: | Line 170: | ||
# Manually invoke the python byte compile macro for each path that needs byte | # Manually invoke the python byte compile macro for each path that needs byte | ||
# compilation. | # compilation. | ||
%py_byte_compile %{ | %py_byte_compile %{__python2} %{buildroot}%{_datadir}/mypackage/foo | ||
%py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar | %py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar | ||
</pre> | </pre> | ||
Line 176: | Line 176: | ||
The <code>%py_byte_compile</code> macro takes two arguments. The first is the python interpreter to use for byte compiling. The second is a file or directory to byte compile. If the second argument is a directory, the macro will recursively byte compile any *.py file in the directory. | The <code>%py_byte_compile</code> macro takes two arguments. The first is the python interpreter to use for byte compiling. The second is a file or directory to byte compile. If the second argument is a directory, the macro will recursively byte compile any *.py file in the directory. | ||
{{admon/warning|No %{} for py_byte_compile|RPM macros can only take arguments when they do not have curly braces around them. Therefore, py_byte_compile won't work correctly if you write: <code>%{py_byte_compile} %{ | {{admon/warning|No %{} for py_byte_compile|RPM macros can only take arguments when they do not have curly braces around them. Therefore, py_byte_compile won't work correctly if you write: <code>%{py_byte_compile} %{__python2}</code>}} | ||
Line 311: | Line 311: | ||
%endif # with_python3 | %endif # with_python3 | ||
find -name '*.py' | xargs sed -i '1s|^#!python|#!%{ | find -name '*.py' | xargs sed -i '1s|^#!python|#!%{__python2}|' | ||
</pre> | </pre> | ||
Line 325: | Line 325: | ||
<pre> | <pre> | ||
%build | %build | ||
CFLAGS="$RPM_OPT_FLAGS" %{ | CFLAGS="$RPM_OPT_FLAGS" %{__python2} setup.py build | ||
%if 0%{?with_python3} | %if 0%{?with_python3} | ||
Line 350: | Line 350: | ||
%endif # with_python3 | %endif # with_python3 | ||
%{ | %{__python2} setup.py install --skip-build --root $RPM_BUILD_ROOT | ||
rm -rf ${buildroot}%{python2_sitelib}/setuptools/tests | rm -rf ${buildroot}%{python2_sitelib}/setuptools/tests | ||
Line 358: | Line 358: | ||
%check | %check | ||
%{ | %{__python2} setup.py test | ||
%if 0%{?with_python3} | %if 0%{?with_python3} | ||
Line 369: | Line 369: | ||
You'll notice that the <code>%build</code>, <code>%install</code>, and <code>%check</code> sections follow a common pattern. They do the normal steps for building the python2 module but then they switch to <code>%{py3dir}</code> and run the same steps for python3. Creating the new sections is generally pretty easy. First copy the existing code. Then wrap it with a <code>pushd/popd</code> to <code>%{py3dir}</code>. The usage of <code>pushd/popd</code> commands will ensure that the directories are logged. Finally, convert all macro references: | You'll notice that the <code>%build</code>, <code>%install</code>, and <code>%check</code> sections follow a common pattern. They do the normal steps for building the python2 module but then they switch to <code>%{py3dir}</code> and run the same steps for python3. Creating the new sections is generally pretty easy. First copy the existing code. Then wrap it with a <code>pushd/popd</code> to <code>%{py3dir}</code>. The usage of <code>pushd/popd</code> commands will ensure that the directories are logged. Finally, convert all macro references: | ||
* <code>%{ | * <code>%{__python2}</code> becomes <code>%{__python3}</code> | ||
* <code>%{python2_sitelib}</code> becomes <code>%{python3_sitelib}</code> | * <code>%{python2_sitelib}</code> becomes <code>%{python3_sitelib}</code> | ||
* <code>%{python2_sitearch}</code> becomes <code>%{python3_sitearch}</code> | * <code>%{python2_sitearch}</code> becomes <code>%{python3_sitearch}</code> | ||
Line 488: | Line 488: | ||
== PyGTK2 and Numpy == | == PyGTK2 and Numpy == | ||
{{admon/note||This is | {{admon/note||This workaround is necessary until [[http://bugzilla.gnome.org/show_bug.cgi?id=591745 Gnome bug #591745]] is fixed.}} | ||
If your package uses pygtk2, and calls the gtk.gdk.get_pixels_array() function, that package needs to explicitly Require: numpy. In the past, pygtk2 had a Requires on numpy, but since it is only used for that one function (and that function is not commonly used), the Requires has been removed to minimize the install footprint of pygtk2. | If your package uses pygtk2, and calls the gtk.gdk.get_pixels_array() function, that package needs to explicitly Require: numpy. In the past, pygtk2 had a Requires on numpy, but since it is only used for that one function (and that function is not commonly used), the Requires has been removed to minimize the install footprint of pygtk2. | ||
[[Category:Packaging guidelines]] [[Category:Python]] | [[Category:Packaging guidelines]] | ||
[[Category:Python]] |
Revision as of 22:22, 27 March 2014
Multiple Python Runtimes
In Fedora we have multiple python runtimes, one for each supported major release. At this point that's one for python2.x and one for python3.x
Each runtime corresponds to a binary of the form /usr/bin/python$MAJOR.$MINOR
One of these python runtimes is the "system runtime" which is what we run when invoking /usr/bin/python
. On Fedora 20 this is /usr/bin/python2.7
All python runtimes have a virtual provide for python(abi) = $MAJOR-$MINOR
. For example, the python-3.1 runtime rpm has:
$ rpm -q --provides python3 |grep -i abi python(abi) = 3.1
python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with. This is done automatically for files below /usr/lib[^/]*/python${PYVER}
BuildRequires
To build a package containing python2 files, you need to have
BuildRequires: python2-devel
Similarly, when building a package which ships python3 files, you need
BuildRequires: python3-devel
A package that has both python2 and python3 files will need to BuildRequire both.
Macros
In RHEL 6 and older, python2 packages that install python modules need to define __python2
, python2_sitelib
, and python2_sitearch
macros. This is not needed in current Fedora or with python3 modules as the macros are defined by rpm
and the python3-devel
package. To define those conditionally you can use this:
%if 0%{?rhel} && 0%{?rhel} <= 6 %{!?__python2: %global __python2 /usr/bin/python2} %{!?python2_sitelib: %global python2_sitelib %(%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")} %{!?python2_sitearch: %global python2_sitearch %(%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib(1))")} %endif
Note that the use of %{!? [...]}
does allow this to work without the check for rhel versions but putting the conditional in documents when we can remove the entire stanza from the spec file.
In Fedora 13 and greater, the following macros are defined for you:
Macro | Normal Definition | Notes |
---|---|---|
__python2 | /usr/bin/python2 | Python 2 interpreter. Also the default python interpreter |
__python3 | /usr/bin/python3 | Python 3 interpreter |
python2_sitelib | /usr/lib/python2.X/site-packages | Where pure python2 modules are installed |
python2_sitearch | /usr/lib64/python2.X/site-packages on x86_64 /usr/lib/python2.X/site-packages on x86 |
Where python2 extension modules that are compiled C are installed |
python3_sitelib | /usr/lib/python3.X/site-packages | Where pure python3 modules are installed |
python3_sitearch | /usr/lib64/python3.X/site-packages on x86_64 /usr/lib/python3.X/site-packages on x86 |
Where python3 extension modules that are compiled C are installed |
py3dir | %{_builddir}/python3-%{name}-%{version}-%{release} | Directory to use when building python3 modules from the same source tarball as python2 modules |
py_byte_compile | (script) | Defined in python3-devel. See the [#Bytecompiling_with_the_correct_python_version bytecompiling] section for usage |
During %install
or when listing %files
you can use the python2_sitearch
and python2_sitelib
macros to specify where the installed modules are to be found. For instance:
%files # A pure python2 module %{python2_sitelib}/foomodule/ # A compiled python2 extension module %{python2_sitearch}/barmodule/ # A compiled python3 extension module %{python3_sitearch}/bazmodule/
Using the macros has several benefits.
- It ensures that the packages are installed correctly on multilib architectures.
- Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if
python2_sitelib
moves into%{_datadir}
)
Files to include
When installing python modules we include several different types of files.
- *.py source files because they are used when generating tracebacks
- *.pyc and *.pyo byte compiled files
- python will try to create them at runtime if they don't exist which leads to spurious SELinux AVC denials in the logs
- If the system administrator invokes python with -OO, .pyos will be created with no docstrings. This can break some programs.
- *.egg-info files or directories. If these are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at runtime.
Source files
Source files (*.py) must be included in the same packages as the byte-compiled versions of them.
Byte compiling
Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run. These files are saved in files with the extension of .pyc (compiled python) or .pyo (optimized compiled python). These files are a byte code that is portable across OSes. If you do not include them in your packages, python will try to create them when the user runs the program. If the system administrator uses them, then the files will be successfully written. Later, when the package is removed, the .pyc and .pyo files will be left behind on the filesystem. To prevent that the byte compiled files need to be compiled and included in the %files
section. Normally, byte compilation is done for you by the brp-python-bytecompile
script. This script runs after the %install
section of the spec file has been processed and byte compiles any .py files that it finds (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the %{buildroot}
in them). All that you need to do is include the files in the %files
section. The following are all acceptable ways to accomplish this:
%install install -d $RPM_BUILD_ROOT%{python2_sitelib}/foo install -pm 0644 foo.py $RPM_BUILD_ROOT%{python2_sitelib}/foo/ Either: %files %{python2_sitelib}/foo/ Or: %files %dir %{python2_sitelib}/foo %{python2_sitelib}/foo/* Or even: %files %dir %{python2_sitelib}/foo %{python2_sitelib}/foo/foo.py %{python2_sitelib}/foo/foo.pyc %{python2_sitelib}/foo/foo.pyo
Bytecompiling with the correct python version
When byte compiling a .py file, python embeds a magic number in the byte compiled files that correspond to the runtime. Files in %{python?_sitelib}
and %{python?_sitearch}
must correspond to the runtime for which they were built. For instance, a pure python module compiled for the 3.1 runtime needs to be below %{_usr}/lib/python3.1/site-packages
The brp-python-bytecompile
script tries to figure this out for you. The script determines which interpreter to use when byte compiling the module by following these steps:
- what directory is the module installed in? If it's
/usr/lib{,64}/pythonX.Y
, thenpythonX.Y
is used to byte compile the module. IfpythonX.Y
is not installed, then an error is returned and the rpm build process will exit on an error so remember toBuildRequire
the proper python package. - the script interpreter defined in
%{__python}
is used to compile the modules. This defaults to the latest python2 version on Fedora. If you need to compile this module for python3, set it to/usr/bin/python3
instead:%global __python %{__python3}
Doing this is useful when you have a python3 application that's installing a private module into its own directory. For instance, if the foobar application installs a module for use only by the command line application in
%{_datadir}/foobar
. Since these files are not in one of the python3 library paths (ie./usr/lib/python3.1
) you have to override%{__python}
to tellbrp-python-bytecompile
to use the python3 interpreter for byte compiling.
These settings are enough to properly byte compile any package that builds python modules in %{python?_sitelib}
or %{python?_sitearch}
or builds for only a single python interpreter. However, if the application you're packaging needs to build with both python2 and python3 and install into a private module directory (perhaps because it provides one utility written in python2 and a second utility written in python3) then you need to do this manually. Here's a sample spec file snippet that shows what to do:
# Turn off the brp-python-bytecompile script %global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompile[[:space:]].*$!!g') # Buildrequire both python2 and python3 BuildRequires: python2-devel python3-devel [...] %install # Installs a python2 private module into %{buildroot}%{_datadir}/mypackage/foo # and installs a python3 private module into %{buildroot}%{_datadir}/mypackage/bar make install DESTDIR=%{buildroot} # Manually invoke the python byte compile macro for each path that needs byte # compilation. %py_byte_compile %{__python2} %{buildroot}%{_datadir}/mypackage/foo %py_byte_compile %{__python3} %{buildroot}%{_datadir}/mypackage/bar
The %py_byte_compile
macro takes two arguments. The first is the python interpreter to use for byte compiling. The second is a file or directory to byte compile. If the second argument is a directory, the macro will recursively byte compile any *.py file in the directory.
Common SRPM vs split SRPMs
Many times when you package a python module you will want to create a module for python2 and a module for python3. There are two ways of doing this: either from a single SRPM or from multiple. The rule to choose which method is simple: if the python2 and python3 modules are distributed as a single tarball (many times as a single directory of source where the /usr/bin/2to3
program is used to transform the code at buildtime) then you must package them as subpackages built from a single SRPM. If they come in multiple tarballs then package them from multiple SRPMs.
Multiple SRPMS
When upstream ships multiple tarballs with one tarball containing python2 code and a different tarball containing python3 code, we should ship those as multiple SRPMs. The two SRPMs could have different maintainers within Fedora and the two packages need not upgrade at the same time. Building from multiple SRPMs has some advantages and disadvantages:
Advantages:
- There can be separate maintainers for python2 and python3 so each maintainer can concentrate on one stack.
- The two packages can evolve separately; if 2 and 3 need to have different versions, they can.
Disadvantages:
- The two specfiles have to be maintained separately
- When upstream releases e.g. security fixes, they have to be tracked in two places
The following practices are designed to help mitigate the disadvantages listed above:
- When packaging a module for python3 contact the maintainers for the python2 module and try to coordinate with them.
- Request at least watchbugzilla and watchcommit acls on each other's packages so you're aware of outstanding bugs.
- Complete any python 2 Merge Review when doing the python 3 version. Doing this gets issues that apply to both packages addressed at the same time.
- Add a link to the python 2 Merge Review/Package Review to the python 3 Package Review
Subpackages
Sometimes upstream will ship one tarball that builds both a python2 and a python3 module. There's several ways that upstream can structure this. When upstream writes their build scripts to build both python2 and python3 modules in a single build this is just like building subpackages for any other package. You expand the tarball and patch the source in %prep
, run upstream's build scripts to build the package in %build
, and then run upstream's build scripts to install it in %install
.
Advantages:
- Single src.rpm to review and build
- Avoids having to update multiple packages when things change.
Disadvantages:
- The Fedora maintainer needs to care about both python 2 and python 3 modules which makes more work to maintain that package.
- The 2 and 3 versions are in lockstep. Bugfixes need to apply to python2 while not breaking the translation into python3.
- Bugzilla components are set up according to source RPM, so they will have a single shared bugzilla component. This could be confusing to end-users, as it would be more difficult to figure out e.g. that a bug with python3-foo needs to be filed against python-foo. There's a similar problem with checking out package sources from CVS, though this is less serious as it is less visible to end users.
Two other ways exist for the upstream to support building python3 modules from a single source:
Building more than once
One way that's currently very common is for the build scripts to create either a python2 or python3 module based on which interpreter is used to run the setup.py script. (The python-setuptools package is currently built this way).
Example spec file
%if 0%{?fedora} > 12 %global with_python3 1 %else %{!?__python2: %global __python2 /usr/bin/python2} %{!?python2_sitelib: %global python2_sitelib %(%{__python2} -c "from distutils.sysconfig import get_python_lib; print (get_python_lib())")} %endif %global srcname distribute
At the top of our spec file we have the standard define for python2_sitelib
on older RHEL releases. We also define with_python3
which we'll use to conditionalize the build whenever we have a section that is only useful when building a python3 module. Using with_python3
allows us to do two things:
- It makes it easy to turn off the python3 build when tracking down problems.
- The conditionals also make it easy to use the same spec for older releases of Fedora and EPEL. .
Name: python-setuptools Version: 0.6.10 Release: 2%{?dist} Summary: Easily build and distribute Python packages Group: Applications/System License: Python or ZPLv2.0 URL: http://pypi.python.org/pypi/%{srcname} Source0: http://pypi.python.org/packages/source/d/%{srcname}/%{srcname}-%{version}.tar.gz # Fix a failing test case Patch0: python-setuptools-test.patch BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) BuildArch: noarch BuildRequires: python2-devel %if 0%{?with_python3} BuildRequires: python3-devel %endif # if with_python3
When we build the python3 module in addition to the python3 module we need both python2-devel
and python3-devel
.
%description Setuptools is a collection of enhancements to the Python distutils that allow you to more easily build and distribute Python packages, especially ones that have dependencies on other packages. This package contains the runtime components of setuptools, necessary to execute the software that requires pkg_resources.py. %if 0%{?with_python3} %package -n python3-setuptools Summary: Easily build and distribute Python 3 packages Group: Applications/System %description -n python3-setuptools Setuptools is a collection of enhancements to the Python 3 distutils that allow you to more easily build and distribute Python 3 packages, especially ones that have dependencies on other packages. This package contains the runtime components of setuptools, necessary to execute the software that requires pkg_resources.py. %endif # with_python3
Here we define the python3 subpackage. Note that we use %package -n
to name the module appropriately.
%prep %setup -q -n %{srcname}-%{version} %patch0 -p1 -b .testfix find -name '*.txt' | xargs chmod -x %if 0%{?with_python3} rm -rf %{py3dir} cp -a . %{py3dir} find %{py3dir} -name '*.py' | xargs sed -i '1s|^#!python|#!%{__python3}|' %endif # with_python3 find -name '*.py' | xargs sed -i '1s|^#!python|#!%{__python2}|'
Our method in building from the same code to make the two separate modules is to keep each build as independent as possible. To do that, we copy the source tree to %{py3dir}
so that the python 2 sources are entirely independent from the python 3 sources. Some things to watch out for:
- Be sure to clean up the
%{py3dir}
before performing the copy. It's easy to forget that since%setup
does that automatically for the python2 module. - Make sure that you are copying the correct code. The example is copying the code from within the top directory of the untarred source. If the
%prep
has changed directory you will need to change back to the tarball location. - Patching the source code is done before copying to
%{py3dir}
. Since you have both a python2 and a python3 directory you might be tempted to patch each one separately. Resist! Upstream for your package has chosen to distribute a single source tree that builds for both python2 and python3. For your patches to get into upstream, you need to write patches that work with both as well.}}
rpmbuild
resets the directory at the end of each phase, so you don't need to restore the directory at the end of %prep
.
%build CFLAGS="$RPM_OPT_FLAGS" %{__python2} setup.py build %if 0%{?with_python3} pushd %{py3dir} CFLAGS="$RPM_OPT_FLAGS" %{__python3} setup.py build popd %endif # with_python3 %install rm -rf %{buildroot} # Must do the python3 install first because the scripts in /usr/bin are # overwritten with every setup.py install (and we want the python2 version # to be the default for now). %if 0%{?with_python3} pushd %{py3dir} %{__python3} setup.py install --skip-build --root $RPM_BUILD_ROOT rm -rf %{buildroot}%{python3_sitelib}/setuptools/tests find %{buildroot}%{python3_sitelib} -name '*.exe' | xargs rm -f chmod +x %{buildroot}%{python3_sitelib}/setuptools/command/easy_install.py popd %endif # with_python3 %{__python2} setup.py install --skip-build --root $RPM_BUILD_ROOT rm -rf ${buildroot}%{python2_sitelib}/setuptools/tests find %{buildroot}%{python2_sitelib} -name '*.exe' | xargs rm -f chmod +x %{buildroot}%{python2_sitelib}/setuptools/command/easy_install.py %check %{__python2} setup.py test %if 0%{?with_python3} pushd %{py3dir} %{__python3} setup.py test popd %endif # with_python3
You'll notice that the %build
, %install
, and %check
sections follow a common pattern. They do the normal steps for building the python2 module but then they switch to %{py3dir}
and run the same steps for python3. Creating the new sections is generally pretty easy. First copy the existing code. Then wrap it with a pushd/popd
to %{py3dir}
. The usage of pushd/popd
commands will ensure that the directories are logged. Finally, convert all macro references:
%{__python2}
becomes%{__python3}
%{python2_sitelib}
becomes%{python3_sitelib}
%{python2_sitearch}
becomes%{python3_sitearch}
%clean rm -rf $RPM_BUILD_ROOT %files %doc psfl.txt zpl.txt docs %{python2_sitelib}/* %{_bindir}/easy_install %{_bindir}/easy_install-2.6 %if 0%{?with_python3} %files -n python3-setuptools %doc psfl.txt zpl.txt docs %{python3_sitelib}/* %{_bindir}/easy_install-3.1 %endif # with_python3 %changelog
In this final section, you can see that we once again switch macros from %{python2_sitelib}
to %{python3_sitelib}
. Since we chose to install the python2 version of %{_bindir}/easy_install
earlier we need to include that file in the python2 package rather than the python3 subpackage.
Running 2to3 from the spec file
Sometimes, upstream hasn't integrated running 2to3 on the code into their build scripts but they support making a python3 module from it if you manually run 2to3 on the source. This is the case when it's documented on the upstream's website, in a file in the tarball, or even when email with the module's author has instructions for building a python3 module from the python2 source and the authors are willing to support the result. In these cases it's usually just a matter of the upstream not having written the build script that can turn the python2 source into python3. When this happens you can run 2to3
from the spec file. Once you have it working, you can also help upstream integrate it into their build scripts which will benefit everyone in the long term.
You should usually follow upstream's directions on how to run 2to3
and build the python3 module in these cases but there's a few things you should check to make sure upstream is doing it correctly.
- Since the code is being built from a unified source, you need to copy the code to a new directory before invoking 2to3 just like the building more than once method.
- If the
2to3
program is invoked instead of using thelib2to3
library functions, make sure it's invoked with--write --nobackups
.--write
is needed to make2to3
actually change the files.--nobackups
avoids leavingfoo.py.bak
files in the module directories that then make it into the final package payload. - Be sure to run 2to3 on the correct directory. When you run
2to3
you need to run it on the whole tree. A common mistake here for distutils packages has been to run it on the directory belowsetup.py
, missing thesetup.py
file itself. This leads to errors whenpython3
tries to executesetup.py
- If you need to run
2to3
to fix code, use2to3
or/usr/bin/2to3
. At the moment, this program is coming from thepython-tools
rpm. Using2to3
means that you'll be using a name that is supported upstream and across distros rather than/usr/bin/python3-2to3
which we have renamed in Fedora to avoid filesystem conflicts. This also makes it easier for us to test and eventually change from using the python22to3
to the python32to3
. We just need to change the python3 package to provide the/usr/bin/2to3
program instead of python and all of our python packages will start using that version instead. - If
2to3
runs into a problem, please file a Fedora bug. Please try to isolate a minimal test case that reproduces the problem when doing so.
Avoiding collisions between the python 2 and python 3 stacks
The python 2 and python 3 stacks are intended to be fully-installable in parallel. When generalizing the package for both python 2 and python 3, it is important to ensure that two different built packages do not attempt to place different payloads into the same path.
Executables in /usr/bin
The problem
Many existing python packages install executables into /usr/bin
.
For example if we have a console_scripts
in a setup.py
shared between
python 2 and python 3 builds: these will spit out files in /usr/bin/
,
and these will collide.
For example python-coverage
has a setup.py
that contains:
entry_points = { 'console_scripts': [ 'coverage = coverage:main', ] },
which thus generates a /usr/bin/coverage
executable (this is a python
script that runs another python script whilst generating code-coverage
information on the latter).
Similarly for the 'scripts' clause; see e.g. python-pygments
:
Pygments-1.1.1/setup.py
has:
scripts = ['pygmentize'],
which generates a /usr/bin/pygmentize
(this is a python script that leverages the pygments syntax-highlighting module, giving a simple command-line interface for generating syntax-highlighted files)
Guidelines
If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only one version of the executable should be packaged. Currently it will be the python 2 implementation, but once the Python 3 implementation is proven to work, the executable can be retired from the python 2 build and enabled in the python 3 package. Be sure to test the new implementation. Transitioning from python2 to python3 is left to individual package maintainers except for packages in Fedora's critical path. For these, we want to port to python3 versions in the same Fedora release if possible.
Examples of this:
/usr/bin/pygmentize
ought to generate the same output regardless of whether it's implemented via Python 2 or Python 3, so only one version needs to be shipped.
If the executables provide different functionality for Python 2 and Python 3, then both versions should be packaged.
Examples of this:
/usr/bin/coverage
runs a python script, augmenting the interpreter with code-coverage information. Given that the interpreter itself is the thing being worked with, it's reasonable to package both versions of the executable./usr/bin/bpython
augments the interpreter with a "curses" interface. Again, it's reasonable to package both versions of this./usr/bin/easy_install
installs a module into one of the Python runtimes: we need a version for each runtime.
As an exception, for the rpms that are part of a python runtime itself, we plan to package both versions of the executables, so that e.g. both the python 2 and python 3 versions of 2to3
are packaged.
Naming
Many executables already contain a "-MAJOR.MINOR" suffix, for example /usr/bin/easy_install-3.1
. These obviously can be used as-is, as they won't conflict.
For other executables, the general rule is:
- if only one executable is to be shipped, then it owns its own slot
- if executables are to be shipped for both python 2 and python 3, then the python 3 version of the executable gains a
python3-
prefix. For example, the python 2 version of "coverage" remains/usr/bin/coverage
and the python 3 version is/usr/bin/python3-coverage
.
See this thread for a discussion of this.
Packaging eggs and setuptools concerns
Eggs can mean several different things because they can be placed on disk in several formats:
- A module and a file with a .egg-info extension that contains the metadata. Created by distutils in Fedora 9 and above.
- As a module and a directory with a .egg-info extension that contains the metadata. Created using setuptools and also the invocation of setup.py in our examples below.
- As a directory with a .egg extension that contains the module and egg metadata. Created when we use easy_install -m to allow installing multiple versions of a module.
- As a single zip file with a .egg extension that contains the module and the egg metadata.
In Fedora packages, these will be installed to %{python2_sitelib} or %{python2_sitearch} directories. We do not install the single zip file version of eggs in Fedora but the three other formats are used.
How to package
The following are a summary of the guidelines for reviewers to go over when a python module is packaged. The complete policy includes examples and rationale for the way we do things.
- Must: Python eggs must be built from source. They cannot simply drop an egg from upstream into the proper directory. (See prebuilt binaries Guidelines for details)
- Must: Python eggs must not download any dependencies during the build process.
- Must: When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
- Must: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
- Should: A package which is used by another package via an egg interface should provide egg info.
Filtering Requires: and Provides:
RPM's dependency generator can often throw in additional dependencies and will often think packages provide functionality contrary to reality. To fix this, the dependency generator needs to be overriden so that the additional dependencies can be filtered out. See Packaging:AutoProvidesAndRequiresFiltering for details.
PyGTK2 and Numpy
If your package uses pygtk2, and calls the gtk.gdk.get_pixels_array() function, that package needs to explicitly Require: numpy. In the past, pygtk2 had a Requires on numpy, but since it is only used for that one function (and that function is not commonly used), the Requires has been removed to minimize the install footprint of pygtk2.