From Fedora Project Wiki

(Writeup of https://pagure.io/packaging-committee/issue/782)
(Writeup of https://pagure.io/packaging-committee/issue/793)
Line 4: Line 4:
== Python Version Support ==
== Python Version Support ==


In Fedora we have multiple Python runtimes, one for each supported major Python release. At this point that's one for python3.x and one for python2.7  However, it is possible that the python2 stack will be removed from Fedora when support for the python2 interpreter officially ends in 2020. If a piece of software supports python3, it MUST be packaged for python3. If it supports python2 as well, it MAY be packaged for python2. If it supports only python2 then (obviously) it MUST NOT be packaged for python3 but upstream SHOULD be contacted and encouraged to rectify this issue.
In Fedora we have multiple Python runtimes, one for each supported major Python release. At this point that's one for python3.x and one for python2.7  However the python2 stack will be removed from Fedora and is [[Packaging:Deprecating_Packages|deprecated]]. Upstream support for the python2 interpreter officially ends in 2020. If a piece of software supports python3, it MUST be packaged for python3. Software using python2 MUST NOT  be newly packaged without FESCo exception.
 
For guidelines on maintaining already existing python2 packages, see the [[Packaging:Python_Appendix|appendix]].


== Multiple Python Runtimes ==
== Multiple Python Runtimes ==


Each runtime corresponds to a binary of the form <code>/usr/bin/python$MAJOR.$MINOR</code>.  One of these python runtimes is the "system runtime" which is what is run when invoking <code>/usr/bin/python</code>.  On Fedora 27, for example, this is a link to <code>/usr/bin/python2</code> (which itself is a link to <code>/usr/bin/python2.7</code>).
For backwards compatibility, <code>/usr/bin/python</code> is, if it is installed, a symbolic link to <code>/usr/bin/python2</code>.


However, packages in Fedora MUST NOT depend on where <code>/usr/bin/python</code> happens to point but instead MUST call the proper executable for the needed python major version directly, either <code>/usr/bin/python2</code> or <code>/usr/bin/python3</code> as appropriate. As a result of that <code>/usr/bin/python</code> (as well as <code>/usr/bin/env python</code> and similar) MUST NOT be used in shebang lines or as a dependency of a package.  As of Fedora 28, all uses of unversioned python executables in shebang lines are automatically converted to <code>/usr/bin/python2</code>.  When this is done, warnings such as the following may appear in the build log:
Packages in Fedora MUST NOT use <code>/usr/bin/python</code>. Instead packages for Python 3 MUST use <code>/usr/bin/python3</code> (even if upstream supports both Python 2 and 3). As a result of that <code>/usr/bin/python</code> (as well as <code>/usr/bin/env python</code> and similar) MUST NOT be used in shebang lines or as a dependency of a package.  As of Fedora 30, all uses of unversioned python executables in shebang lines will fail the build.
<pre>
These shebangs MUST be fixed.  If it is necessary to disable the checks, please see the information in [[Packaging:Guidelines#Shebang_lines]].
*** WARNING: mangling shebang in ./usr/bin/taskotron_result from #!/usr/bin/python to #!/usr/bin/python2. This will become an ERROR, fix it manually!
</pre>
These MUST be fixed manually, as the warnings will eventually become errors which fail the build.  If it is necessary to disable the checks, please see the information in [[Packaging:Guidelines#Shebang_lines]].


All python runtimes have a virtual provide for <code>python(abi) = $MAJOR-$MINOR</code>.  For example, the python-3.4 runtime package has:
All python runtimes have a virtual provide for <code>python(abi) = $MAJOR-$MINOR</code>.  For example, the Python 3.7 runtime package has:
   $ rpm -q --provides python3 |grep -i abi
   $ rpm -q --provides python3 |grep -i abi
   python(abi) = 3.4
   python(abi) = 3.7


python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with.  This is done automatically for files below <code>/usr/lib[^/]*/python${PYVER}</code>
Python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with.  This is done automatically for files below <code>/usr/lib[^/]*/python${PYVER}</code>


Mirroring the policy for regular packages, the Python-version-specific subpackages of your package MUST NOT be removed in a release branch of Fedora.
Mirroring the policy for regular packages, the Python-version-specific subpackages of your package MUST NOT be removed in a release branch of Fedora.
== Naming ==
The source package for a Python library MUST be named with the <code>python-</code> prefix. A built package however must include the Python major version in the name, using the <code>python3-</code> prefix. This is accomplished by adding a subpackage. See example bellow.
This rule does not apply to applications.


== Dependencies ==
== Dependencies ==
Packages building for Python2 will need <code>BuildRequires: python2-devel</code>.
Packages building for Python 3 will need <code>BuildRequires: python3-devel</code>.
Packages building for Python3 will need <code>BuildRequires: python3-devel</code>.
 
Packages building for for both will need build-time dependencies on both.
Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned <code>python-</code> prefix. Dependencies on Python packages instead MUST use names beginning with <code>python3-</code>.


Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned <code>python-</code> prefix unless no properly versioned package exists.  Dependencies on Python packages instead MUST use names beginning with <code>python2-</code> or <code>python3-</code> as appropriate.


=== Automatically generated dependencies ===
=== Automatically generated dependencies ===
Line 83: Line 86:


== Provides ==
== Provides ==
Using a fictional module named "example", the subpackage containing the python2 version '''must''' provide <code>python2-example</code>.  This is of course always the case if the subpackage is named <code>python2-example</code> (as in the examples below).  If the subpackage has some other name then then <code>Provides: python2-example</code> must be added explicitly (but see the <code>%python_provide</code> macro below).
Using a fictional module named "example", the subpackage containing the python3 version '''must''' provide <code>python3-example</code>.  This is of course always the case if the subpackage is named <code>python3-example</code> (as in the examples below).  If the subpackage has some other name then then <code>Provides: python3-example</code> must be added explicitly (but see the <code>%python_provide</code> macro below).


The python3 subpackage '''must''' provide <code>python3-example</code>.  However, as the naming guidelines mandate that the python3 subpackage be named <code>python3-example</code>, this will happen automatically.


=== The %python_provide macro ===
=== The %python_provide macro ===


In addition, the subpackage corresponding to the current system runtime '''must''' provide <code>Provides: python-example</code>.  However, the system version of python in Fedora will almost certainly change at some point in the future.
In order to make the switch from Python 2 to Python 3 automatic, all packages that provide <code>python3-%{srcname}</code> (for any <code>%{srcname}</code>) SHOULD use the <code>%python_provide</code> macro with the package name, for example:
 
To avoid having to alter every python module package when this change happens, and to allow the same spec to build for releases with different system versions of python, a convenient <code>%python_provide</code> macro is provided.  When given (as an argument) the name of the current subpackage, it will evaluate to either the proper <code>Provide:</code> line or nothing at all as appropriate for the version of the system runtime.  Thus if the package follows the suggested subpackage naming scheme as shown in the below example, placing this macro in the definition of each subpackage will guarantee that the necessary <code>Provide:</code> line appears where it should, and does not appear where it shouldn't.


If <code>%python_provide</code> is called with an argument beginning with <code>python2-</code>, it will also generate a suitable <code>Obsoletes:</code> line for the unversioned <code>python-</code> package.  This facilitate renaming those unversioned subpackages.
  %{?python_provide:%python_provide python3-%{srcname}}


Finally, if the macro called with an argument like <code>python-example</code> (no python major version in the name) then it will evaluate to either <code>Provides: python2-example</code> or <code>Provides: python3-example</code> depending on the version of the current system runtime.
This eases distribution-wide renaming of Python packages. (For example, in the future a virtual provide of <code>python-%{srcname}</code> might become appropriate for Python 3 libraries. In that case, <code>%python_provide</code> will be changed to add it.) Packages that do not include this macro would need to be adapted to such changes manually.


== Automatic Provides with a standardized name ==
== Automatic Provides with a standardized name ==
Line 102: Line 102:


<pre>
<pre>
Provides: pythonX.Ydist(CANONICAL_STANDARDIZED_NAME)
Provides: python3.Ydist(CANONICAL_STANDARDIZED_NAME)
Provides: pythonXdist(CANONICAL_STANDARDIZED_NAME)
Provides: python3dist(CANONICAL_STANDARDIZED_NAME)
</pre>
</pre>


The <code>X</code> and <code>X.Y</code> are the Python version used (usually 2.7 or 3.5 and higher), and between the parentheses is the name of the software in a '''canonical format''' used by Python tools and services such as setuptools, pip and PyPI. The canonical name is obtained by switching the standardized name to lower case and converting all runs of non-alphanumeric characters to single “-” characters. Example: “The $$$ Tree” becomes “the-tree”.
The <code>3.Y</code> is the Python version used (usually 3.6 and higher), and between the parentheses is the name of the software in a '''canonical format''' used by Python tools and services such as setuptools, pip and PyPI. The canonical name is obtained by switching the standardized name to lower case and converting all runs of non-alphanumeric characters to single “-” characters. Example: “The $$$ Tree” becomes “the-tree”.


=== Requires and BuildRequires with standardized names ===
=== Requires and BuildRequires with standardized names ===


Starting with Fedora 26, these Provides tags can be used to list Requires and BuildRequires of a package using the ''standardized names'' (i.e. dist name, name on PyPI) of Python modules. To make it easier, you can use macros <code>%{py2_dist}</code> and <code>%{py3_dist}</code> that accept one or more parameters: the ''standardized name(s)'' of the desired Python software. They will convert the name(s) to the ''canonical format'' and create the proper <code>pythonXdist(...)</code> tag(s).
These Provides tags can be used to list Requires and BuildRequires of a package using the ''standardized names'' (i.e. dist name, name on PyPI) of Python modules. To make it easier, you can use the <code>%{py3_dist}</code> macro that accept one or more parameters: the ''standardized name(s)'' of the desired Python software. It will convert the name(s) to the ''canonical format'' and create the proper <code>python3dist(...)</code> tag(s).


In addition, you can use the %{py_dist_name} macro that simply transforms any ''standardized name'' to the ''canonical format''.
In addition, you can use the %{py_dist_name} macro that simply transforms any ''standardized name'' to the ''canonical format''.
Line 117: Line 117:


<pre>
<pre>
BuildRequires: %{py2_dist PyMySQL} >= 0.7.5
BuildRequires: %{py3_dist PyMySQL} >= 0.7.5
# => BuildRequires: python2dist(pymysql) >= 0.7.5
# => BuildRequires: python3dist(pymysql) >= 0.7.5


Requires: %{py3_dist virtualenv pyPEG2}
Requires: %{py3_dist virtualenv pyPEG2}
Line 142: Line 142:
|-
|-
|__python||/usr/bin/python||Prohibited (see note below)
|__python||/usr/bin/python||Prohibited (see note below)
|-
|__python2||/usr/bin/python2||Python 2 interpreter.
|-
|-
|__python3||/usr/bin/python3||Python 3 interpreter
|__python3||/usr/bin/python3||Python 3 interpreter
|-
|-
|python_provide|| (Lua script) ||Given a package name, evaluates to either <code>Provides: python-example</code> or nothing at all depending on the version of the system runtime.  See [[Packaging:Python#The_.25python_provide_macro|here]] for an example.
|python_provide|| (Lua script) ||Given a package name, evaluates to either <code>Provides: python-example</code> or nothing at all depending on the Python version.  See [[Packaging:Python#The_.25python_provide_macro|here]] for an example.
|-
|py2_dist|| (Lua script) ||Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format, and evaluates to <code>python2dist(CANONICAL_NAME)</code>, which is useful when listing dependencies. See [[Packaging:Python#Automatic_Provides_with_a_standardized_name|above]] for more information.
|-
|python2_sitelib||/usr/lib/python2.X/site-packages||Where pure python2 modules are installed
|-
|python2_sitearch||/usr/lib64/python2.X/site-packages on x86_64<BR>/usr/lib/python2.X/site-packages on x86||Where python2 extension modules that are compiled C are installed
|-
|-
|python3_sitelib||/usr/lib/python3.X/site-packages||Where pure python3 modules are installed
|python3_sitelib||/usr/lib/python3.X/site-packages||Where pure python3 modules are installed
Line 166: Line 158:
|-
|-
|python3_version_nodots|| 3X ||Defined in python3-devel.  Useful when listing files explicitly in %files section , such as %{python3_sitelib}/foo/*.cpython-%{python3_version_nodots}.pyo
|python3_version_nodots|| 3X ||Defined in python3-devel.  Useful when listing files explicitly in %files section , such as %{python3_sitelib}/foo/*.cpython-%{python3_version_nodots}.pyo
|-
|py2_build|| %{__python2} setup.py build ... || Various flags are added, see /usr/lib/rpm/macros.d/macros.python2 for details and similar macros. Define %py_setup_args to pass custom command line arguments to setup.py.
|-
|py2_install|| %{__python2} setup.py install --skip-build ... ||
|-
|-
|py3_build|| %{__python3} setup.py build ... || Various flags are added, see /usr/lib/rpm/macros.d/macros.python3 for details and similar macros. Define %py_setup_args to pass custom command line arguments to setup.py.
|py3_build|| %{__python3} setup.py build ... || Various flags are added, see /usr/lib/rpm/macros.d/macros.python3 for details and similar macros. Define %py_setup_args to pass custom command line arguments to setup.py.
Line 182: Line 170:


{{admon/note|The generic %{_python} macros|
{{admon/note|The generic %{_python} macros|
The unversioned macros <code>%{__python}</code>, <code>%{python_sitelib}</code>, and <code>%{python_sitearch}</code> are generic macros that point to or use <code>/usr/bin/python</code>.  Since <code>/usr/bin/python</code> invocation is prohibited, you MUST NOT use these macros without manually setting <code>%{__python}</code> to either <code>%{__python2}</code> or <code>%{__python3}</code>.
The unversioned macros <code>%{__python}</code>, <code>%{python_sitelib}</code>, and <code>%{python_sitearch}</code> are generic macros that point to or use <code>/usr/bin/python</code>.  Since <code>/usr/bin/python</code> invocation is prohibited, you MUST NOT use these macros without manually setting <code>%{__python}</code> to either <code>%{__python3}</code>.
 
Instead, you SHOULD use <code>%{__python2}</code>, <code>%{python2_sitelib}</code>, and <code>%{python2_sitearch}</code> to explicitly reference the python2 interpreter.
}}
}}


During <code>%install</code> or when listing <code>%files</code> you can use the <code>python2_sitearch</code> and <code>python2_sitelib</code> macros to specify where the installed modules are to be found.  For instance:
During <code>%install</code> or when listing <code>%files</code> you can use the <code>python3_sitearch</code> and <code>python3_sitelib</code> macros to specify where the installed modules are to be found.  For instance:


<pre>
<pre>
%files
%files
# A pure python2 module
# A pure python3 module
%{python2_sitelib}/foomodule/
%{python3_sitelib}/foomodule/
# A compiled python2 extension module
%{python2_sitearch}/barmodule/
# A compiled python3 extension module
# A compiled python3 extension module
%{python3_sitearch}/bazmodule/
%{python3_sitearch}/barmodule/
</pre>
</pre>


Use of the macros has several benefits:
Use of the macros has several benefits:
* It ensures that the packages are installed correctly on multilib architectures.
* It ensures that the packages are installed correctly on multilib architectures.
* Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if <code>python2_sitelib</code> moves into <code>%{_datadir}</code>).
* Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if <code>python3_sitelib</code> moves into <code>%{_datadir}</code>).


== Files to include ==
== Files to include ==
When packaging python modules, several types of files are included:
When packaging python modules, several types of files are included:
* *.py source files because they are used when generating tracebacks.
* *.py source files because they are used when generating tracebacks.
* *.pyc and *.pyo byte compiled files.
* *.pyc byte compiled files.
** Python will try to create them at runtime if they don't exist which leads to spurious SELinux AVC denials in the logs.
** Python will try to create them at runtime if they don't exist which leads to spurious SELinux AVC denials in the logs.
** If the system administrator invokes python with -OO, .pyos will be created with no docstrings.  This can break some programs.
** If the system administrator invokes python with -OO, they will be created with no docstrings.  This can break some programs.
* *.egg-info files or directories.  If these are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at runtime.
* *.egg-info files or directories.  If these are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at runtime.


Line 214: Line 198:


Packagers SHOULD NOT simply glob everything under the sitelib or sitearch directories.  The following SHOULD NOT be used:
Packagers SHOULD NOT simply glob everything under the sitelib or sitearch directories.  The following SHOULD NOT be used:
* <code>%{python2_sitelib}/*</code>
* <code>%{python2_sitearch}/*</code>
* <code>%{python3_sitelib}/*</code>
* <code>%{python3_sitelib}/*</code>
* <code>%{python3_sitearch}/*</code>
* <code>%{python_sitelib}/*</code>
* <code>%{python_sitearch}/*</code>


And packages MUST NOT include the <code>__pycache__</code> directory (see below), so the following MUST NOT be used:
And packages MUST NOT include the top-level <code>__pycache__</code> directory (see below).
* <code>%{python3_sitearch}/*</code>


== Byte compiling ==
== Byte compiling ==


Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run.  These files are saved in files with the extension of .pyc (compiled python) or .pyo (optimized compiled python). With current versions of python 3, these files will be located inside a directory named <code>__pycache__</code>.  Older versions of python will simply place them alongside the .py files.
Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run.  These files are saved in files with the extension of .pyc (compiled python). These files will be located inside a directory named <code>__pycache__</code>.  


The .pyc and .pyo files contain byte code that is portable across OSes.  If you do not include them in your packages, python will try (and generally fail) to create them when the user runs the program.  If the system administrator runs the program, then the files will be successfully written, causing stray .pyc and .pyo files which will not be removed when the package is removed.  To prevent that the byte compiled files need to be compiled and included in the <code>%files</code> section.  Normally, byte compilation is done for you by the <code>brp-python-bytecompile</code> script. This script runs after the <code>%install</code> section of the spec file has been processed and byte compiles any .py files that it finds (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the <code>%{buildroot}</code> in them).
The .pyc files contain byte code that is portable across OSes.  If you do not include them in your packages, python will try (and generally fail) to create them when the user runs the program.  If the system administrator runs the program, then the files will be successfully written, causing stray .pyc files which will not be removed when the package is removed.  To prevent that the byte compiled files need to be compiled and included in the <code>%files</code> section.  Normally, byte compilation is done for you by the <code>brp-python-bytecompile</code> script. This script runs after the <code>%install</code> section of the spec file has been processed and byte compiles any .py files that it finds in <code>%{python3_sitelib}</code> or <code>%{python3_sitearch}</code> (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the <code>%{buildroot}</code> in them).


You must include in your package the .pyc and .pyo files.  If the build process creates a __pycache__ directory in a subdirectory of  %{python3_sitearch} or %{python3_sitelib}, you must also include all items in the __pycache__ directory.  You MUST NOT include the directories %{python3_sitearch}/__pycache__ or %{python3_sitelib}/__pycache__ because they are already owned by the python3-libs package.
You must include the .pyc files in your package.  If the build process creates a __pycache__ directory in a subdirectory of  %{python3_sitearch} or %{python3_sitelib}, you must also include all items in the __pycache__ directory.  You MUST NOT include the directories %{python3_sitearch}/__pycache__ or %{python3_sitelib}/__pycache__ because they are already owned by the python3-libs package.


All that you need to do is include the files in the <code>%files</code> section (replacing %{python3_sitelib} with the appropriate macro for your package):
All that you need to do is include the files in the <code>%files</code> section (replacing %{python3_sitelib} with the appropriate macro for your package):
Line 244: Line 228:
</pre>
</pre>


{{admon/warning|Avoid INSTALLED_FILES|python's distutils has an <code>INSTALLED_FILES</code> feature that lists which files are installed when you run <code>%py_install</code>.  Do not use it for packaging as that will not list the directories which need to be specified in the <code>%files</code> section as well. Using globs in the <code>%files</code> section is simpler and safer.}}
{{admon/warning|Avoid INSTALLED_FILES|python's distutils has an <code>INSTALLED_FILES</code> feature that lists which files are installed when you run <code>%py3_install</code>.  Do not use it for packaging as that will not list the directories which need to be specified in the <code>%files</code> section as well. Using globs in the <code>%files</code> section is simpler and safer.}}
 
{{admon/warning|Including egg info|When you run <code>%py_install</code> in any current Fedora, distutils generates a <code>.egg-info</code> file with metadata about the python module that is installed.  These files need to be included as well. (See [[#Packaging_eggs_and_setuptools_concerns|Packaging:Python_Eggs]] )}}
 
=== Optimization ===
 
Fedora packages running with 2 MUST NOT invoke python with the <code>-OO</code> option or set the environment variable <code>PYTHONOPTIMIZE</code> to 2 or greater.  (Using -O or <code>PYTHONOPTIMIZE</code> less than 2 is fine, though unnecessary.)


Similarly, any <code>.pyo</code> shipped in a Fedora package for Python 2 MUST NOT have been byte compiled using optimization level 2 or higher.
{{admon/warning|Including egg info|When you run <code>%py3_install</code> in any current Fedora, distutils generates a <code>.egg-info</code> file or directory with metadata about the python module that is installed.  These files need to be included as well. (See [[#Packaging_eggs_and_setuptools_concerns|Packaging:Python_Eggs]] )}}


=== Manual byte compilation ===
=== Manual byte compilation ===
For more details on the internals of byte compilation, please see [[Packaging:Python_Appendix#Manual_byte_compilation|the appendix]].
For more details on the internals of byte compilation, please see [[Packaging:Python_Appendix#Manual_byte_compilation|the appendix]].


== Common SRPM vs split SRPMs ==
== Example Python spec file ==
 
Many times when you package a python module you will want to create a module for python2 and a module for python3.  Both versions should be built from the same SRPM.  An exception to this would be if the two versions are distributed as separate archives and do not follow the same release schedule.
 
{{admon/warning|Use of the <code>%python_provide</code> macro|When building more than once from the same spec file, you must not have a <code>%files</code> section. Instead you must build one subpackage for each python runtime and use the <code>%python_provide</code> macro as shown in the example specfile below.}}
 
== Example common spec file ==


The following is a very simple spec file for a module building for both python2
The following is a very simple spec file for a Python module.
and python3.  It builds both versions in the same directory; this is possible
because the build products for different versions of Python usually do not conflict.
 
There are cases where it is not possible to build in a single directory.  Most
commonly this happens when the sources are modified during the build process to
convert them from python2 to python3 using the the <code>2to3</code> tool.  In
that case, please see [[Packaging:Python_Appendix#Using_separate_build_directories|the appendix]].
 
As you can see in the <code>%install</code> section below, the order in which
you do the python2 versus python3 install can sometimes matter.  You need to be
aware of when the install is writing to the same file in both packages (in this
example, a script in <code>%{_bindir}</code> and make sure that you're getting
the version you expect.


<pre>
<pre>
Line 293: Line 252:


BuildArch:      noarch
BuildArch:      noarch
BuildRequires:  python2-devel python3-devel
BuildRequires:  python3-devel


%description
%description
An python module which provides a convenient example.
%package -n python2-%{srcname}
Summary:        %{summary}
%{?python_provide:%python_provide python2-%{srcname}}
%description -n python2-%{srcname}
An python module which provides a convenient example.
An python module which provides a convenient example.


Line 318: Line 270:


%build
%build
%py2_build
%py3_build
%py3_build


%install
%install
# Must do the python2 install first because the scripts in /usr/bin are
# overwritten with every setup.py install, and in general we want the
# python3 version to be the default.
# If, however, we're installing separate executables for python2 and python3,
# the order needs to be reversed so the unversioned executable is the python2 one.
%py2_install
%py3_install
%py3_install


%check
%check
%{__python2} setup.py test
%{__python3} setup.py test
%{__python3} setup.py test


# Note that there is no %%files section for the unversioned python module if we are building for several python runtimes
# Note that there is no %%files section for the unversioned python module
%files -n python2-%{srcname}
%license COPYING
%doc README.rst
%{python2_sitelib}/%{srcname}/
 
%files -n python3-%{srcname}
%files -n python3-%{srcname}
%license COPYING
%license COPYING
%doc README.rst
%doc README.rst
%{python3_sitelib}/%{srcname}/
%{python3_sitelib}/%{srcname}/
%{python3_sitelib}/%{srcname}-*.egg-info/
%{_bindir}/sample-exec
%{_bindir}/sample-exec


%changelog
%changelog
</pre>
</pre>
== Avoiding collisions between the python 2 and python 3 stacks ==
The python 2 and python 3 stacks are intended to be fully-installable in parallel. When generalizing the package for both python 2 and python 3, it is important to ensure that two different built packages do not attempt to place different payloads into the same path.
=== Executables in <code>/usr/bin</code> ===
Many existing python packages install executables into <code>/usr/bin</code>.
For example if we have a <code>console_scripts</code> in a <code>setup.py</code> shared between
python 2 and python 3 builds: these will spit out files in <code>/usr/bin/</code>,
and these will collide.
For example <code>python-coverage</code> has a <code>setup.py</code> that contains:
<pre>
    entry_points = {
        'console_scripts': [
            'coverage = coverage:main',
            ]
        },
</pre>
which thus generates a <code>/usr/bin/coverage</code> executable (this is a python
script that runs another python script whilst generating code-coverage
information on the latter).
Similarly for the 'scripts' clause; see e.g. <code>python-pygments</code>:
<code>Pygments-1.1.1/setup.py</code> has:
<pre>
    scripts = ['pygmentize'],
</pre>
which generates a <code>/usr/bin/pygmentize</code> (this is a python script that leverages the pygments syntax-highlighting module, giving a simple command-line interface for generating syntax-highlighted files)
If the executables provide the same functionality independent of whether they are run on top of Python 2 or Python 3, then only the Python 3 version of the executable should be packaged.  Transitioning from python2 to python3 is left to individual package maintainers except for packages in Fedora's critical path.  For these, we want to port to python3 versions in the same Fedora release if possible.
Examples of this:
* <code>/usr/bin/pygmentize</code> ought to generate the same output regardless of whether it's implemented via Python 2 or Python 3, so only one version needs to be shipped.
If the executables provide different functionality for Python 2 and Python 3, then both versions should be packaged.
Examples of this:
* <code>/usr/bin/coverage</code> runs a python script, augmenting the interpreter with code-coverage information.  Given that the interpreter itself is the thing being worked with, it's reasonable to package both versions of the executable.
* <code>/usr/bin/bpython</code> augments the interpreter with a "curses" interface.  Again, it's reasonable to package both versions of this.
* <code>/usr/bin/easy_install</code> installs a module into one of the Python runtimes: we need a version for each runtime.
=== Naming ===
Many executables already contain a "-MAJOR.MINOR" suffix, for example <code>/usr/bin/easy_install-3.4</code>.  These obviously can be used as-is, as they won't conflict.
For other executables, the general rule is:
* If only one executable is to be shipped, then it owns its own slot and should use /usr/bin/python3 from Fedora 22 on.
* If executables are to be shipped for both python 2 and python 3:
** Both python 2 and python 3 variants must provide symlinks with a '-X' and '-X.Y' suffix (python runtime major version, or python runtime major.minor version), unless upstream already provides appropriately versioned executables without the dash.
** The unversioned executable '''must''' be the python2 version.
** For example, the python3 version of "coverage" '''must''' ship executables <code>/usr/bin/coverage-3</code> and <code>/usr/bin/coverage-3.4</code> (assuming python3 is currently version 3.4), while the python2 version '''must''' provide <code>/usr/bin/coverage</code>, <code>/usr/bin/coverage-2</code> and <code>/usr/bin/coverage-2.7</code> (assuming python2 version 2.7).
** For compatibility packages, the Python version is appended *after* the specific package version, for example <code>/usr/bin/coverage-v1.2-3</code> and <code>/usr/bin/coverage-v1.2-3.4</code> for python3-coverage1.2 compat package.
See [http://lists.fedoraproject.org/pipermail/devel/2010-January/129217.html this thread] and a newer thread [https://lists.fedoraproject.org/pipermail/packaging/2014-December/010360.html] for discussions of this.


== Packaging eggs ==
== Packaging eggs ==
Line 408: Line 294:
== Reviewer checklist ==
== Reviewer checklist ==
The following briefly summarizes the guidelines for reviewers to go over:
The following briefly summarizes the guidelines for reviewers to go over:
* '''Must''': If you build for more than one python runtime you must use the <code>%python_provide</code> macro.
* '''Must''': Python modules must be built from source.  They cannot simply drop an egg or whl from upstream into the proper directory. (See [[Packaging:Guidelines#No_inclusion_of_pre-built_binaries_or_libraries| prebuilt binaries Guidelines]] for details)
* '''Must''': If you build for a single python runtime you must add <code>%python_provide python-$module</code> so that the current default python is provided from the unversioned python package.
* '''Must''': Python modules must be built from source.  They cannot simply drop an egg from upstream into the proper directory. (See [[Packaging:Guidelines#No_inclusion_of_pre-built_binaries_or_libraries| prebuilt binaries Guidelines]] for details)
* '''Must''': Python modules must not download any dependencies during the build process.
* '''Must''': Python modules must not download any dependencies during the build process.
* '''Must''': When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
* '''Must''': When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
* '''Must''': When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
* '''Must''': When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
* '''Should''': If you build a python module you should use the <code>%python_provide</code> macro.
* '''Should''': A package which is used by another package via an egg interface should provide egg info.
* '''Should''': A package which is used by another package via an egg interface should provide egg info.


[[Category:Packaging guidelines]]
[[Category:Packaging guidelines]]
[[Category:Python]]
[[Category:Python]]

Revision as of 17:21, 13 September 2018

Python Version Support

In Fedora we have multiple Python runtimes, one for each supported major Python release. At this point that's one for python3.x and one for python2.7 However the python2 stack will be removed from Fedora and is deprecated. Upstream support for the python2 interpreter officially ends in 2020. If a piece of software supports python3, it MUST be packaged for python3. Software using python2 MUST NOT be newly packaged without FESCo exception.

For guidelines on maintaining already existing python2 packages, see the appendix.

Multiple Python Runtimes

For backwards compatibility, /usr/bin/python is, if it is installed, a symbolic link to /usr/bin/python2.

Packages in Fedora MUST NOT use /usr/bin/python. Instead packages for Python 3 MUST use /usr/bin/python3 (even if upstream supports both Python 2 and 3). As a result of that /usr/bin/python (as well as /usr/bin/env python and similar) MUST NOT be used in shebang lines or as a dependency of a package. As of Fedora 30, all uses of unversioned python executables in shebang lines will fail the build. These shebangs MUST be fixed. If it is necessary to disable the checks, please see the information in Packaging:Guidelines#Shebang_lines.

All python runtimes have a virtual provide for python(abi) = $MAJOR-$MINOR. For example, the Python 3.7 runtime package has: 

 $ rpm -q --provides python3 |grep -i abi
 python(abi) = 3.7

Python modules using these runtimes should have a corresponding "Requires" line on the python runtime that they are used with. This is done automatically for files below /usr/lib[^/]*/python${PYVER}

Mirroring the policy for regular packages, the Python-version-specific subpackages of your package MUST NOT be removed in a release branch of Fedora.

Naming

The source package for a Python library MUST be named with the python- prefix. A built package however must include the Python major version in the name, using the python3- prefix. This is accomplished by adding a subpackage. See example bellow.

This rule does not apply to applications.

Dependencies

Packages building for Python 3 will need BuildRequires: python3-devel.

Packages MUST NOT have dependencies (either build-time or runtime) on packages named with the unversioned python- prefix. Dependencies on Python packages instead MUST use names beginning with python3-.


Automatically generated dependencies

Packages MAY use the automatic Python dependency generator. This generator uses upstream egg/dist metadata (such as setuptool's install_requires) to determine what the package should depend on. The generator parses the installed metadata from /usr/lib(64)?/pythonX.Y/site-packages/<module>-<version>-pyX.Y.(egg|dist)-info/requires.txt, so it will not work with software that uses plain distutils.

To enable this feature, add: 

%{?python_enable_dependency_generator}

Although this statement can be used anywhere in the spec, we recommend putting it just before the main package's %description declaration. This generates run time requires in the form of pythonX.Ydist(foo). If the generated dependencies are not accurate, additional ones can still be added manually. To remove some, a packager MAY modify upstream-provided metadata (usually specified in the setup.py file) in the %prep section of the specfile or fall back to filtering those dependencies.

The packager MUST inspect the generated requires for correctness. All dependencies MUST be resolvable within the targeted Fedora version.

As an example, the upstream notebook package has (as of version 5.6.0): 

install_requires = [
    'jinja2',
    'tornado>=4',
    'pyzmq>=17',
    'ipython_genutils',
    'traitlets>=4.2.1',
    'jupyter_core>=4.4.0',
    'jupyter_client>=5.2.0',
    'nbformat',
    'nbconvert',
    'ipykernel',
    'Send2Trash',
    'terminado>=0.8.1',
    'prometheus_client'
],

And the resulting python3-notebook dependencies: 

python3.7dist(ipykernel)
python3.7dist(ipython-genutils)
python3.7dist(jinja2)
python3.7dist(jupyter-client) >= 5.2.0
python3.7dist(jupyter-core) >= 4.4.0
python3.7dist(nbconvert)
python3.7dist(nbformat)
python3.7dist(prometheus-client)
python3.7dist(pyzmq) >= 17
python3.7dist(send2trash)
python3.7dist(terminado) >= 0.8.1
python3.7dist(tornado) >= 4
python3.7dist(traitlets) >= 4.2.1

This generator will most likely be enabled by default in the future. If a packager wishes to explicitly opt out of the generator because the upstream metadata are not applicable, a packager MUST NOT assume that the generator won't be used simply because it is not enabled explicitly. Rather, they SHOULD opt out explicitly by adding: 

%{?python_disable_dependency_generator}

Provides

Using a fictional module named "example", the subpackage containing the python3 version must provide python3-example. This is of course always the case if the subpackage is named python3-example (as in the examples below). If the subpackage has some other name then then Provides: python3-example must be added explicitly (but see the %python_provide macro below).


The %python_provide macro

In order to make the switch from Python 2 to Python 3 automatic, all packages that provide python3-%{srcname} (for any %{srcname}) SHOULD use the %python_provide macro with the package name, for example: 

  %{?python_provide:%python_provide python3-%{srcname}}

This eases distribution-wide renaming of Python packages. (For example, in the future a virtual provide of python-%{srcname} might become appropriate for Python 3 libraries. In that case, %python_provide will be changed to add it.) Packages that do not include this macro would need to be adapted to such changes manually.

Automatic Provides with a standardized name

When building a Python package, RPM looks for .dist-info and .egg-info files or directories in the %files sections of all packages. If one or more are found, RPM parses them to find the standardized name (i.e. dist name, name on PyPI) of the packaged software, and then automatically creates two Provides: tags in the following format: 

Provides: python3.Ydist(CANONICAL_STANDARDIZED_NAME)
Provides: python3dist(CANONICAL_STANDARDIZED_NAME)

The 3.Y is the Python version used (usually 3.6 and higher), and between the parentheses is the name of the software in a canonical format used by Python tools and services such as setuptools, pip and PyPI. The canonical name is obtained by switching the standardized name to lower case and converting all runs of non-alphanumeric characters to single “-” characters. Example: “The $$$ Tree” becomes “the-tree”.

Requires and BuildRequires with standardized names

These Provides tags can be used to list Requires and BuildRequires of a package using the standardized names (i.e. dist name, name on PyPI) of Python modules. To make it easier, you can use the %{py3_dist} macro that accept one or more parameters: the standardized name(s) of the desired Python software. It will convert the name(s) to the canonical format and create the proper python3dist(...) tag(s).

In addition, you can use the %{py_dist_name} macro that simply transforms any standardized name to the canonical format.

For example: 

BuildRequires: %{py3_dist PyMySQL} >= 0.7.5
# => BuildRequires: python3dist(pymysql) >= 0.7.5

Requires: %{py3_dist virtualenv pyPEG2}
# => Requires: python3dist(virtualenv) python3dist(pypeg2)

%{py_dist_name 0-._.-._.-._.-._.-._.-._.-0}
# => 0-0

Source Files from PyPI

When packaging software which is available from PyPI, you can make use of the %pypi_source macro. This macro accepts from zero to three arguments and evaluates to an appropriate URL for the source file on PyPI. The arguments are:

  1. The name of the PyPI project. Defaults to %srcname if defined, or to %pypi_name if defined, or to %name (the package name).
  2. The version of the PyPI project. Defaults to %version (the package version).
  3. The file extension to use. Defaults to tar.gz.

In most cases it is not necessary to specify any arguments.

Macros

The following macros are defined for you in all supported Fedora and EPEL releases:

Macro Normal Definition Notes
__python /usr/bin/python Prohibited (see note below)
__python3 /usr/bin/python3 Python 3 interpreter
python_provide (Lua script) Given a package name, evaluates to either Provides: python-example or nothing at all depending on the Python version. See here for an example.
python3_sitelib /usr/lib/python3.X/site-packages Where pure python3 modules are installed
python3_sitearch /usr/lib64/python3.X/site-packages on x86_64
/usr/lib/python3.X/site-packages on x86		 Where python3 extension modules that are compiled C are installed
py3_dist (Lua script) Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format, and evaluates to python3dist(CANONICAL_NAME), which is useful when listing dependencies. See above for more information.
py_byte_compile (script) Defined in python3-devel. See the byte compiling section for usage
python3_version 3.X Defined in python3-devel. Useful when running programs with Python version in filename, such as nosetest-%{python3_version}
python3_version_nodots 3X Defined in python3-devel. Useful when listing files explicitly in %files section , such as %{python3_sitelib}/foo/*.cpython-%{python3_version_nodots}.pyo
py3_build %{__python3} setup.py build ... Various flags are added, see /usr/lib/rpm/macros.d/macros.python3 for details and similar macros. Define %py_setup_args to pass custom command line arguments to setup.py.
py3_install %{__python3} setup.py install --skip-build ...
py_dist_name (Lua script) Given a standardized name (i.e. dist name, name on PyPI) of Python software, it will convert it to a canonical format. See above for more information.
pypi_source (Lua script) Evaluates to the appropriate URL for the package. See above for more information.
The generic %{_python} macros
The unversioned macros %{__python}, %{python_sitelib}, and %{python_sitearch} are generic macros that point to or use /usr/bin/python. Since /usr/bin/python invocation is prohibited, you MUST NOT use these macros without manually setting %{__python} to either %{__python3}.

During %install or when listing %files you can use the python3_sitearch and python3_sitelib macros to specify where the installed modules are to be found. For instance: 

%files
# A pure python3 module
%{python3_sitelib}/foomodule/
# A compiled python3 extension module
%{python3_sitearch}/barmodule/

Use of the macros has several benefits:

  • It ensures that the packages are installed correctly on multilib architectures.
  • Using these macros instead of hardcoding the directory in the specfile ensures your spec remains compatible with the installed python version even if the directory structure changes radically (for instance, if python3_sitelib moves into %{_datadir}).

Files to include

When packaging python modules, several types of files are included:

  • *.py source files because they are used when generating tracebacks.
  • *.pyc byte compiled files.
    • Python will try to create them at runtime if they don't exist which leads to spurious SELinux AVC denials in the logs.
    • If the system administrator invokes python with -OO, they will be created with no docstrings. This can break some programs.
  • *.egg-info files or directories. If these are generated by the module's build scripts they must be included in the package because they might be needed by other applications and modules at runtime.

The source files MUST be included in the same package as the byte compiled versions.

Packagers SHOULD NOT simply glob everything under the sitelib or sitearch directories. The following SHOULD NOT be used:

  • %{python3_sitelib}/*
  • %{python3_sitearch}/*
  • %{python_sitelib}/*
  • %{python_sitearch}/*

And packages MUST NOT include the top-level __pycache__ directory (see below).

Byte compiling

Python will automatically try to byte compile files when it runs in order to speed up startup the next time it is run. These files are saved in files with the extension of .pyc (compiled python). These files will be located inside a directory named __pycache__.

The .pyc files contain byte code that is portable across OSes. If you do not include them in your packages, python will try (and generally fail) to create them when the user runs the program. If the system administrator runs the program, then the files will be successfully written, causing stray .pyc files which will not be removed when the package is removed. To prevent that the byte compiled files need to be compiled and included in the %files section. Normally, byte compilation is done for you by the brp-python-bytecompile script. This script runs after the %install section of the spec file has been processed and byte compiles any .py files that it finds in %{python3_sitelib} or %{python3_sitearch} (this recompilation puts the proper filesystem paths into the modules otherwise tracebacks would include the %{buildroot} in them).

You must include the .pyc files in your package. If the build process creates a __pycache__ directory in a subdirectory of  %{python3_sitearch} or %{python3_sitelib}, you must also include all items in the __pycache__ directory. You MUST NOT include the directories %{python3_sitearch}/__pycache__ or %{python3_sitelib}/__pycache__ because they are already owned by the python3-libs package.

All that you need to do is include the files in the %files section (replacing %{python3_sitelib} with the appropriate macro for your package): 

%files
%{python3_sitelib}/foo/

or, if the python code installs directly into %{python3_sitelib}: 

%files
%{python3_sitelib}/foo.py
%{python3_sitelib}/__pycache__/*
Avoid INSTALLED_FILES
python's distutils has an INSTALLED_FILES feature that lists which files are installed when you run %py3_install. Do not use it for packaging as that will not list the directories which need to be specified in the %files section as well. Using globs in the %files section is simpler and safer.
Including egg info
When you run %py3_install in any current Fedora, distutils generates a .egg-info file or directory with metadata about the python module that is installed. These files need to be included as well. (See Packaging:Python_Eggs )

Manual byte compilation

For more details on the internals of byte compilation, please see the appendix.

Example Python spec file

The following is a very simple spec file for a Python module. 

%global srcname example

Name:           python-%{srcname}
Version:        1.2.3
Release:        1%{?dist}
Summary:        An example python module

License:        MIT
URL:            https://pypi.python.org/pypi/%{srcname}
Source0:        %pypi_source

BuildArch:      noarch
BuildRequires:  python3-devel

%description
An python module which provides a convenient example.


%package -n python3-%{srcname}
Summary:        %{summary}
%{?python_provide:%python_provide python3-%{srcname}}

%description -n python3-%{srcname}
An python module which provides a convenient example.


%prep
%autosetup -n %{srcname}-%{version}

%build
%py3_build

%install
%py3_install

%check
%{__python3} setup.py test

# Note that there is no %%files section for the unversioned python module
%files -n python3-%{srcname}
%license COPYING
%doc README.rst
%{python3_sitelib}/%{srcname}/
%{python3_sitelib}/%{srcname}-*.egg-info/
%{_bindir}/sample-exec

%changelog

Packaging eggs

Please see the Python eggs guidelines for information specific to Python eggs.

Reviewer checklist

The following briefly summarizes the guidelines for reviewers to go over:

  • Must: Python modules must be built from source. They cannot simply drop an egg or whl from upstream into the proper directory. (See prebuilt binaries Guidelines for details)
  • Must: Python modules must not download any dependencies during the build process.
  • Must: When building a compat package, it must install using easy_install -m so it won't conflict with the main package.
  • Must: When building multiple versions (for a compat package) one of the packages must contain a default version that is usable via "import MODULE" with no prior setup.
  • Should: If you build a python module you should use the %python_provide macro.
  • Should: A package which is used by another package via an egg interface should provide egg info.
Retrieved from "https://fedoraproject.org/w/index.php?title=Packaging:Python&oldid=524856"