User:Toshio/AutoProvidesAndRequiresFilteringDraft: Difference between revisions

Revision as of 21:13, 25 February 2013

Summary

EPEL Differences
As of rpm-4.9 (Fedora 15), rpm has a standard method to enable filtering. This page documents that. EPEL5 and 6 do not have a recent enough version of rpm to follow these guidelines. See LINK_TO_OLD_GUIDELINES_IN_EPEL_NAMESPACE if your package is to be built there as well. (Toshio: Also merge or link the EPEL5 guidelines: EPEL:Packaging#Perl_Provides_and_Requires_Filtering on the EPEL6 page.)

The auto requires and provides system contained in RPM is quite useful; however, it sometimes picks up "private" package capabilities that shouldn't be advertised as global, things that are "just wrong", or things prohibited by policy (e.g. deps from inside %{_docdir}).

For example:

Various "plugin" packages (e.g. Pidgin, Perl, Apache, KDE) are marked as "providing" private shared libraries outside the system path.
Files in %{_docdir} are routinely scanned, and can trigger prov/req when this is explicitly forbidden by policy.

This Guideline describes how to filter provides and requires on Fedora.

MUST: Packages must not provide RPM dependency information when that information is not global in nature, or are otherwise handled (e.g. through a virtual provides system). e.g. a plugin package containing a binary shared library must not "provide" that library unless it is accessible through the system library paths.
MUST: When filtering automatically generated RPM dependency information, the filtering system implemented by Fedora must be used, except where there is a compelling reason to deviate from it.

Usage

Verify that the new method does not interfere with coloring
The old method had this warning: "They are not permitted in any other cases, because the macros interfere with the "coloring" of elf32/64 executables done internally by RPM to support multilib installs." I assume the new method doesn't have this limitation but should check with panu/rpm-team

Escaping backslashes
Rpm interprets backslashes as part of its parsing of spec files. If you need to include a backslash in a regular expression, you have to double escape it (//).

Location of macro invocation

It's strongly recommended that these filtering macros be invoked before %description, but after any other definitions. This will keep them in a consistent place across packages, and help prevent them from being mixed up with other sections.

Preventing files/directories from being scanned for deps (pre-scan filtering)

The macros %__requires_exclude_from and %__provides_exclude_from can be defined in a spec file to keep the dependency generator from scanning specific files or directories for deps. These macros should be defined with a regular expression that matches all of the directories or files. For instance:

# Do not check any files in docdir for requires
%global __requires_exclude_from ^%{_docdir}/.*$

# Do not check .so files in the python_sitelib directory
# or any files in the application's directory for provides
%global __provides_exclude_from ^(%{python_sitelib}/*.\\.so|%{_datadir}/myapp/.*)$

Note that this macro replaces the %filter_provides_in macro from the old filtering guidelines but it does not do the same thing. In particular:

The old macro could be invoked multiple times. This one will only use the regex defined last.
The old macro advised against anchoring the beginning of the regex (Using "^"). This macro recommends anchoring as it doesn't suffer from the compatibility problems of the old one.
With the old macro it was common to specify a directory name to match everything in a directory recursively. With the new macro you may need to specify .* because you should be anchoring your regular expressions.

Filtering provides and requires after scanning

In addition to preventing rpm from scanning files and directories for automatic dependency generation you can also tell rpm to discard a discovered dependency before it records the dependency in the rpm metadata. Use __requires_exclude and __provides_exclude for this. These macros should be defined as regular expressions. If an entry that rpm's automatic dependency generator created matches the regular expression then it will be filtered out of the requires or provides. For example:

# This might be useful if plugins are being picked up by the dependency generator
%global __provides_exclude ^libfoo-plugin.so.*$

# Something like this could be used to prevent excess deps from an
# example python script in %doc
%global __requires_exclude ^/usr/bin/python$

These macros serves a similar purpose to the old %filter_from_provides macro but it has a different implementation. In particular, that macro took sed expressions whereas this one needs a regular expression.

Simplified macros for common cases

In some cases, the filtering of extraneous Provides: is fairly generic to all packages which provide similar things. There are simple macros that setup filters correctly for those cases so that you can do the filtering with one line. If you need to filter a bit more than the simple macro provides, you still have the option to use the macros listed above.

Perl

Needs rewriting of the perl macro as well as here
This section needs to have the perl_default_filter macro updated to use the rpm-4.9 macros instead. Otherwise, we'll still need to worry about elf coloring and other cases of interference with the internal dependency generator.

Perl extension modules can be filtered using this macro:

%{?perl_default_filter}

This is equivalent to:

%filter_provides_in %{perl_vendorarch}/.*\\.so$ 
%filter_provides_in -P %{perl_archlib}/(?!CORE/libperl).*\\.so$ 
%filter_from_provides /perl(UNIVERSAL)/d; /perl(DB)/d 
%filter_provides_in %{_docdir} 
%filter_requires_in %{_docdir} 
%filter_setup

Examples

Pidgin plugin package

On a x86_64 machine, the pidgin-libnotify provides pidgin-libnotify.so()(64bit) which it shouldn't as this library is not inside the paths searched by the system for libraries. It's a private, not global, "provides" and as such must not be exposed globally by RPM.

To filter this out, we could use:

%global __provides_filter_from ^%{_libdir}/purple-2/.*\.so$

Arch-specific extensions to scripting languages

e.g. to ensure an arch-specific perl-* package won't provide or require things that it shouldn't, we could use an invocation as such:

# we don't want to provide private Perl extension libs
%{?perl_default_filter}

A recipe for python:

# we don't want to provide private python extension libs in either the python2 or python3 dirs
%global __provides_exclude_from ^(%{python_sitearch}|%{python3_sitearch})/.*\\.so$

%_docdir filtering

By policy, nothing under %_docdir is allowed to either "provide" or "require" anything. We can prevent this from happening by preventing anything under %_docdir from being scanned:

# we don't want to either provide or require anything from _docdir, per policy
%global __provides_exclude_from ^%{_docdir}/.*$
%global __requires_exclude_from ^%{_docdir}/.*$

Additional Information

Additional information about rpm-4.9's dependency generator can be found here: http://rpm.org/wiki/PackagerDocs/DependencyGenerator

@@ Line 1: / Line 1: @@
 == Summary ==
-{{admon/note|EPEL Differences|As of rpm-4.9 (Fedora 15), rpm has a standard method to enable filtering.  This page documents that.  EPEL5 and 6 do not have a recent enough version of rpm to follow these guidelines.  See LINK_TO_OLD_GUIDELINES_IN_EPEL_NAMESPACE if your package is to be built there as well.}}
+{{admon/note|EPEL Differences|As of rpm-4.9 (Fedora 15), rpm has a standard method to enable filtering.  This page documents that.  EPEL5 and 6 do not have a recent enough version of rpm to follow these guidelines.  See LINK_TO_OLD_GUIDELINES_IN_EPEL_NAMESPACE if your package is to be built there as well.  (Toshio: Also merge or link the EPEL5 guidelines: [[EPEL:Packaging#Perl_Provides_and_Requires_Filtering]] on the EPEL6 page.)}}
-This Guideline describes how to filter provides and requires on Fedora.
+The auto requires and provides system contained in RPM is quite useful; however, it sometimes picks up "private" package capabilities that shouldn't be advertised as global, things that are "just wrong", or things prohibited by policy (e.g. deps from inside <code>%{_docdir}</code>).
-* '''MUST:''' Packages must not provide RPM dependency information when that information is not global in nature, or are otherwise handled (e.g. through a virtual provides system).  e.g. a plugin package containing a binary shared library must not "provide" that library unless it is accessible through the system library paths.
-* '''MUST:''' When filtering automatically generated RPM dependency information, the filtering system implemented by Fedora must be used, except where there is a compelling reason to deviate from it.
-== Rationale ==
-The auto requires and provides system contained in RPM is quite useful; however, it often picks up "private" package capabilities that shouldn't be advertised as global, things that are "just wrong", or things prohibited by policy (e.g. deps from inside %{_docdir}).
 For example:
 * Various "plugin" packages (e.g. Pidgin, Perl, Apache, KDE) are marked as "providing" private shared libraries outside the system path.
-* Files in %{_docdir} are routinely scanned, and can trigger prov/req when this is explicitly forbidden by policy.
+* Files in <code>%{_docdir}</code> are routinely scanned, and can trigger prov/req when this is explicitly forbidden by policy.
-As it stands, filtering these auto-generated requires and provides is difficult and messy at best, and horribly deep magic in many cases; with little guidance on how to do it. This feature aims to make the following tasks easy:
-* preventing files/directories from being scanned for requires (pre-scan filtering)
+This Guideline describes how to filter provides and requires on Fedora.
-* preventing files/directories from being scanned for provides (pre-scan filtering)
-* removing items from the requires stream (post-scan filtering)
-* removing items from the provides stream (post-scan filtering)
-These macros are available in all non-EOL Fedora and RHEL6 or higher.
+* '''MUST:''' Packages must not provide RPM dependency information when that information is not global in nature, or are otherwise handled (e.g. through a virtual provides system).  e.g. a plugin package containing a binary shared library must not "provide" that library unless it is accessible through the system library paths.
+* '''MUST:''' When filtering automatically generated RPM dependency information, the filtering system implemented by Fedora must be used, except where there is a compelling reason to deviate from it.
-{{admon/note|EPEL|These filtering mechanisms will not work on RHEL 5 or older. For details on how to do Provides and Requires Filtering for EPEL 4 and 5, see [[EPEL:Packaging#Perl_Provides_and_Requires_Filtering]].}}
 == Usage ==
-These filtering macros '''MUST''' only be used with packages which meet one of the following criteria:
+{{admon/question|Verify that the new method does not interfere with coloring|The old method had this warning: "They are not permitted in any other cases, because the macros interfere with the "coloring" of elf32/64 executables done internally by RPM to support multilib installs."  I assume the new method doesn't have this limitation but should check with panu/rpm-team}}
-* Noarch packages
-* Architecture specific packages with no binaries in $PATH (e.g. /bin, /usr/bin, /sbin, /sbin) or libexecdir and no system libs in libdir. This includes all of the subpackages generated from the spec file.
-They are not permitted in any other cases, because the macros interfere with the "coloring" of elf32/64 executables done internally by RPM to support multilib installs.
+{{admon/note|Escaping backslashes|Rpm interprets backslashes as part of its parsing of spec files.  If you need to include a backslash in a regular expression, you have to double escape it (<code>//</code>).}}
 === Location of macro invocation ===
@@ Line 40: / Line 26: @@
 It's strongly recommended that these filtering macros be invoked before %description, but after any other definitions.  This will keep them in a consistent place across packages, and help prevent them from being mixed up with other sections.
-=== Pathnames ===
+=== Preventing files/directories from being scanned for deps (pre-scan filtering) ===
-Some rpm versions pass pathnames to these macros with the build root prepended; some do not. It is strongly recommended that your regular expressions not anchor the match at the beginning of the string (i.e. not use "^") so that they work regardless of whether the build root is present in the string or not.
-=== Preventing files/directories from being scanned for provides (pre-scan filtering) ===
+The macros <code>%__requires_exclude_from</code> and <code>%__provides_exclude_from</code> can be defined in a spec file to keep the dependency generator from scanning specific files or directories for deps.  These macros should be defined with a regular expression that matches all of the directories or files.  For instance:
-The '''%filter_provides_in''' macro is used to define the files or directories that should not be scanned for any "provides" information.  This macro may be safely invoked multiple times, and can handle regular expressions.  The -P flag can be passed to specify that a PCRE is being used.
-We can filter by regex:
 <pre>
-%filter_provides_in %{perl_vendorarch}/.*\.so$
+# Do not check any files in docdir for requires
-%filter_provides_in -P %{perl_archlib}/(?!CORE/libperl).*\.so$
+%global __requires_exclude_from ^%{_docdir}/.*$
-</pre>
-Or by anything matching, say, a directory:
+# Do not check .so files in the python_sitelib directory
-<pre>
+# or any files in the application's directory for provides
-%filter_provides_in %{_docdir}
+%global __provides_exclude_from ^(%{python_sitelib}/*.\\.so|%{_datadir}/myapp/.*)$
 </pre>
-=== Preventing files/directories from being scanned for requires (pre-scan filtering) ===
+Note that this macro replaces the <code>%filter_provides_in</code> macro from the old filtering guidelines but it does not do the same thing.  In particular:
+* The old macro could be invoked multiple times.  This one will only use the regex defined last.
-The '''%filter_requires_in''' macro is used to define the files or directories that should not be scanned for any "requires" information; it does for requires what the %filter_provides_in macro does for provides and is invoked in the same fashion.
+* The old macro advised against anchoring the beginning of the regex (Using "^").  This macro recommends anchoring as it doesn't suffer from the compatibility problems of the old one.
+* With the old macro it was common to specify a directory name to match everything in a directory recursively.  With the new macro you may need to specify <code>.*</code> because you should be anchoring your regular expressions.
-=== Removing items from the provides stream (post-scan filtering) ===
-Post-scan provides filtering is invoked through the '''%filter_from_provides'''.  This macro can be fed a sed expression to filter from the stream of auto-found provides.
+=== Filtering provides and requires after scanning ===
-For example, if we're finding that the auto-prov system is finding an incorrect provide, we can filter it:
+In addition to preventing rpm from scanning files and directories for automatic dependency generation you can also tell rpm to discard a discovered dependency before it records the dependency in the rpm metadata.  Use <code>__requires_exclude</code> and <code>__provides_exclude</code> for this.  These macros should be defined as regular expressions.  If an entry that rpm's automatic dependency generator created matches the regular expression then it will be filtered out of the requires or provides.  For example:
 <pre>
-%filter_from_provides /bad-provide/d
+# This might be useful if plugins are being picked up by the dependency generator
-</pre>
+%global __provides_exclude ^libfoo-plugin.so.*$
-Note that we should always specify this in terms of a regexp.
-=== Removing items from the requires stream (post-scan filtering) ===
-The '''%filter_from_requires''' macro is used to filter "requires"; it does for requires what the %filter_from_provides macro does for provides and is invoked in the same fashion.
-=== General filter setup ===
-The '''%filter_setup''' macro must be invoked after defining any specific overrides; this macro does all the heavy lifting of implementing the filtering desired:
-<pre>
+# Something like this could be used to prevent excess deps from an
-# ... filtering defines here
+# example python script in %doc
-%filter_setup
+%global __requires_exclude ^/usr/bin/python$
 </pre>
-These macros were not defined in EPEL5.  People wanting to share one spec file with Fedora and EPEL need to conditionalize use of the macros.  That can be done like this:
+These macros serves a similar purpose to the old <code>%filter_from_provides</code> macro but it has a different implementation.  In particular, that macro took sed expressions whereas this one needs a regular expression.
-<pre>
-%{?filter_setup:
-%filter_provides_in %{python_sitearch}.*\.so$
-%filter_setup
-}
-</pre>
 === Simplified macros for common cases ===
@@ Line 102: / Line 64: @@
 ==== Perl ====
+{{admon/question|Needs rewriting of the perl macro as well as here|This section needs to have the perl_default_filter macro updated to use the rpm-4.9 macros instead.  Otherwise, we'll still need to worry about elf coloring and other cases of interference with the internal dependency generator.}}
 Perl extension modules can be filtered using this macro:
@@ Line 124: / Line 89: @@
 === Pidgin plugin package ===
-On a x86_64 machine, the pidgin-libnotify provides pidgin-libnotify.so()(64bit), which it shouldn't, as this library is not inside the paths searched by the system for libraries; that is, it's a private, not global, "provides" and as such must not be exposed globally by RPM.
+On a x86_64 machine, the pidgin-libnotify provides <code>pidgin-libnotify.so()(64bit)</code> which it shouldn't as this library is not inside the paths searched by the system for libraries.  It's a private, not global, "provides" and as such must not be exposed globally by RPM.
 To filter this out, we could use:
 <pre>
-%{?filter_setup:
+%global __provides_filter_from ^%{_libdir}/purple-2/.*\.so$
-%filter_provides_in %{_libdir}/purple-2/.*\.so$
-%filter_setup
-}
 </pre>
@@ Line 146: / Line 108: @@
 A recipe for python:
 <pre>
-# we don't want to provide private python extension libs
+# we don't want to provide private python extension libs in either the python2 or python3 dirs
-%{?filter_setup:
+%global __provides_exclude_from ^(%{python_sitearch}|%{python3_sitearch})/.*\\.so$
-%filter_provides_in %{python_sitearch}/.*\.so$
-%filter_setup
-}
 </pre>
 === %_docdir filtering ===
-By policy, nothing under %_docdir is allowed to either "provide" or "require" anything.  We can prevent this from happening by preventing anything under %_docdir from being scanned:
+By policy, nothing under <code>%_docdir</code> is allowed to either "provide" or "require" anything.  We can prevent this from happening by preventing anything under <code>%_docdir</code> from being scanned:
 <pre>
 # we don't want to either provide or require anything from _docdir, per policy
-%{?filter_setup:
+%global __provides_exclude_from ^%{_docdir}/.*$
-%filter_provides_in %{_docdir}
+%global __requires_exclude_from ^%{_docdir}/.*$
-%filter_requires_in %{_docdir}
-%filter_setup
-}
 </pre>
+== Additional Information ==
+Additional information about rpm-4.9's dependency generator can be found here: http://rpm.org/wiki/PackagerDocs/DependencyGenerator
 [[Category:Packaging_guidelines_draft]]

Search