Software Collections (SCLs) give rpm packagers a means of packaging multiple versions of software for a single distribution. The version from the software collection does not interact with the system version. SCLs can provide backwards and forwards compat versions of a package compared to what's on the system.
Definitions
Term | Definition |
---|---|
SCL | A Software Collection |
SCL Metapackage (SRPM) | SRPM package that defines the SCL. |
SCL Metapackage (RPM) | RPM built from the SCL Metapackage SRPM. Defines the essential packages you get when you install the scl. |
SCL Runtime Package | Built from the SCL Metapackage SRPM. Contains the SCL filesystem and essential scripts (usually just enable ) used to work with this SCL.
|
SCL Build Package | Built from the SCL Metapackage SRPM. Contains macro definitions needed to build packages for the collection. |
SCL Package | Any binary RPM built for an SCL. |
SCL Approval
SCLs need to be approved on several levels:
- The SCL itself needs approval. At the moment we're rather strict about what can be an SCL so that we limit the number and types of SCLs. At least in this phase of getting SCLs into Fedora we want to avoid a proliferation of SCLs with only minor differences. This may be relaxed in the future.
- Each SCL Package needs to be approved so that it meets the packaging guidelines.
SCLs are approved by the FPC for now but we want to move it to some other body in the future. This is pending discussion by FPC, FESCo, and possibly the Environments and Stacks Working Group. SCL Packages are approved by packagers just like any other rpm package.
SCL Criteria
Size
The software ecosystem that an SCL targets must be of a moderate to large size. A language interpreter would satisfy this. A web framework would as well. In general, a single library would not. Use the standard backwards/forwards-compat packaging standards for those. (An exception to this would be alternate libc implementation, older boost versions and similar, very basic libraries).
In some cases, an application may also be deemed large enough to satisfy this requirement when the application itself is the major portion of an ecosystem. Servers that have client libraries that must be linked against (httpd with its modules and postgres which has programs linked against its client libraries are examples of this.)
Conflicts with non-SCLs and other SCLs
SCLs must not conflict with other packages on a filesystem level. They must follow the Conflicts Guidelines for use of the rpm Conflicts tag. They may Conflict with other packages in the use of tcp and udp ports. The use of conflicting ports does not have to be marked using the rpm Conflicts tag if the end user is able to modify the ports used via a config file.
Naming the SCL
The SCL and the SCL metapackage are known by the same name. This name must make it unique from non-scl packages. Thus it carries a prefix of scl-
. Following the prefix, the SCL must carry the name and version of the primary package being packaged. For instance, if this to be ruby at version 1.9.3 then the SCL metapackage would be named scl-ruby1.9.3
. All SCL packages within the SCL must carry the SCL metapackage base name as a prefix. So, for instance, the ruby interpreter package contained within the metapackage would be named scl-ruby1.9.3-ruby
.
SCL Metapackage
Every SCL must have an SRPM-level metapackage. This metapackage must build three binary rpm packages:
- One which has the same name as the SRPM metapackage. This RPM contains Requires to pull in the packages essential to this SCL.
- An
$SCLNAME-runtime
subpackage. This package needs to be installed if an end user intends to use the scl.
- A
-build
subpackage. This package contains rpm macros needed for building packages into the SCL. This subpackage is needed for Koji builds, but may be left out for local builds).
An example SCL metapackage follows:
%global scl ruby1.9.3 %scl_package %scl %_scl_prefix /opt/fedoraproject/scls %global install_scl 1 Summary: Package that installs ruby1.9.3 Name: %scl_name Version: 1 Release: 1%{?dist} License: GPLv2+ Requires: %{scl_prefix}js Requires: %{scl_prefix}rubygem-sqlite3 BuildRequires: scl-utils-build %description This is the main package for the ruby1.9.3 Software Collection. [Provide some useful info about this SCL.] %package runtime Summary: Package that handles %scl Software Collection Requires: scl-utils %description runtime Package shipping essential scripts to work with the ruby1.9.3 Software Collection. %package build Summary: Package shipping basic build configuration Requires: scl-utils-build %description build Package shipping essential configuration macros to build packages for the ruby1.9.3 Software Collection. %prep %setup -T -c %build %install rm -rf %{buildroot} mkdir -p %{buildroot}%{_scl_scripts}/root cat >> %{buildroot}%{_scl_scripts}/enable << EOF export PATH=%{_bindir}\${PATH:+:\${PATH}} export LD_LIBRARY_PATH=%{_libdir}\${LD_LIBRARY_PATH:+:\${LD_LIBRARY_PATH}} export MANPATH=%{_mandir}:\${MANPATH} EOF %scl_install %files %files runtime %scl_files %files build %{_root_sysconfdir}/rpm/macros.%{scl}-config %changelog * Fri Mar 30 2012 Bohuslav Kabrda <bkabrda@redhat.com> - 1-1 - Initial package.
Things to note here:
- The
-build
subpackage should include Requires: scl-utils-build.
- The enable scriptlet must contain certain path redefinitions, according to what packages SCL contains and what it is supposed to do. Most importantly, use these, if needed:
PATH=%{_bindir}\${PATH:+:\${PATH}}
to run SCL binariesLD_LIBRARY_PATH=%{_libdir}\${LD_LIBRARY_PATH:+:\${LD_LIBRARY_PATH}}
to properly link against SCL shared objectsMANPATH=%{_mandir}:\${MANPATH}
to be able to diplay manpages present in the SCLPKG_CONFIG_PATH=%{_libdir}/pkgconfig\${PKG_CONFIG_PATH:+:\${PKG_CONFIG_PATH}}
orPKG_CONFIG_PATH=%{_datadir}/pkgconfig\${PKG_CONFIG_PATH:+:\${PKG_CONFIG_PATH}}
to enable usingpkg-config
filesXDG_DATA_PATH=%{_datadir}\${XDG_DATA_PATH:+:\${XDG_DATA_PATH}}
to use systemtap tapsets from the SCL
- SCL metapackages must not be noarch. The filesystem directories contained in the metapackage may contain arch-specific paths.
- Among other things, the
%scl_install
macro creates a macro filemacros.%{scl}-config
(that ends up in the%{scl}-build
package). This file is always located at%{buildroot}%{_root_sysconfdir}/rpm/macros.%{scl}-config
. If you need to add some more macros specific for this SCL, add them into this file.
- Unlike in specfiles of normal SCL packages (see below), the metapackage doesn't need to conditionalize SCL specific macros, as it can only be used as a part of SCL (for example, instead of
Requires: %{?scl_prefix}foo
, use justRequires: %{scl_prefix}foo
- notice the missing questionmark, that makes the first macro conditional). - When user runs
yum install ruby193
, he expects that the whole SCL with all dependencies gets installed. Because of that, it should haveRequires
on all packages of that SCL, that are needed for the SCL to fulfil its purpose. E.g.yum install ruby193
definitely has to install Ruby 1.9.3, but it needn't install collection packages that were used to build the interpreter (and that are not needed at Ruby runtime).
Converting Packages for SCL
This section sums up the steps needed to take in order to convert a normal specfile to SCL specfile step by step. The resulting package should be buildable even without SCL - that means both without %{scl}-build
in the buildroot and with it. Advantage of this approach is, that both non-SCL packages like rubygem-foo
and SCL packages %{scl_prefix}rubygem_foo (depending on the SCL) can be built from the same SRPM, depending on the buildroot.
Converting Tags and Macro Definitions
Every SCL specfile must have %scl_package
macro specified (like any other macro, it should be conditionalized). This macro does this:
- Rewrites the standard path macros by prefixing them with
%{_scl_prefix}/%{scl}/root/
. E.g.%{_datadir}
changes (depending on value of%_scl_prefix
) from/usr/share
to e.g./opt/fedora/myscl/root/usr/share
. - Introduces some SCL specific macros (like
%pkg_name
,%scl_prefix
or%_root_*
set of macros, that contain values of the original RPM macros (e.g.%_root_datadir
contains/usr/share
).
One of the important macros is %pkg_name
, which represents the original package name - the %name
macro stands for the name with SCL prefix during SCL build. Therefore it is a good practice to also define %pkg_name
macro for non-SCL builds, to be able to use it consistently throughout the whole specfile.
So here is what the first two lines should look like:
%{?scl:%scl_package foo} %{!?scl:%global pkg_name %{name}}
Usual steps to adapt tag definitions for SCL builds are these:
- Name must be modified like this:
-Name: foo +Name: %{?scl_prefix}foo
- Requires and BuildRequires have to be considered carefully. These depend on what you are building/linking with and it is your decision as a packager. The only rule here is, that if building/linking with other SCL packages, their names must be also prefixed with conditionalized macro
%{?scl_prefix}
like this:
-Requires: bar +Requires: %{?scl_prefix}bar
- For example, since ifconfig is a binary, we're going to use its system version, which means not using
%scl_prefix
with it. rubygem-foo, on the other hand, has to be in the path of the Ruby interpreter which will be part of this SCL, therefore it needs to be used with%scl_prefix
. When depending on system packages, you should be very general in your requirements (avoid using versioned Requires on specific versions) and if you need a package that might be updated, you have to either add it to your SCL or be willing to rebuild the SCL when the system package updates.
Requires ifconfig -Requires: rubygem-foo = 1.0.0 +Requires: %{?scl_prefix}rubygem-foo = 1.0.0 BuildRequires: ifconfig -BuildRequires: rubygem-foo = 1.0.0 +BuildRequires: %{?scl_prefix}rubygem-foo = 1.0.0
- Obsoletes, Conflicts and BuildConflicts must always be prefixed with
%{?scl_prefix}
. This is extremely important, as the SCLs are often used for deploying new packages on older systems (that may contain old packages, now obsoleted by the new ones), but they shouldn't Obsolete or Conflict with the non-SCL RPMs installed on the system (that's the idea of SCL :) ). For example:
-Obsoletes: foobar < 1.0 +Obsoletes: %{?scl_prefix}foobar < 1.0
- Provides tag must always be prefixed with
%{?scl_prefix}
. For example:
-Provides: foo(bar) +Provides: %{?scl_prefix}foo(bar)
- All the other tag definitions should be unchanged, unless they contain
%{name}
macro, which may need to be substituted for%pkg_name
(for example in SourceN tag, where it may be a part of URL). - There are some additional tags you should add. The package must require
%{scl}-runtime
, unless it depends on another package that requires%{scl}-runtime
.- Here is a rule of thumb: If packaging SCL with a language interpreter, like Ruby or Python, typically all other packages in the SCL depend on the interpreter. Therefore it is sufficient when only the interpreter runtime package requires
%{scl}-runtime
. Generally, every SCL package that can be installed on its own (without other packages from this SCL) should require the%{scl}-runtime
package. - The line to add is:
- Here is a rule of thumb: If packaging SCL with a language interpreter, like Ruby or Python, typically all other packages in the SCL depend on the interpreter. Therefore it is sufficient when only the interpreter runtime package requires
%{?scl:Requires %{scl}-runtime}
Subpackages
The same rules as for normal tags apply for subpackages tags. The only thing needs to be changed:
If (and only if) a package define its name with -n
, the name must be prefixed with %{?scl_prefix}
like this:
-%package -n foo +%package -n %{?scl_prefix}foo
This applies not only to %package
macro, but also for %description
and %files
.
Inter-SCL Dependencies
Available from scl-utils version 20120613.
There are situations where packages from an SCL have to depend on packages from another SCL. While using simple [Build]Requires: rails323-rubygem-rails
is possible, it is not the wisest thing to do, since it hardcodes the information about how %{?scl_prefix}
gets expanded. If the expansion of this macro was changed (for example from rails323-
to scl-rails-323
, all the packages using such dependencies would need to be manually altered and rebuilt.
The scl-utils package therefore provides two ways of inter-SCL dependencies using macros. Note that both of them are in fact functions, so they need to be invoked without the curly braces to work properly:
%scl_require foo
macro is used to depend on the whole SCL. This is used if you want to use a functionality of an SCL, no matter what packages that SCL will install along the way. For example, therails323
SCL contains the whole Ruby on Rails stack in version 3.2.3. If packaging an SCL that requires the whole Rails stack, you can simply use%{?scl:Requires: %scl_require rails323}
.%scl_require_package foo pkg
provides the ability to depend on a specific package from another SCL. For example, if your package needs rubygem-minitest package fromrails323
SCL, you can simply use%{?scl:BuildRequires: %scl_require_package rails323 rubygem-minitest}
. Installing the whole Rails stack in this case would be useless (and since rubygem-minitest is only a build dependency, it's not drawn in by default).
You can specify versioned Requires like this: %{?scl:BuildRequires: %{scl_require_package rails323 rubygem-minitest} = 1.0.0}
Note that both of these macros abstract from the way that %{?scl_prefix}
expands. Therefore, if any changes were made, a simple rebuild with new versions of SCL utils would be sufficient.
When specifying inter-SCL dependencies, these macros must be used.
Inside RPM Scripts
It is not possible to describe the general process of rewriting %prep
, %build
, %install
, %check
and the %pre*
and %post*
scripts. There are however some general rules:
- Substitute all occurencies of
%name
for%pkg_name
. Most importantly, the%setup
macro will need the-n
argument for SCL builds (thanks to%pkg_name
, we can use the same for non-SCL builds). Due to this, the%setup
macro must always be used:
-%setup +%setup -n %{pkg_name}-%{version}
- If using a
%_root_*
macro to point to the original filesystem, you must conditionalize it, so that the package can be rebuilt for non-SCL use:
-mkdir -p %{_sysconfdir} +mkdir -p %{?scl:%_root_sysconfdir}%{?!scl:%_sysconfdir}
- If building SCL packages that depend on other SCL packages, you might need the
scl enable
functionality to link properly/run proper binaries, etc. It is not generally possible to say where this will be needed, but an example might be compiling against an SCL library or running an interpreted script with the interpreter in SCL:
+%{?scl:scl enable %scl - << \EOF} ruby foo.rb RUBYOPT="-Ilib" ruby bar.rb # more stuff +%{?scl:EOF}
- All hardcoded paths (such as
/usr/share
) must be replaced with proper macros (%{_datadir}
in this case).
Files
The files section is usually OK as it is. The only adjustments needed are for the names of subpackages (see Subpackages) and possible of some path macros with their %_root_*
alternatives (this is determined by what you do in the scripts, see Inside RPM Scripts.
Dealing With Automatic Provides/Requires and Filtering
RPM has some automatic Provides/Requires searching capabilities as well as filtering capabilities. For example all Python libraries have an automatically added Requires: python(abi) = (version)
(and in SCL, the proper way to have this is Requires: %{?scl_prefix}python(abi) = (version)
). The scripts that search for these dependencies must sometimes be rewritten for SCL, as the original RPM ones are not extensible enough (and in some cases, filtering is not usable). This is the example of rewriting Python provides and requires (the following lines can be placed into the macros.%{scl}-config
):
%__python_provides /usr/lib/rpm/pythondeps-scl.sh --provides %{_scl_root} %{scl_prefix} %__python_requires /usr/lib/rpm/pythondeps-scl.sh --requires %{_scl_root} %{scl_prefix}
The pythondeps-scl.sh
is a file created from pythondeps.sh
by adjusting some search paths.
If there are some Provides/Requires, that you need to alter (for example pkg_config
provides), there are two ways to do it.
- Either with the following lines in
macros.%{scl}-config
(this will then apply to all packages built in the SCL):
%_use_internal_dependency_generator 0 %__deploop() while read FILE; do /usr/lib/rpm/rpmdeps -%{1} ${FILE}; done | /bin/sort -u %__find_provides /bin/sh -c "%{?__filter_prov_cmd} %{__deploop P} %{?__filter_from_prov}" %__find_requires /bin/sh -c "%{?__filter_req_cmd} %{__deploop R} %{?__filter_from_req}" # Handle pkgconfig virtual [provides/requires]. %__filter_from_req | %{__sed} -e 's|pkgconfig|%{?scl_prefix}pkgconfig|g' %__filter_from_prov | %{__sed} -e 's|pkgconfig|%{?scl_prefix}pkgconfig|g'
- Or in every single specfile that you want to filter Provides/Requires in, place following lines after tag definitions:
%{?scl:%filter_from_provides s|pkgconfig|%{?scl_prefix}pkgconfig|g} %{?scl:%filter_from_requires s|pkgconfig|%{?scl_prefix}pkgconfig|g} %{?scl:%filter_setup}
Filters don't work in EL-6, because rpm is using different filter mechanism, as mentioned in RHBZ#1001674. Consult EPEL:Packaging_Autoprovides_and_Requires_Filtering if you need to do dependency filtering in EPEL.
Dealing With Macro Files
Sometimes the package ships with macro files, that go into /etc/rpm
(in SCL terms, they go into %{?scl:%{_root_sysconfdir}}%{!?scl:%{_sysconfdir}}
). This is fine, if two conditions are met:
- The macro files must be renamed by appending
.%{scl}
to their name, so that they don't conflict with system files. - The defined macros must be present in a form, that doesn't break any macros from non-SCL packages (with exception of macros from
%{scl}-build
subpackage, as mentioned below).
The second rule means, that macros in macro files must be unexpanded or be properly conditionalized. This is fine:
# the %gem_docdir macro depends on a macro that may be redefined by a collection and thus is ok %gem_docdir %{gem_dir}/docs # the %python2_sitelib macro evaluates depending on whether we build for collection or not and thus is ok %python2_sitelib %(%{?scl:scl enable %scl '}%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"%{?scl:'})
and this is not fine:
# the %gem_dir macro hardcodes collection path and thus would break non-SCL builds, which is not ok %gem_dir /opt/fedora/ruby193/root/usr/share/gems # the %python2_sitelib macro would run "scl enable" everytime, so if the collection would redefine %__python2, this would not work for non-SCL builds %python2_sitelib %(scl enable %scl '%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"')
Macros and Depending Collections
Now consider a situation where you need to create collection mypython
that depends on python26
collection that defines macro %{__python2}
in a way shown above. The macro will evaluate to /opt/<provider>/mycollection/root/usr/bin/python2
, but the python2
binary is not in your collection, but in python26
- this means nothing can be built in depending collection now. When you need to follow this use-case, here is what you need to do:
- In
macros.python.python26
(part of python26-python-devel), define
%__python26_python2 /opt/<provider>/python26/root/usr/bin/python2
- And in macros file in
python26-build
and any depending collection's%{scl}-build
use:
%scl_package_override() {%global __python2 %__python26_python2}
This will redefine the %{__python2}
macro only if such %{scl}-build
package is present (which usually means that you want to build for the collection and don't mind breaking the macro for local non-SCL builds).
Dealing With Shebangs
Shebangs have two important aspects: they are processed by the automatic dependency processor and they point to a certain (possibly non-SCL location).
As one of its functions, the automatic dependency processor goes through the shebangs. And adds dependencies according to the binaries they point to. From the SCL point of view, there two types of shebangs:
/usr/bin/env foo
- 'auto-dependency generation' point of view: The resulting package will depend on
/usr/bin/env
, which is not a problem. - 'pointing to non-SCL location' point of view: If the
$PATH
is redefined properly in the enable scriptlet, thefoo
binary is found in the SCL hierarchy, so not a problem either. - But you generally should avoid this. Packages that belong into collection should make sure that they invoke collection binaries, so they should hardcore full path to them in shebangs.
- 'auto-dependency generation' point of view: The resulting package will depend on
/usr/bin/foo
- 'auto-dependency generation' point of view: The resulting package will depend on
/usr/bin/foo
(part of non-SCL package), but we may have wanted to depend on%{?_scl_root}/usr/bin/foo
when building for SCL. - 'pointing to non-SCL location' point of view: The
$PATH
redefinition has no effect, the/usr/bin/foo
binary is still used (which is probably not what was intended). - If you have some shebangs like this in the specfile and the should point to SCL locations when built for SCLs, you can use similar command to adapt them:
- 'auto-dependency generation' point of view: The resulting package will depend on
find %{buildroot} -type f | \ xargs sed -i -e '1 s"^#!/usr/bin/foo"#!%{?_scl_root}/usr/bin/foo"'
Building Packages
The converted packages should be buildable both in SCL and non-SCL buildroots. This means, that after correct conversion of the specfile, build in non-SCL buildroot will produce standard system RPMs and build in buildroot containing %{scl}-build will produce SCL packages.
Before building in mock, you will want to create SRPMs. The process here is standard - if the SCL macroes are properly conditionalized, everything will work even without the SCL metapackage and scl-utils-build
installed locally. However, for the build of the metapackage itself, you will need scl-utils-build
, otherwise rpmbuild
will be unable to parse the macros inside it.
For testing the SCL builds in mock (you will get the same environment in BREW), you have to create a standard config file with few adjustments (let's say that you are building SCL named ruby193
):
- Use
'install @build scl-utils-build ruby193-build'
forconfig_opts['chroot_setup_cmd']
. - Add a local repo with your builds of the SCL packages to the mock config file:
[ruby193] name=ruby193 enabled=1 baseurl=file:///home/bkabrda/ruby193/repo cost=0 metadata_expire=0
Testing non-SCL builds is done in normal mock configurations (obviously :) ).