SoftwareCollections

How to work with Software Collections

Not approved for Fedora Packages
Please note that official Fedora packages must not be configured as Software Collection packages. Fedora does not permit relocatable packages, packages using hierarchies that conflict or violate the FHS, or packages storing files in /opt. This documentation is NOT part of the Fedora Packaging Guidelines, and is only here should you wish to generate unofficial Software Collections against Fedora in a third-party repository.

What are Software Collections?

Software Collections is a concept for allowing multiple versions of softwareto be installed at the same time without interfering in any negative way with standard versions provided by the system. In other words, a Software Collection consists of several components that are "bundled" together and which are expected to work smoothly together without conflicting or overwriting any system files. Instalation and updating of Software Collections is managed by RPM.

One can start to use and build Software Collections after installation of these utilities:

 # yum install scl-utils scl-utils-build

Converting spec file to a Software Collection

The Software Collection concept allows that the same spec file can be built either like a normal or Software Collection package. To allow this option, a special macro usage needs to be followed while writing or converting a package for Software Collection.

Software Collection macros simply relocate package to non-conflicting namespaced path by redefining essential build macros such as %_bindir, %_sysconfdir, etc. and adding Software Collection specific virtual provides.

With this concept the Software Collection specific actions in spec file need to be put into the %{?scl: <something> } constructs.

In a spec file one needs to use %scl_package at the beginning of spec in the following format:

 %{?scl:%scl_package <package_name>}

This macro:

• will set up environment for Software Collection package and eventually adds Provides spec file tag with a Software Collection virtual provide.
• defines %pkg_name which is to be used instead of any %name references within the spec file as %name now includes Software Collection name. So %name DOES NOT correspond to a package name any more.

Then Name preamble tag needs to deal with both Software Collection and non-Software Collection package name so e.g. ruby package will contain:

 Name: %{?scl_prefix}ruby

In the %prep section %setup needs to be taught about different package name for Software Collection:

 %setup -q %{?scl:-n %{pkg_name}-%{version}}

As a sidenote, in practise one needs to reference not only paths related to Software Collection root but also absolute, unrelocated paths. These paths are saved in %_root_<name> macros, e.g. %_root_bindir expands to /usr/bin and not to /usr/scl/<software_collection1>/<arch/>root/usr/bin.

List of all Software Collection macros to be used in spec follows.

macro	meaning	example value
%scl_name	name of the Software Collection	core
%pkg_name	original package name	perl
%_scl_prefix	Software Collection root (not package root)	/usr/scl
%_scl_scripts	location of Software Collection scriptlets	/usr/scl/example/x86_64/
%_scl_root	package install-root	/usr/scl/example/x86_64/root
%_root_prefix	Software Collection unrelocated %_prefix	/usr
%_root_exec_prefix	Software Collection unrelocated %_exec_prefix	/usr
%_root_bindir	Software Collection unrelocated %_bindir	/usr/bin
%_root_sbindir	Software Collection unrelocated %_sbindir	/usr/sbin
%_root_datadir	Software Collection unrelocated %_datadir	/usr/share
%_root_sysconfdir	Software Collection unrelocated %_sysconfdir	/etc
%_root_libexecdir	Software Collection unrelocated %_libexecdir	/usr/libexec
%_root_sharedstatedir	Software Collection unrelocated %_sharedstatedir	/usr/com
%_root_localstatedir	Software Collection unrelocated %_localstatedir	/usr/var
%_root_libdir	Software Collection unrelocated %_libdir	/usr/lib
%_root_includedir	Software Collection unrelocated %_includedir	/usr/include
%_root_infodir	Software Collection unrelocated %_infodir	/usr/share/info
%_root_mandir	Software Collection unrelocated %_mandir	/usr/share/man
%_root_initddir	Software Collection unrelocated %_mandir	/etc/rc.d/init.d

IMPORTANT The following example shows that due to path relocation one needs to strictly use path macros instead of absolute references. So build of Software Collection package will always fail if one defines path absolutely like e.g. /etc instead of %_sysconfdir.

An example of converting less package to a Software Collection is shown here:

 --- a/less.spec
 +++ b/less.spec
 @@ -1,10 +1,12 @@
 +%{?scl:%scl_package less}
 +
  Summary: A text file browser similar to more, but better
 -Name: less
 +Name: %{?scl_prefix}less
  Version: 443
  Release: 1%{?dist}
  License: GPLv3+
  Group: Applications/Text
 -Source: http://www.greenwoodsoftware.com/less/%{name}-%{version}.tar.gz
 +Source: http://www.greenwoodsoftware.com/less/less-%{version}.tar.gz
  Source1: lesspipe.sh
  Source2: less.sh
  Source3: less.csh
 @@ -19,6 +21,7 @@ Buildroot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
  BuildRequires: ncurses-devel
  BuildRequires: pcre-devel
  BuildRequires: autoconf automake libtool
 +%{?scl:Requires:%scl_runtime}
  
  %description
  The less utility is a text file browser that resembles more, but has
 @@ -31,7 +34,7 @@ You should install less because it is a basic utility for viewing text
  files, and you'll use it frequently.
  
  %prep
 -%setup -q
 +%setup -q %{?scl:-n %{pkg_name}-%{version}}
  %patch1 -p1 -b .Foption
  %patch4 -p1 -b .time
  %patch5 -p1 -b .fsync
 @@ -52,16 +55,16 @@ make CC="gcc $RPM_OPT_FLAGS -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOU
  rm -rf $RPM_BUILD_ROOT
  %makeinstall
  strip -R .comment $RPM_BUILD_ROOT/%{_bindir}/less
 -mkdir -p $RPM_BUILD_ROOT/etc/profile.d
 +mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/profile.d
  install -p -c -m 755 %{SOURCE1} $RPM_BUILD_ROOT/%{_bindir}
 -install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT/etc/profile.d
 -install -p -c -m 644 %{SOURCE3} $RPM_BUILD_ROOT/etc/profile.d
 -ls -la $RPM_BUILD_ROOT/etc/profile.d
 +install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_sysconfdir}/profile.d
 +install -p -c -m 644 %{SOURCE3} $RPM_BUILD_ROOT%{_sysconfdir}/profile.d
 +ls -la $RPM_BUILD_ROOT%{_sysconfdir}/profile.d
  
  %files
  %defattr(-,root,root,-)
  %doc LICENSE
 -/etc/profile.d/*
 +%{_sysconfdir}/profile.d/*
  %{_bindir}/*
  %{_mandir}/man1/*

How to specify which packages belong to a Software Collection

Every Software collection needs to ship a meta package which installs essential (minimal) part of the packages (e.g. interpreter without extension modules). It also owns basic filesystem structure of the Software collection and ships Software collection scriptlets. This package assures that all essential packages are installed and there exist a way how to enable Software collection. This meta package produces the following packages:

• main package %{scl_prefix}%scl, e.g. scl_core after expansion
  • contains dependencies to base packages in a Software Collection, no files
• -runtime subpackage, e.g. scl_core-runtime after expansion
  • owns base directory structure
  • ships scriptlets used to enable environment for the set of Software Collection packages
• -build subpackage, e.g. scl_core-build after expansion
  • contains essential RPM macros for build of Software collection packages

Example meta package for Software Collection named example follows:

 %{!?scl:%global scl example}
 %scl_package %scl
 
 Summary: Package that installs %scl
 Name: %scl_name
 Version: 1
 Release: 1%{?dist}
 BuildArch: noarch
 License: GPLv2+
 Requires: %{scl_prefix}less
 BuildRequires: scl-utils-build
 
 %description
 This is the main package for %scl Software Collection.
 
 %package runtime
 Summary: Package that handles %scl Software Collection.
 Requires: scl-utils
 
 %description runtime
 Package shipping essential scripts to work with %scl Software Collection.
 
 %package build
 Summary: Package shipping basic build configuration
 
 %description build
 Package shipping essential configuration macros to build %scl Software Collection.
 
 %install
 rm -rf %{buildroot}
 mkdir -p %{buildroot}%{_scl_scripts}/root
 cat >> %{buildroot}%{_scl_scripts}/enable << EOF
 export PATH=%{_bindir}:\$PATH
 EOF
 %scl_install
 
 %files
 
 %files runtime
 %scl_files
 
 %files build
 %{_root_sysconfdir}/rpm/macros.%{scl}-config
 
 %changelog
 * Thu Jun 07 2011 Jindrich Novy <jnovy@redhat.com> 1-1
 - initial packaging

scl and scl-build package requirement and what they offer

There are two packages which are essential for running and building Software Collections, they are:

• scl - provides scl utility which is used to enable Software Collection for executing commands in Software Collection-enabled environments
• scl-build - provides essential macros to build Software Collections

File locations

The root of Software Collections is relocated to a different directory which does not interfere with base system installation. This directory is optional and can be set via %global _scl_prefix /path/to/my/collection/root in the spec file before invocation of %scl_package. If _scl_prefix macro is not defined explicitly then it defaults to /usr/scl. The file system hierarchy of a Software Collection follows:

• /usr/scl (configurable via %_scl_prefix)
  • collection1/
    • arch
      • root/ - package root for Software collection packages
      • global scl scriptlets
  • collection2/
  • ...

How Software Collections requirements and provides look like

In order to specify a dependency on a package within a Software Collection, one should use this style of dependencies:

 Require: %{scl_prefix}<package1>, %{scl_prefix}<package2>

The %scl_prefix will take care of unique Software Collection package namespacing so packager doesn't need to specify the full package names directly. The <package1>, etc. is to be a package names when built as system packages. Since these are not virtual provides the requirements actually corresponds to real package names so there is no need to add special Provides: ....

scl tool

This utility exists in order to perform a clean execution of Software Collection program in a separate environment than system one. This utility is part of the scl-utils package and has the following synopsis:

 $ scl <action> [<collection1>, <collection2> ...] <command>

It executes user command after setting up a separate environment based on execution of enable scriptlets for every Software collection package in separate bash instance than it is called from. Currenlty only enable action is implemented.

Special cases

Typically Software Collections and their applications are expected to be very self contained and only interacting with the system from their end. But there are cases where Software Collections and/or their applications might need and rely on the underlying system and need to be more integrated with it while still being a separated Software Collection.

The following subsections provide some guidance on how to achieve this using the Software Collection concept while interfering with the host system as little as possible and avoiding conflicts with it.

The general principle here for all cases is the same: As a system version of any component in your Software Collection might exists care needs to be taken to not overwrite any files associated with the system version. To do so the files installed into the system instead of /usr/scl/... need to be renamed in such a way as they won't overwrite or conflict with any files in the system. This can typically easily be achieved by prefixing the names of said files (or directories) with the %{?scl_prefix} name available for Software Collection specfiles.

Initscripts

For Software Collections or associated applications that provide a service those should typically be manageable from the system directly using the default tools like service or chkconfig. In order to avoid conflicts with the system versions of the same services the following packaging should be done:

 %install
 install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_root_sysconfdir}/rc.d/%{?scl_prefix}servicename

This will avoid name conflicts with system versions of the same service. Administrators can then easily manage the Software Collection version of the service using the %{?scl_prefix}servicename name.

Libraries

If a Software Collection ships libraries to be used only within a Software Collection or in addition to system libraries in Software collection environment (which should be a default case). Then just addition to LD_LIBRARY_PATH environment variable in enable scriptlet should be enough. This is how it can be done in spec:

 export LD_LIBRARY_PATH=%{_libdir}:$LD_LIBRARY_PATH

to prefer Software Collection libraries over system ones when the Software Collection is enabled.

To make libraries in a Software Collection available to the system, out of the Software Collection environment, the directory /etc/ld.so.conf.d/ can be used.

This is only recommended for libraries that don't exists in the core system at all, as otherwise the Software Collection version of a library might get preference over the system version of a library which could have fatal results including crashes of system applications.

In order to do this simply create a file with the name

 %{?scl_prefix}libs.conf

add that to your specfile:

 SOURCE2: %{?scl_prefix}libs.conf

The file itself contains a list of directories where your Software Collection libraries are located, like this example:

/usr/scl/collection_name/root/path/to/your/libdirectory

Finally install this file into the system /etc/ld.so.conf.d/ directory in the %install section:

 %install
 install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_root_sysconfdir}/ld.so.conf.d/

Manpath

In order to allow the system man command to display man pages from Software Collections after it is enabled, you will need to add the paths in your Software Collection containing your man pages to the MANPATH environment variable. This can easily be achieved using a line added to enable scriptlet. It can be done like this is spec:

 export MANPATH=${MANPATH}:%{_mandir}

So Software Collection man pages will not be visible as long as Software Collection is not enabled. (Note that %{_mandir} is relocated to Software Collection path)

If a Software Collection provides a wrapper script visible to the system enabling the Software Collection (say, in /usr/bin) then man pages should be visible to system as well even out of the enabled Software Collection environment.

In order to allow the system man command to display man pages from Software Collections you will need to add the paths in your Software Collection containing your man pages to the MANPATH environment variable. This can easily be achieved using a custom /etc/profile.d/foo.sh script that will be preloaded by any shell being started.

As an example, create the following file:

 %{?scl_prefix}manpage.sh

add that to your specfile:

 SOURCE2: %{?scl_prefix}manpage.sh

The file itself is a short script that only modifies the MANPATH variable to refer to your man path directory:

 export MANPATH=${MANPATH}:/usr/scl/collection_name/arch/path/to/your/manpages

Finally install this file into the system /etc/profile.d/ directory in the %install section:

 %install
 install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_root_sysconfdir}/profile.d/

Cronjob

Your Software Collection might require regular tasks to be performed while the system is running. This is usually achieved either with a dedicated service itself, covered in the initscripts section before or via cronjobs. To do the later those need to be installed again in the system path for cron to be picked up properly while the system is running. Simply put your a dedicated crontab file for your Software Collection in the /etc/cron.d directory with a Software Collection name again.

As an example, create the following file:

 %{?scl_prefix}crontab

add that to your specfile:

 SOURCE2: %{?scl_prefix}crontab

The file itself is a typical crontab file. An sample line might look like this:

 0 1 * * Sun root /usr/scl/collection_name/arch/usr/bin/yourcronjob

Finally install this file into the system /etc/cron.d/ directory in the %install section:

 %install
 install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_root_sysconfdir}/cron.d/

Logrotate

Similar to the cronjobs your Software Collection or applications within it might create logfiles that you want to trim using the logrotate functionality from the system. To do so you need to install your custom logrotate file for your Software Collection in the system directory for logrotate jobs /etc/logrotate.d/.

As an example, create the following file:

 %{?scl_prefix}logrotate

add that to your specfile:

 SOURCE2: %{?scl_prefix}logrotate

The file itself is a custom logrotate file. An sample line might look like this:

 /usr/scl/collection_name/var/log/yourapp.log {
     missingok
     notifempty
     size 30k
     yearly
     create 0600 root root
 }

Finally install this file into the system /etc/logrotate.d/ directory in the %install section:

 %install
 install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_root_sysconfdir}/logrotate.d/

Lock files

Lockfiles should typically be able to reside inside the Software Collection directory itself as the applications and/or services should be isolated from the system versions. There might be cases though where you would want to prevent the version of your Software Collection to be running while the system version is running or vice versa.

To do so your Software Collection version of your service or app that requires a lock should instead of writing it's lock to /usr/scl/collection_name/var/lock/ to the system directory /var/lock. And this time you don't want that lockfile to be renamed but keep the name of the system version in order to achieve the wanted behaviour.

Should your Software Collection version of the service be able to run concurrently with the system version (e.g. because it is configured to listen to a different port or doesn't otherwise conflict over resources from the system itself) you can keep the lockfile in your Software Collection directory.

Config files

Config files should typically always be kept in the /usr/scl/collection_name/ directory structure to prevent any overlaps with system versions of the same applications. If that is for any reason not possible at all then the principle of a Software Collection config filename applies again to make sure to avoid conflicts or overwrite with system versions of those config files.

How to do that is very dependant on the application or project that requires config files, but almost all provide some way to configure during build or install time where the application or project related config files will be stored. Using that method is strongly recommended.

Kernel modules

Kernel modules are a very special case as they are typically tied to a specific kernel version. Unfortunately there is no easy way to trigger an automatic update or install of newer versions of kernel modules should a newer system kernel be installed, but as a guidance there are a few things to make working with kernel module packages easier:

* Name of your kernel module package needs to include the kernel version
* Precise Requires: kernel-V-R need to be in the kernel module package

Enforcing Single Software Collection version installs

Although the concept of Software Collections is designed for multiple versions of the same components to be installed at the same time there might be cases where this is not only not desirable but needs to be avoided at all costs. This can be achieved using the Conflicts: tag in rpm specfiles.

Here a short example:

Imagine you have multiple Software Collection versions for Foo:

• Foo25
• Foo26
• Foo27
• Foo30
• Foo31

Now all of them automatically provide "python" with a specific version number. If you would now add the following to e.g. your Python25 Software Collection version of python:

 Conflicts: Foo != V-R

where V and R are the version respectively release of your Foo25 Software Collection then this would result in a rejection of any other Foo version to be installed. Unfortunately that doesn't work as this would prevent the system version of python to be installed as well.

Therefore all of your Software Collection of the different versions of python need to have an additional Provides: to be able to distinguish the Software Collection versions from the system one. So each of your python Software Collections needs to have the following added to their specfiles:

 Provides: CollectionFoo-V-R
 Conflicts: CollectionFoo != V-R

That way only your Software Collection versions of python will now conflict with one another while the system python is unaffected.

How to build your Software Collection package locally

Difference between building a normal and Software collection package locally can be illustrated by this example:

Normal package: rpmbuild -bb package.spec

Software Collection package: rpmbuild -bb package.spec --define 'scl <name>'

The build for Software collection only differs by definition of scl macro which triggers usage of Software Collection in spec.

How use Software Collections for your application

If an application requires a Software Collection then the dependency can be defined in the application's spec file like:

 BuildRequires: scl-utils-build
 Requires: %scl_require collection1
 Requires: %scl_require collection2

How to install a Software Collection or an App based on Software Collections

A Software Collection is installed by installation of the main Software Collection meta package, e.g. yum install scl_core for a Software Collection named core. This will install all packages within the noted Software Collection. If an application is dependent on Software Collection which is present in a distribution then normal installation procedure is followed using RPM, YUM or PackageKit.

How to run Software Collection applications

The scl tool is used to run applications in Software Collection environments. One can execute Software Collection applications directly as noted bellow or packager can create a wrapper script which will simplify the execution of a Software Collection application.

Example usages:

1. Get perl version from the Software Collection core:

 $ scl enable core 'perl --version'

2. Run bash with collection1 and collection2 enabled environment:

 $ scl enable collection1 collection2 bash

3. Execute commands in cmd file in core Software Collection environment:

 $ cat cmd | scl enable core -

SElinux

Basically if you want to install a package in an alternate location you need to tell SELinux about it.

If you have the same basic structure, say /usr/scl/example/x86_64/root/usr == /usr. Then you could setup labels

 semanage fcontext -a -e /usr /usr/scl/example/x86_64/root/usr
 restorecon -R -v /usr/scl/example/x86_64/root/usr

Which would then label everything in /usr/scl/example/x86_64/root/usr as if it was under /usr.

These policy will go into post installation macro in spec-file.

References

http://jnovy.fedorapeople.org/scl-utils/scl.pp.pdf - Presentation on Software Collections - guide for the impatient.