From Fedora Project Wiki

(Fixing typos -- assuming that /usr/src should be /usr/scl here)
(Just refer to SCG.)
Line 1: Line 1:
= How to work with Software Collections =
= Software Collections =
 


{{admon/important|Not approved for Fedora Packages|Please note that official Fedora packages '''must not''' be configured as Software Collection packages. Fedora does not permit relocatable packages, packages using hierarchies that conflict or violate the FHS, or packages storing files in /opt. This documentation is '''NOT''' part of the Fedora Packaging Guidelines, and is only here should you wish to generate unofficial Software Collections against Fedora in a third-party repository.}}
{{admon/important|Not approved for Fedora Packages|Please note that official Fedora packages '''must not''' be configured as Software Collection packages. Fedora does not permit relocatable packages, packages using hierarchies that conflict or violate the FHS, or packages storing files in /opt. This documentation is '''NOT''' part of the Fedora Packaging Guidelines, and is only here should you wish to generate unofficial Software Collections against Fedora in a third-party repository.}}
Line 6: Line 5:
== What are Software Collections? ==
== What are Software Collections? ==


Software Collections is a concept for allowing multiple versions of softwareto be installed at the same time without interfering in any negative way with standard versions provided by the system. In other words, a Software Collection consists of several components that are "bundled" together and which are expected to work smoothly together without conflicting or overwriting any system files. Instalation and updating of Software Collections is managed by RPM.
The concept of Software Collections allows multiple versions of software to be installed at the same time without interfering in any negative way with the standard versions provided by the system. In other words, a Software Collection consists of several components that are "bundled" together and which are expected to work smoothly together without conflicting or overwriting any system files. Installation and updating of Software Collections is managed by RPM.
 
One can start to use and build Software Collections after installation of these utilities:
 
  # yum install scl-utils scl-utils-build
 
== Converting spec file to a Software Collection ==
 
The Software Collection concept allows that the same spec file can be built either like a normal or Software Collection package. To allow this option, a special macro usage needs to be followed while writing or converting a package for Software Collection.
 
Software Collection macros simply relocate package to non-conflicting namespaced path by redefining essential build macros such as '''%_bindir''', '''%_sysconfdir''', etc. and adding Software Collection specific virtual provides.
 
With this concept the Software Collection specific actions in spec file need to be put into the '''%{?scl: <something> }''' constructs.
 
In a spec file one needs to use '''%scl_package''' at the beginning of spec in the following format:
 
  %{?scl:%scl_package <package_name>}
 
 
This macro:
* will set up environment for Software Collection package and eventually adds '''Provides''' spec file tag with a Software Collection virtual provide.
* defines '''%pkg_name''' which is to be used instead of any '''%name''' references within the spec file as '''%name''' now includes Software Collection name. So '''%name''' ''DOES NOT'' correspond to a package name any more.
 
Then '''Name''' preamble tag needs to deal with both Software Collection and non-Software Collection package name so e.g. '''ruby''' package will contain:
 
  Name: %{?scl_prefix}ruby
 
In the '''%prep''' section '''%setup''' needs to be taught about different package name for Software Collection:
 
  %setup -q %{?scl:-n %{pkg_name}-%{version}}
 
 
As a sidenote, in practise one needs to reference not only paths related to Software Collection root but also absolute, unrelocated paths. These paths are saved in '''%_root_<name>''' macros, e.g. '''%_root_bindir''' expands to '''/usr/bin''' and not to '''/usr/scl/<software_collection1>/<arch/>root/usr/bin'''.
 
List of all Software Collection macros to be used in spec follows.
 
{|
! macro            !! meaning          !! example value 
|-
| %'''scl_name''' || name of the Software Collection || '''core'''
|-
| %'''pkg_name'''  || original package name || '''perl'''
|-
| %'''_scl_prefix''' || Software Collection root (not package root) || '''/usr/scl'''
|-
| %'''_scl_scripts''' || location of Software Collection scriptlets || '''/usr/scl/example/x86_64/'''
|-
| %'''_scl_root''' || package install-root || '''/usr/scl/example/x86_64/root'''
|-
| %'''_root_prefix''' || Software Collection unrelocated %_prefix || '''/usr'''
|-
| %'''_root_exec_prefix''' || Software Collection unrelocated %_exec_prefix || '''/usr'''
|-
| %'''_root_bindir''' || Software Collection unrelocated %_bindir || '''/usr/bin'''
|-
| %'''_root_sbindir''' || Software Collection unrelocated %_sbindir || '''/usr/sbin'''
|-
| %'''_root_datadir''' || Software Collection unrelocated %_datadir || '''/usr/share'''
|-
| %'''_root_sysconfdir''' || Software Collection unrelocated %_sysconfdir || '''/etc'''
|-
| %'''_root_libexecdir''' || Software Collection unrelocated %_libexecdir || '''/usr/libexec'''
|-
| %'''_root_sharedstatedir''' || Software Collection unrelocated %_sharedstatedir || '''/usr/com'''
|-
| %'''_root_localstatedir''' || Software Collection unrelocated %_localstatedir || '''/usr/var'''
|-
| %'''_root_libdir''' || Software Collection unrelocated %_libdir || '''/usr/lib'''
|-
| %'''_root_includedir''' || Software Collection unrelocated %_includedir || '''/usr/include'''
|-
| %'''_root_infodir''' || Software Collection unrelocated %_infodir || '''/usr/share/info'''
|-
| %'''_root_mandir''' || Software Collection unrelocated %_mandir || '''/usr/share/man'''
|-
| %'''_root_initddir''' || Software Collection unrelocated %_mandir || '''/etc/rc.d/init.d'''
|}
 
'''IMPORTANT'''
The following example shows that due to path relocation one needs to strictly use path macros instead of absolute references. So build of Software Collection package will always fail if one defines path absolutely like e.g. '''/etc''' instead of '''%_sysconfdir'''.
 
An example of converting '''less''' package to a Software Collection is shown here:
 
  --- a/less.spec
  +++ b/less.spec
  @@ -1,10 +1,12 @@
  +%{?scl:%scl_package less}
  +
  Summary: A text file browser similar to more, but better
  -Name: less
  +Name: %{?scl_prefix}less
  Version: 443
  Release: 1%{?dist}
  License: GPLv3+
  Group: Applications/Text
  -Source: http://www.greenwoodsoftware.com/less/%{name}-%{version}.tar.gz
  +Source: http://www.greenwoodsoftware.com/less/less-%{version}.tar.gz
  Source1: lesspipe.sh
  Source2: less.sh
  Source3: less.csh
  @@ -19,6 +21,7 @@ Buildroot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
  BuildRequires: ncurses-devel
  BuildRequires: pcre-devel
  BuildRequires: autoconf automake libtool
  +%{?scl:Requires:%scl_runtime}
 
  %description
  The less utility is a text file browser that resembles more, but has
  @@ -31,7 +34,7 @@ You should install less because it is a basic utility for viewing text
  files, and you'll use it frequently.
 
  %prep
  -%setup -q
  +%setup -q %{?scl:-n %{pkg_name}-%{version}}
  %patch1 -p1 -b .Foption
  %patch4 -p1 -b .time
  %patch5 -p1 -b .fsync
  @@ -52,16 +55,16 @@ make CC="gcc $RPM_OPT_FLAGS -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOU
  rm -rf $RPM_BUILD_ROOT
  %makeinstall
  strip -R .comment $RPM_BUILD_ROOT/%{_bindir}/less
  -mkdir -p $RPM_BUILD_ROOT/etc/profile.d
  +mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/profile.d
  install -p -c -m 755 %{SOURCE1} $RPM_BUILD_ROOT/%{_bindir}
  -install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT/etc/profile.d
  -install -p -c -m 644 %{SOURCE3} $RPM_BUILD_ROOT/etc/profile.d
  -ls -la $RPM_BUILD_ROOT/etc/profile.d
  +install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_sysconfdir}/profile.d
  +install -p -c -m 644 %{SOURCE3} $RPM_BUILD_ROOT%{_sysconfdir}/profile.d
  +ls -la $RPM_BUILD_ROOT%{_sysconfdir}/profile.d
 
  %files
  %defattr(-,root,root,-)
  %doc LICENSE
  -/etc/profile.d/*
  +%{_sysconfdir}/profile.d/*
  %{_bindir}/*
  %{_mandir}/man1/*
 
== How to specify which packages belong to a Software Collection ==
 
Every Software collection needs to ship a meta package which installs essential (minimal) part of the packages (e.g. interpreter without extension modules). It also owns basic filesystem structure of the Software collection and ships Software collection scriptlets. This package assures that all essential packages are installed and there exist a way how to enable Software collection. This meta package produces the following packages:
 
* '''main package''' '''%{scl_prefix}%scl''', e.g. '''scl_core''' after expansion
** contains dependencies to base packages in a Software Collection, no files
* '''-runtime subpackage''', e.g. '''scl_core-runtime''' after expansion
** owns base directory structure
** ships scriptlets used to enable environment for the set of Software Collection packages
* '''-build subpackage''', e.g. '''scl_core-build''' after expansion
** contains essential RPM macros for build of Software collection packages
 
Example meta package for Software Collection named '''example''' follows:
 
  %{!?scl:%global scl example}
  %scl_package %scl
 
  Summary: Package that installs %scl
  Name: %scl_name
  Version: 1
  Release: 1%{?dist}
  BuildArch: noarch
  License: GPLv2+
  Requires: %{scl_prefix}less
  BuildRequires: scl-utils-build
 
  %description
  This is the main package for %scl Software Collection.
 
  %package runtime
  Summary: Package that handles %scl Software Collection.
  Requires: scl-utils
 
  %description runtime
  Package shipping essential scripts to work with %scl Software Collection.
 
  %package build
  Summary: Package shipping basic build configuration
 
  %description build
  Package shipping essential configuration macros to build %scl Software Collection.
 
  %install
  rm -rf %{buildroot}
  mkdir -p %{buildroot}%{_scl_scripts}/root
  cat >> %{buildroot}%{_scl_scripts}/enable << EOF
  export PATH=%{_bindir}:\$PATH
  EOF
  %scl_install
 
  %files
 
  %files runtime
  %scl_files
 
  %files build
  %{_root_sysconfdir}/rpm/macros.%{scl}-config
 
  %changelog
  * Thu Jun 07 2011 Jindrich Novy <jnovy@redhat.com> 1-1
  - initial packaging
 
 
== scl and scl-build package requirement and what they offer ==
 
There are two packages which are essential for running and building Software Collections, they are:
 
* '''scl''' - provides '''scl''' utility which is used to enable Software Collection for executing commands in Software Collection-enabled environments
* '''scl-build''' - provides essential macros to build Software Collections
 
== File locations ==
 
The root of Software Collections is relocated to a different directory which does not interfere with base system installation. This directory is optional and can be set via '''%global _scl_prefix /path/to/my/collection/root''' in the spec file before invocation of '''%scl_package'''. If '''_scl_prefix''' macro is not defined explicitly then it defaults to '''/usr/scl'''. The file system hierarchy of a Software Collection follows:
 
* '''/usr/scl''' (configurable via '''%_scl_prefix''')
** '''collection1/'''
*** '''arch'''
**** '''root/''' - package root for Software collection packages
**** global scl scriptlets
** '''collection2/'''
** ...
 
== How Software Collections requirements and provides look like ==
 
In order to specify a dependency on a package within a Software Collection, one should use this style of dependencies:
 
  Require: %{scl_prefix}<package1>, %{scl_prefix}<package2>
 
 
The '''%scl_prefix''' will take care of unique Software Collection package namespacing so packager doesn't need to specify the full package names directly. The '''<package1>''', etc. is to be a package names when built as system packages. Since these are not virtual provides the requirements actually corresponds to real package names so there is no need to add special '''Provides: ...'''.
 
== scl tool ==
This utility exists in order to perform a clean execution of Software Collection program in a separate environment than system one. This utility is part of the '''scl-utils''' package and has the following synopsis:
 
  $ scl <action> [<collection1>, <collection2> ...] <command>
 
It executes user command after setting up a separate environment based on execution of '''enable''' scriptlets for every Software collection package in separate bash instance than it is called from. Currenlty only '''enable''' action is implemented.
 
== Special cases ==
 
Typically Software Collections and their applications are expected to be very self contained and only interacting with the system from their end. But there are cases where Software Collections and/or their applications might need and rely on the underlying system and need to be more integrated with it while still being a separated Software Collection.
 
The following subsections provide some guidance on how to achieve this using the Software Collection concept while interfering with the host system as little as possible and avoiding conflicts with it.
 
The general principle here for all cases is the same: As a system version of any component in your Software Collection might exists care needs to be taken to not overwrite any files associated with the system version. To do so the files installed into the system instead of '''/usr/scl/...''' need to be renamed in such a way as they won't overwrite or conflict with any files in the system. This can typically easily be achieved by prefixing the names of said files (or directories) with the '''%{?scl_prefix'''} name available for Software Collection specfiles.
 
=== Initscripts ===
 
For Software Collections or associated applications that provide a service those should typically be manageable from the system directly using the default tools like '''service''' or '''chkconfig'''. In order to avoid conflicts with the system versions of the same services the following packaging should be done:
 
  %install
  install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_root_sysconfdir}/rc.d/%{?scl_prefix}servicename
 
This will avoid name conflicts with system versions of the same service. Administrators can then easily manage the Software Collection version of the service using the %{?scl_prefix}servicename name.
 
=== Libraries ===
 
If a Software Collection ships libraries to be used only within a Software Collection or in addition to system libraries in Software collection environment (which should be a default case). Then just addition to '''LD_LIBRARY_PATH''' environment variable in '''enable''' scriptlet should be enough. This is how it can be done in spec:
 
  export LD_LIBRARY_PATH=%{_libdir}:$LD_LIBRARY_PATH
 
to prefer Software Collection libraries over system ones when the Software Collection is enabled.
 
To make libraries in a Software Collection available to the system, out of the Software Collection environment, the directory '''/etc/ld.so.conf.d/''' can be used.
 
'''This is only recommended for libraries that don't exists in the core system at all, as otherwise the Software Collection version of a library might get preference over the system version of a library which could have fatal results including crashes of system applications.'''
 
In order to do this simply create a file with the name
 
  %{?scl_prefix}libs.conf
 
add that to your specfile:
 
  SOURCE2: %{?scl_prefix}libs.conf
 
The file itself contains a list of directories where your Software Collection libraries are located, like this example:
 
/usr/scl/collection_name/root/path/to/your/libdirectory
 
Finally install this file into the system '''/etc/ld.so.conf.d/''' directory in the '''%install''' section:
 
  %install
  install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_root_sysconfdir}/ld.so.conf.d/
 
 
=== Manpath ===
 
In order to allow the system '''man''' command to display man pages from Software Collections after it is enabled, you will need to add the paths in your Software Collection containing your man pages to the '''MANPATH''' environment variable. This can easily be achieved using a line added to '''enable''' scriptlet. It can be done like this is spec:
 
  export MANPATH=${MANPATH}:%{_mandir}
 
So Software Collection man pages will not be visible as long as Software Collection is not enabled. (Note that '''%{_mandir'''} is relocated to Software Collection path)
 
If a Software Collection provides a wrapper script visible to the system enabling the Software Collection (say, in '''/usr/bin''') then man pages should be visible to system as well even out of the enabled Software Collection environment.
 
In order to allow the system '''man''' command to display man pages from Software Collections you will need to add the paths in your Software Collection containing your man pages to the '''MANPATH''' environment variable. This can easily be achieved using a custom '''/etc/profile.d/foo.sh''' script that will be preloaded by any shell being started.
 
As an example, create the following file:
 
  %{?scl_prefix}manpage.sh
 
add that to your specfile:
 
  SOURCE2: %{?scl_prefix}manpage.sh
 
The file itself is a short script that only modifies the '''MANPATH''' variable to refer to your man path directory:
 
  export MANPATH=${MANPATH}:/usr/scl/collection_name/arch/path/to/your/manpages
 
Finally install this file into the system '''/etc/profile.d/''' directory in the '''%install''' section:
 
  %install
  install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_root_sysconfdir}/profile.d/
 
=== Cronjob ===
 
Your Software Collection might require regular tasks to be performed while the system is running. This is usually achieved either with a dedicated service itself, covered in the initscripts section before or via cronjobs. To do the later those need to be installed again in the system path for cron to be picked up properly while the system is running. Simply put your a dedicated '''crontab''' file for your Software Collection in the '''/etc/cron.d''' directory with a Software Collection name again.
 
As an example, create the following file:
 
  %{?scl_prefix}crontab
 
add that to your specfile:
 
  SOURCE2: %{?scl_prefix}crontab
 
The file itself is a typical crontab file. An sample line might look like this:
 
  0 1 * * Sun root /usr/scl/collection_name/arch/usr/bin/yourcronjob
 
Finally install this file into the system '''/etc/cron.d/''' directory in the '''%install''' section:
 
  %install
  install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_root_sysconfdir}/cron.d/
 
=== Logrotate ===
 
Similar to the cronjobs your Software Collection or applications within it might create logfiles that you want to trim using the logrotate functionality from the system. To do so you need to install your custom '''logrotate''' file for your Software Collection in the system directory for logrotate jobs '''/etc/logrotate.d/'''.
 
As an example, create the following file:
 
  %{?scl_prefix}logrotate
 
add that to your specfile:
 
  SOURCE2: %{?scl_prefix}logrotate
 
The file itself is a custom logrotate file. An sample line might look like this:
 
  /usr/scl/collection_name/var/log/yourapp.log {
      missingok
      notifempty
      size 30k
      yearly
      create 0600 root root
  }
 
Finally install this file into the system '''/etc/logrotate.d/''' directory in the '''%install''' section:
 
  %install
  install -p -c -m 644 %{SOURCE2} $RPM_BUILD_ROOT%{_root_sysconfdir}/logrotate.d/
 
=== Lock files ===
 
Lockfiles should typically be able to reside inside the Software Collection directory itself as the applications and/or services should be isolated from the system versions. There might be cases though where you would want to prevent the version of your Software Collection to be running while the system version is running or vice versa.
 
To do so your Software Collection version of your service or app that requires a lock should instead of writing it's lock to '''/usr/scl/collection_name/var/lock/''' to the system directory '''/var/lock'''. And this time you don't want that lockfile to be renamed but keep the name of the system version in order to achieve the wanted behaviour.
 
Should your Software Collection version of the service be able to run concurrently with the system version (e.g. because it is configured to listen to a different port or doesn't otherwise conflict over resources from the system itself) you can keep the lockfile in your Software Collection directory.
 
=== Config files ===
 
Config files should typically always be kept in the '''/usr/scl/collection_name/''' directory structure to prevent any overlaps with system versions of the same applications. If that is for any reason not possible at all then the principle of a Software Collection config filename applies again to make sure to avoid conflicts or overwrite with system versions of those config files.
 
How to do that is very dependant on the application or project that requires config files, but almost all provide some way to configure during build or install time where the application or project related config files will be stored. Using that method is strongly recommended.
 
=== Kernel modules ===
 
Kernel modules are a very special case as they are typically tied to a specific kernel version. Unfortunately there is no easy way to trigger an automatic update or install of newer versions of kernel modules should a newer system kernel be installed, but as a guidance there are a few things to make working with kernel module packages easier:
 
* Name of your kernel module package needs to include the kernel version
* Precise Requires: kernel-V-R need to be in the kernel module package
 
=== Enforcing Single Software Collection version installs ===
 
Although the concept of Software Collections is designed for multiple versions of the same components to be installed at the same time there might be cases where this is not only not desirable but needs to be avoided at all costs. This can be achieved using the '''Conflicts:''' tag in rpm specfiles.
 
Here a short example:
 
Imagine you have multiple Software Collection versions for Foo:
 
* Foo25
* Foo26
* Foo27
* Foo30
* Foo31
 
Now all of them automatically provide "python" with a specific version number. If you would now add the following to e.g. your Python25 Software Collection version of python:
 
  Conflicts: Foo != V-R
 
where V and R are the version respectively release of your Foo25 Software Collection then this would result in a rejection of any other Foo version to be installed. Unfortunately that doesn't work as this would prevent the system version of python to be installed as well.
 
Therefore all of your Software Collection of the different versions of python need to have an additional '''Provides:''' to be able to distinguish the Software Collection versions from the system one. So each of your python Software Collections needs to have the following added to their specfiles:
 
  Provides: CollectionFoo-V-R
  Conflicts: CollectionFoo != V-R
 
 
That way only your Software Collection versions of python will now conflict with one another while the system python is unaffected.
 
== How to build your Software Collection package locally ==
 
Difference between building a normal and Software collection package locally can be illustrated by this example:
 
Normal package:
'''rpmbuild -bb package.spec'''
 
Software Collection package:
'''rpmbuild -bb package.spec --define 'scl <name>''''
 
The build for Software collection only differs by definition of '''scl''' macro which triggers usage of Software Collection in spec.
 
== How use Software Collections for your application ==
 
If an application requires a Software Collection then the dependency can be defined in the application's spec file like:
 
  BuildRequires: scl-utils-build
  Requires: %scl_require collection1
  Requires: %scl_require collection2
 
== How to install a Software Collection or an App based on Software Collections ==
 
A Software Collection is installed by installation of the main Software Collection meta package, e.g. '''yum install scl_core''' for a Software Collection named '''core'''. This will install all packages within the noted Software Collection. If an application is dependent on Software Collection which is present in a distribution then normal installation procedure is followed using RPM, YUM or PackageKit.
 
== How to run Software Collection applications ==
 
The '''scl''' tool is used to run applications in Software Collection environments. One can execute Software Collection applications directly as noted bellow or packager can create a wrapper script which will simplify the execution of a Software Collection application.
 
Example usages:
 
1. Get '''perl''' version from the Software Collection '''core''':
 
  $ scl enable core 'perl --version'
 
 
2. Run bash with '''collection1''' and '''collection2''' enabled environment:
 
  $ scl enable collection1 collection2 bash
 
3. Execute commands in '''cmd''' file in '''core''' Software Collection environment:
 
  $ cat cmd | scl enable core -
 
= SElinux =
Basically if you want to install a package in an alternate location you need to tell SELinux about it.
 
If you have the same basic structure, say '''/usr/scl/example/x86_64/root/usr''' == '''/usr'''. Then you could setup labels
 
  semanage fcontext -a -e /usr /usr/scl/example/x86_64/root/usr
  restorecon -R -v /usr/scl/example/x86_64/root/usr
 
Which would then label everything in '''/usr/scl/example/x86_64/root/usr''' as if it was under '''/usr'''.
 
These policy will go into post installation macro in spec-file.
 


== References ==
For detailed information on Software Collections, refer to the [http://docs.fedoraproject.org/en-US/Fedora_Contributor_Documentation/1/html/Software_Collections_Guide/index.html|Fedora Software Collections Guide].


http://jnovy.fedorapeople.org/scl-utils/scl.pp.pdf - Presentation on Software Collections - guide for the impatient.
[[Category:Packaging]]

Revision as of 14:35, 21 June 2012

Software Collections

Not approved for Fedora Packages
Please note that official Fedora packages must not be configured as Software Collection packages. Fedora does not permit relocatable packages, packages using hierarchies that conflict or violate the FHS, or packages storing files in /opt. This documentation is NOT part of the Fedora Packaging Guidelines, and is only here should you wish to generate unofficial Software Collections against Fedora in a third-party repository.

What are Software Collections?

The concept of Software Collections allows multiple versions of software to be installed at the same time without interfering in any negative way with the standard versions provided by the system. In other words, a Software Collection consists of several components that are "bundled" together and which are expected to work smoothly together without conflicting or overwriting any system files. Installation and updating of Software Collections is managed by RPM.

For detailed information on Software Collections, refer to the Software Collections Guide.

Retrieved from "https://fedoraproject.org/w/index.php?title=SoftwareCollections&oldid=293274"