From Fedora Project Wiki
(scl_prefix confusion)
 
(85 intermediate revisions by 3 users not shown)
Line 1: Line 1:
Software Collections (SCLs) give rpm packagers a means of packaging multiple versions of software for a single distribution. The version from the software collection does not interact with the system version. SCLs can provide backwards and forwards compat versions of a package compared to what's on the system.
Software Collections (SCLs) give rpm packagers a means of packaging multiple versions of software for a single distribution. The version from the software collection does not interact with the system version. SCLs can provide backwards and forwards compat versions of a package compared to what's on the system.
{{admon/question|Add text about SCL use| SCLs are used to create an alternate platform.}}
SCLs are used to create a different platform than that which is shipped by the system.  The platform may then be shipped on top different OS's and different OS releases




Line 6: Line 9:
{|
{|
! Term !! Definition
! Term !! Definition
|-
!colspan="2"|General
|-
|-
| SCL || A Software Collection
| SCL || A Software Collection
|-
|-
| SCL Metapackage (SRPM) || SRPM package that defines the SCL.
| SCL Definition || The SCL Definition documents the platform the SCL establishes including the backwards compatibility guarantees for end users.  This is also the form that needs to be submitted to the FPC to approve the SCL.
|-
!colspan="2"|Packages
|-
| SCL Metapackage (SRPM) || SRPM package that defines the SCL.  Each SCL has exactly one of these.
|-
|-
| SCL Metapackage (RPM) || RPM built from the SCL Metapackage SRPM. Defines the essential packages you get when you install the scl.
| SCL Metapackage (RPM) || RPM built from the SCL Metapackage SRPM. Defines the essential packages you get when you install the scl.
Line 17: Line 26:
| SCL Build Package || Built from the SCL Metapackage SRPM. Contains macro definitions needed to build packages for the collection.
| SCL Build Package || Built from the SCL Metapackage SRPM. Contains macro definitions needed to build packages for the collection.
|-
|-
| General SCL Package || Any binary RPM built for an SCL.
| General SCL Package || Any binary RPM built for use within an SCL.  The ones that establish the platform are mentioned in the SCL Definition.
|-
|-
| Mainstream Package || Package containing the software in Fedora that is not part of an SCL.
| Mainstream Package || Package containing the software in Fedora that is not part of an SCL.
Line 27: Line 36:


* The SCL itself needs approval.  At the moment we're rather strict about what can be an SCL so that we limit the number and types of SCLs.  At least in this phase of getting SCLs into Fedora we want to avoid a proliferation of SCLs with only minor differences.  This may be relaxed in the future.
* The SCL itself needs approval.  At the moment we're rather strict about what can be an SCL so that we limit the number and types of SCLs.  At least in this phase of getting SCLs into Fedora we want to avoid a proliferation of SCLs with only minor differences.  This may be relaxed in the future.
* Each SCL Package needs to be approved so that it meets the packaging guidelines.
* Each General SCL Package needs to be approved so that it meets the packaging guidelines.


SCLs are approved by the FPC for now but we may move it to some other body in the future.  This is pending discussion by FPC, FESCo, and possibly the Environments and Stacks Working Group.  General SCL Packages are approved by packagers just like any other rpm package.
SCLs are approved by the FPC.  General SCL Packages are approved by packagers just like any other rpm package.  The SCL Maintainers must own the General SCL Packages that are listed in the SCL Definition and thus both what packages are "officially" part of the SCL and the backwards compat guarantees for them.  Additional General SCL packages not listed in the SCL Definition can be submitted and owned by any packager.  They do not have to be specially approved by the SCL owners.  They '''can not''' carry any official backwards compat guarantees.


{{admon/note||There's currently not much will within either FESCo or FPC to look into the "other body" question so we'll keep this in FPC and look at another body if it becomes a problem in the future.}}
{{admon/note|Vendor|All General SCL packages '''MUST''' be part of an existing collection in Fedora/EPEL.  They cannot extend a collection from another vendor, such as RHSCL or RHDTS, the metapackage must also be part of Fedora/EPEL.  If you wish to build packages that add onto another vendor's SCL, you need to create a new SCL that has an interSCL dependency on the other vendor's SCL you want to extend.)}}


{{admon/note||Toshio envisions the SCL Criteria section being broken off into its own section if we create a new group that handles SCL approval}}
{{admon/question|Need to make other vendor packages available for dep solving| Depending on other vendor SCLs may well mean that we need those SCLs for building and installing on end-user systems.  Need to figure out how that is done.}}


=== SCL Criteria ===
=== SCL Criteria ===
Line 54: Line 63:
SCLs '''must not''' conflict with other packages on a filesystem level.  They '''must''' follow the Conflicts Guidelines for use of the rpm Conflicts tag.  They '''may''' Conflict with other packages in other ways but be sure to read the rest of this section to understand what you must do in those cases.
SCLs '''must not''' conflict with other packages on a filesystem level.  They '''must''' follow the Conflicts Guidelines for use of the rpm Conflicts tag.  They '''may''' Conflict with other packages in other ways but be sure to read the rest of this section to understand what you must do in those cases.


Conflicts between tcp and udp ports, plugins that provides a non-version-protected API (different versions of apache modules, for instance), and other non-filesystem conflicts may occur.  These '''should not''' be marked using the rpm Conflicts tag as the person installing the packages can usually configure the packages to coexist with a bit of manual effort.  A service in an SCL '''must not''' override a service started from a mainstream package.  This is so that an already installed mainstream package gets disabled by installing an SCL.  Unfortunately we do not make the same protection in the opposite direction at this time.
Conflicts between tcp and udp ports, plugins that provides a non-version-protected API (different versions of apache modules, for instance), and other non-filesystem conflicts may occur.  These '''should not''' be marked using the rpm Conflicts tag as the person installing the packages can usually configure the packages to coexist with a bit of manual effort.  A service in an SCL '''must not''' override a service started from a mainstream package.  This is so that an already installed mainstream package does not get disabled by installing an SCL.  At this time we do not make the same protection guarantee for SCLs being disabled by installing a mainstream package or a different SCL.


For standalone services like httpd, we implement this by not allowing those services to start by default.  If a service should start by default when it is installed, it must be packaged as a mainstream package.
For standalone services like httpd, we implement this by not allowing those services to start by default.  If a service should start by default when it is installed, it must be packaged as a mainstream package.


For add on modules (like mod_php), the service must either be configured to only load if the mainstream package is not installed or configured to not load at all without the user configuring it to do so.
For add on modules (like mod_php), the service must either be configured to only load if the mainstream package is not installed or configured to not load at all without the user configuring it to do so.
{{admon/question|Expand this|Are there other things that SCLs should be allowed to conflict with than ports? If so, please list them here. Debug packages are conflicting. It's the way how are debug packages automatically created.}}


{{admon/question|Change Conflicts Guidelines|FPC should change the Conflicts guidelines to mention how to deal with Conflicting ports there.  
{{admon/question|Change Conflicts Guidelines|FPC should change the Conflicts guidelines to mention how to deal with Conflicting ports there.  
Line 70: Line 77:


=== SCL Approval Process ===
=== SCL Approval Process ===
{{admon/question||Need to write the template for this}}


SCLs need to create a web page similar to that used for Fedora Changes.  The page needs to document what the purpose of the SCL is including the main package/stack that is being enabled by the SCL.  SCLs '''must not''' target more than one main package/stack.  (ie: rails3 and ruby1.9.3 would be separate SCLs).  Use inter-SCL dependencies for this.
SCLs are approved by the FPC in a process similar to that used for Fedora Changes.  To make a request, copy the [[User:Toshio/SCL_Request_Template| SCL Definition Template]] to a new page and fill it out.  Then file an FPC ticket to have the FPC approve the SCL.
 
The SCL Defintion page documents the purpose of the SCL including the main package/stack that is being enabled by the SCL.  SCLs '''must not''' target more than one main package/stack.  (ie: rails3 and ruby1.9.3 would be two separate SCLs).  Use inter-SCL dependencies for making one SCL depend on the platform provided by another.


==== Compatibility Guarantees ====
==== Compatibility Guarantees ====


Each SCL can make compatibility guarantees.  These set expectations for the user about what things they can expect to remain backwards compatible for the life of an SCL and what things they should use only if they are prepared to update their code if needed.  The Compatibility Guarantees should list a set of the packages or libraries being provided by the SCL as things that won't change in a backwards incompatible manner.  Compatibility Guarantees are per-SCL so an individual SCL may choose to list everything that they provide or or very little as having guaranteed backwards compat.  At a minimum, the SCL '''must''' list a backwards compat guarantee that encompasses the version information encoded in the SCL metapackage's name.  SCLs '''must''' also mention whether they are supporting the backwards compat guarantees that SCL's they depend on are making or if the SCL could change that.  (For instance, if scl-rails3 depends on scl-ruby1.9.3 it '''must''' mention whether the SCL will always use ruby1.9.3 or if it could be targeted against a different version of ruby in the future.)
Each SCL can make compatibility guarantees.  These set expectations for the user about what things they can expect to remain backwards compatible for the life of an SCL and what things they should use only if they are prepared to update their code if needed.  The Compatibility Guarantees should list a set of the packages or libraries being provided by the SCL as things that won't change in a backwards incompatible manner.  Compatibility Guarantees are per-SCL so an individual SCL may choose to list everything that they provide or or very little as having guaranteed backwards compat.  At a minimum, the SCL '''must''' list a backwards compat guarantee that encompasses the version information encoded in the SCL metapackage's name.  SCLs '''must''' also mention whether they are supporting the backwards compat guarantees that SCL's they depend on are making or if the SCL could change that.  (For instance, if fdr-rails3 depends on fdr-ruby1.9.3 it '''must''' mention whether the SCL will always use ruby1.9.3 or if it could be targeted against a different version of ruby in the future.)


The SCL may choose to say they are going to maintain backwards compat even if that means backporting code or merely that they'll follow upstream's lead.  For instance, an scl named scl-httpd2.4 may choose to write the following guarantee:
The SCL may choose to say they are going to maintain backwards compat even if that means backporting code or merely that they'll follow upstream's lead.  For instance, an scl named fdr-httpd2.4 may choose to write the following guarantee:


* httpd -- follow upstream's 2.4.x releases
* httpd -- follow upstream's 2.4.x releases
Line 90: Line 98:
SCL maintainers '''should''' make every attempt to balance their ability to keep up with evaluating and pushing changes while maintaining a stable platform for people developing on top of their SCL when writing what guarantees they are willing to make.
SCL maintainers '''should''' make every attempt to balance their ability to keep up with evaluating and pushing changes while maintaining a stable platform for people developing on top of their SCL when writing what guarantees they are willing to make.


{{admon/warning|Security trumps compatibility| If a security issue needs to be fixed but that breaks backwards compatibility, the security issue '''must''' be fixed despite violating the guarantee.  The SCL Request '''must''' be updated to tell the Fedora versions and the version of the relevant packages from the SCL which needed a backwards incompatible change to fix the issue).}}
{{admon/warning|Security trumps compatibility| If a security issue needs to be fixed but that breaks backwards compatibility, the security issue '''must''' be fixed despite violating the guarantee.  The SCL Definition '''must''' be updated to tell the Fedora versions and the version of the relevant packages from the SCL which needed a backwards incompatible change to fix the issue).}}


The Guarantees are written down in two places:  The SCL Request form has a section that needs to list the guarantees being made.  The SCL metapackage's %description also needs to list the guarantees so that a user can query the information from their system.
The Guarantees are written down in two places:  The SCL Definition form has a section that needs to list the guarantees being made.  The SCL metapackage's %description also needs to list the guarantees so that a user can query the information from their system.


{{admon/question|May change from %description|There needs to be something associated with the SCL metapackage that lets the user discover what is guaranteed and what is not.  Although %description is used here, we may be able to reimplement this as a virtual Provide or another tag at a later date.}}
{{admon/question|May change from %description|There needs to be something associated with the SCL metapackage that lets the user discover what is guaranteed and what is not.  Although %description is used here, we may be able to reimplement this as a virtual Provide or another tag at a later date.}}
{{admon/question|Notes for the SCL Request template|It needs to have a Compatibility Guarantees section.  The template should contain boilerplate about security updates sometimes breaking compatibility and that guarantees may be changed (see the next section)}}


===== Changing a Guarantee =====
===== Changing a Guarantee =====
Line 104: Line 110:
Changing the compatibility guarantee requires:
Changing the compatibility guarantee requires:


# Changing the SCL Request to list the new compatibility guarantee.  The document '''must''' continue to document what the old compatibility guarantee was and the Fedora version and SCL metapackage version at which the compatibility guarantee changed.
# Changing the SCL Definition to list the new compatibility guarantee.  The document '''must''' continue to document what the old compatibility guarantee was and the Fedora version and SCL metapackage version at which the compatibility guarantee changed.
# Updating the SCL metapackage with the new compatibility guarantee in its %description
# Updating the SCL metapackage with the new compatibility guarantee in its %description
# These changes '''must''' go through a re-review of the SCL.
# These changes '''must''' go through a re-review of the SCL.
Line 111: Line 117:
=== SCL Retirement ===
=== SCL Retirement ===


{{admon/question|Draft|These are a few things that were discussed on IRCI still need to improve the language.}}
==== Planned Retirement ====
SCL Maintainers need to commit to supporting an SCL explicitly every cycle.  This '''must''' be done by the Change Proposals Submission Deadline.  If an SCL does not have a commitment to be maintained, the SCL will be retired and not branched for the next releaseThis gives people who depend on an SCL from 9 months to a year (depending on when the Change Proposal Submission Deadline is in relation to the previous Fedora release's EOL date) to port their software away from the old SCL.


If an SCL has open security bugs that apply to the SCL version in rawhide and have been open for 6 months or more than the SCL will be retired.  For the purposes of this requirement, a bug against the SCL the bug is defined as a bug against one of the packages that is mentioned in the compatibility guarantee or one of the packages that those packages depend upon.
==== Security Retirement ====


{{admon/question|Shorten security window|geppetto suggested shortening the window for unfixed security issues to 2 months}}
If an SCL has open security bugs that apply to the SCL version in rawhide and have been open for 2 months or more then the SCL will be retired in rawhide.  For the purposes of this requirement, a bug against the SCL is defined as a bug against one of the general SCL packages that is mentioned in the compatibility guarantee or one of the packages that those packages depend upon.
{{admon/question|Clarify general scl packages not in the compat guarantee|My suggestion -- retire just that package}}


SCL Maintainers need to commit to supporting an SCL explicitly every cycle.  This '''must''' be done by the Change Proposals Submission Deadline.  If an SCL does not have a commitment to be maintained, the SCL will be retired and not branched for the next release.  This gives people who depend on an SCL from 9 months to a year (depending on when the Change Proposal Submission Deadline is in relation to the previous Fedora release's EOL date) to port their software away from the old SCL.
A similar but less far reaching policy also applies to general SCL packages not needed to satisfy the compatibility guarantee.  If one of those packages has a security bug opened against it for 2 months or more, that general SCL package and things that depend on it will be retired in rawhide.


To be clear, just like normal packages, retiring of scls and scl general packages in already released Fedora is not to be done except for package renames.  provenpackagers will need to step in to try to deal with the security issues if they are bad enough.  They should also institute the [[Policy_for_nonresponsive_package_maintainers| Nonresponsive maintainer policy]] on the packager so that the SCL/General SCL package is removed at the next Fedora release.


When an SCL is no longer going to be supported, the fact that it is no longer available needs to be in the release notes for the Fedora Release where it no longer appears.  We also need to notify people that the SCLs are being retired.  Typically, this will happen in an email to announce@lists.fedoraproject.org right after the Change Proposal Submission Deadline.
==== How to Retire ====


{{admon/question|Timing|Is the timing right? If we want to give people more time to port, we can choose another date (like Fn-1 Release or even Fn-1 branch).}}
When an SCL is going to be retired the fact that it is no longer available needs to be in the release notes for the Fedora Release where it no longer appears. We also need to notify people that the SCLs are being retired.  Typically, this will happen in an email to announce@lists.fedoraproject.org right after the Change Proposal Submission Deadline.


{{admon/question|EPEL|Should EPEL choose to allow SCLS, it will need to heavily modify this section as they don't have the natural 6 month boundaries to deprecate and retire things on.  Might make sense for EPEL to consider creating cycles based on RHEL point releases or just pick some annual dates for themselves.  Creating cycles could also be leveraged by EPEL to address the backwards-incompatibilities-in-the-main-repository problem that EPEL has traditionally suffered from.}}
{{admon/question|EPEL|Should EPEL choose to allow SCLS, it will need to heavily modify this section as they don't have the natural 6 month boundaries to deprecate and retire things on.  Might make sense for EPEL to consider creating cycles based on RHEL point releases or just pick some annual dates for themselves.  Creating cycles could also be leveraged by EPEL to address the backwards-incompatibilities-in-the-main-repository problem that EPEL has traditionally suffered from.}}
Line 129: Line 136:
== Naming the SCL ==
== Naming the SCL ==


The SCL and the SCL metapackage are known by the same name.  This name '''must''' make it unique from non-scl packages.  Thus it carries a prefix of <code>scl-</code>.  Following the prefix, the SCL must carry the name and version of the primary package being packaged.  For instance, if this to be ruby at version 1.9.3 then the SCL metapackage would be named <code>scl-ruby1.9.3</code>.  All SCL packages within the SCL must carry the SCL metapackage base name as a prefix.  So, for instance, the ruby interpreter package contained within the metapackage would be named <code>scl-ruby1.9.3-ruby</code>.
The SCL and the SCL metapackage are known by the same name.  This name '''must''' make it unique from non-scl packages and also from scl packages that could be released by other vendors.  Thus it is prefixed with a vendor string.  In Fedora, the prefix is <code>fdr-</code>.  Following the prefix, the SCL must carry the name and version of the primary package being packaged.  For instance, if this is to be ruby at version 1.9.3 then the SCL metapackage would be named <code>fdr-ruby1.9.3</code>.  All SCL packages within the SCL must carry the SCL metapackage base name as a prefix.  So, for instance, the ruby interpreter package contained within the metapackage would be named <code>fdr-ruby1.9.3-ruby</code>.
 
The SCL Name also contains the version.  In general, this embedded version number follows [[Packaging:NamingGuidelines#Multiple_packages_with_the_same_base_name| the same rules as other compat packages]].  One difference to note is that SCLs are required to keep the dots in their embedded version as omitting the dots is considered a legacy practice and the greater number of packages with version in their name that scls introduce mean that we'll end up with ambiguous and conflicting versions more frequently (for instance, without dots upstream v8-3.14 and v8-3.14 would both want to take the name fdr-v8_314).


{{admon/note|Choosing the version in the SCL name|The version in the SCL metapackage name should be decided upon based on some sort of backwards compatibility guarantee.  Some upstreams will use a $MAJOR.$MINOR convention for backwards compatibility and should use just the $MAJOR.$MINOR components of the version in the name.  Others, like the ruby example, make backwards compatibility guarantees at a different level and the version should reflect that upstream policy. }}
{{admon/note|Choosing the version in the SCL name|The version in the SCL metapackage name should be decided upon based on some sort of backwards compatibility guarantee.  Some upstreams will use a $MAJOR.$MINOR convention for backwards compatibility and should use just the $MAJOR.$MINOR components of the version in the name.  Others, like the ruby example, make backwards compatibility guarantees at a different level and the version should reflect that upstream policy. }}


{{admon/question|Prefix|Toshio is a little bit flexible about what prefix is usedIf we end up putting the SCLs into a vendor directory, he could see prefixing the scls with the same vendor string, for instanceUnknown how the rest of the FPC would feel about using different prefixes.}}
{{admon/question|Get vendor prefix with LANANA|Need to get Fedora registered with LANANA http://www.lanana.org/lsbreg/instructions.html  In process.  licquia is our contact }}
 
{{admon/question|Vendor and Non-Fedora-Provided SCls|This has ramifications both here and in the SCL Criteria section: One of SCLs benefits is that you could create a platform that people can target independent of underlying OS.  This includes them building their own packages for the SCL.  Unfortunately, this means that they need to be able to install that exact SCL for both OSes.  This is going to mean that someone targeting the RHSCL SCLs will need access to install the RHSCL SCLs in FedoraSimilarly, if we wanted to extend the RHSCL SCLs in Fedora/EPEL (currently, by using a new Fedora SCL and inter-SCL dependencies) then we'd need the RHSCL SCLs present so that deps would be fulfillableThere's lots of issues around solving that.}}


{{admon/question|Prefix|marcela doesn't understand the reason for scl prefix before the name. Is it the prefix there because of our build system? I believe I already proved koji can build without the prefix. http://miroslav.suchy.cz/blog/archives/2013/10/09/how_to_set_up_koji_for_scl/index.html Also are the dots in name of package mandatory? No-one is using dots in existing collections.}}
{{admon/question|This does not address intra-SCL dependencies.  If a package is being built for two SCLs, what naming do we have to adopt to avoid collisions? ie: we have a fdr-python3.2 SCL and a fdr-mysql5.1 SCL. Some people writing code for fdr-python3.2 want to use an fdr-python3.2-mysql binding that targets the system version of mysqlOther people writing code want to use a mysql binding that targets the SCL version of mysql.  What name does this second package need to have to avoid collisions?}}


=== Version and Release String ===
=== Version and Release String ===
Line 145: Line 156:
* One which has the same name as the SRPM metapackage.  This RPM contains Requires to pull in the packages essential to this SCL.
* One which has the same name as the SRPM metapackage.  This RPM contains Requires to pull in the packages essential to this SCL.
* An <code>$SCLNAME-runtime</code> subpackage.  This package needs to be installed if an end user intends to use the scl.
* An <code>$SCLNAME-runtime</code> subpackage.  This package needs to be installed if an end user intends to use the scl.
{{admon/question|Requirement to help the end user install the scl|since the -runtime subpackage contains the things needed for the end user to use the SCL, why doesn't the main package have a dep on the -runtime subpackage?  Why is there a separation between the main binary rpm and the -runtime subpackage?}}
* An <code>$SCLNAME-build</code> subpackage.  This package contains rpm macros needed for building packages into the SCL.  If it is installed, a spec file with SCL macros inside will be built as a General SCL Package for <code>$SCLNAME</code>.  If it is not installed, it will be built as a mainstream package.
{{admon/question|explanation of the above|slavek: Installing the main binary RPM should install the "whole collection" (including the -runtime package), whereas installing the -runtime package itself shouldn't. This is because you may not want to install the whole collection, but only specific packages - the packages depend on -runtime, not the main package. E.g. if you install "ruby193", it should install the whole collection, but if you install only ruby193-foo, it will only install "ruby193-foo" and "ruby193-runtime"}}
 
* An <code>$SCLNAME-build</code> subpackage.  This package contains rpm macros needed for building packages into the SCL.  This subpackage is needed for Koji builds, but may be left out for local builds).
{{admon/question||I'm not certain I understand the above.  Is it really only needed for building in koji?  Since it includes an rpm macros file, it seems more like it's needed for building the SCL whether locally or in koji.  maybe it's trying to say that it may (must?) be left out for building a non-SCL version?}}
{{admon/question||marcela: $SCLNAME-build is a build requirement. It's not a good idea to install it on workstation, because it might produce unexpected results after builds. It's recommended to install it into virtual machine or mock, where are builds done in clean build environment and scl non-scl builds can't be mixed. }}
{{admon/question|scl_prefix|Need to decide on scl_prefix - /opt with pieces in /var and /etc?  %{_libdir}/scls (with pieces in /var and /etc?)  }}
{{admon/question||marcela: List of things installed outside of /opt.
/etc/logrotate.d/*
 
/etc/rc.d/init.d/*
 
/etc/rpm/macros.*-config
 
/etc/scl/prefixes/*
 
/var/lib/mysql


/var/log/mariadb55-mysqld.log
{{admon/note||It is not recommended to install <code>$SCLNAME-build</code> packages onto a live system.  If you forget that it is installed (or this is a multiuser system and someone else builds a package) you may end up building General SCL Packages by mistake later.  Use mock or a throw away virtual machine if you want to build locally.}}


/usr/lib/rpm/*
{{admon/question|Update this section for scldevel| with pointer to the scldevel section for more information}}


}}
{{admon/question|scl_prefix|  Decided on: /opt/$vendor with pieces in /var/opt/$vendor and /etc/opt/$vendor}}




{{admon/question|scl_prefix vs _scl_prefix|Just noticed we have both %scl_prefix and %_scl_prefix with two different meanings.  Need to rename one of those.}}


An example SCL metapackage follows:
An example SCL metapackage follows:
<pre>
<pre>
%global scl ruby1.9.3
%global scl ruby1.9.3
%global scl_vendor fedoraproject/scls
# Note: this must be invoked *after* defining scl_vendor
%scl_package %scl
%scl_package %scl
%_scl_prefix /opt/fedoraproject/scls


%global install_scl 1
%global install_scl 1


Summary: Package that installs ruby1.9.3
%global macrosdir %(d=%{_rpmconfigdir}/macros.d; [ -d $d ] || d=%{_root_sysconfdir}/rpm; echo $d)
Name: %scl_name
 
Name: %{scl_name}
Version: 1
Version: 1
Release: 1%{?dist}
Release: 1%{?dist}
Summary: SCL that installs ruby1.9.3
License: GPLv2+
License: GPLv2+
Requires: %{scl_prefix}js
Url: # Put the URL to the completed SCL Defintion here
Requires: %{scl_prefix}rubygem-sqlite3
BuildRequires: scl-utils-build
BuildRequires: scl-utils-build
Requires: %{name}-runtime = %{version}-%{release}
# List the packages essential for this SCL.
Requires: %{scl_prefix}ruby


%description
%description
This is the main package for the ruby1.9.3 Software Collection.
This is the main package for the %{scl} Software Collection.


[Provide some useful info about this SCL.]
[Provide some useful info about this SCL.]
Line 198: Line 199:


%package runtime
%package runtime
Summary: Package that handles %scl Software Collection
Summary: Minimal runtime environment for %{scl} Software Collection
Requires: scl-utils
Requires: scl-utils


%description runtime
%description runtime
Package shipping essential scripts to work with the ruby1.9.3 Software Collection.
This package contains essential scripts to work with the %{scl} Software
Collection.


%package build
%package build
Summary: Package shipping basic build configuration
Summary: Configuration to build packages for the %{scl} Software Collection
Requires: scl-utils-build
Requires: scl-utils-build
Requires: %{name}-runtime = %{version}-%{release}
# Only include if you are creating an scldevel package.  See the scldevel section for details
#Requires: %{name}-scldevel


%description build
%description build
Package shipping essential configuration macros to build packages for the ruby1.9.3 Software Collection.
This package containts essential configuration macros to build packages
for the %{scl} Software Collection.
 
# See the scldevel section for information on this optional section
#%%package scldevel
#Summary: Package shipping development files for %scl
#
#%%description scldevel
#Package shipping development files, especially useful for development of
#packages depending on %{scl} Software Collection.
 


%prep
%prep
Line 224: Line 239:
export MANPATH=%{_mandir}:\${MANPATH}
export MANPATH=%{_mandir}:\${MANPATH}
EOF
EOF
%scl_install
%{scl_install}
# Note: this echo can go away once a fix for this is deployed: https://bugzilla.redhat.com/show_bug.cgi?id=1084095
echo '%%scl_vendor %{scl_vendor}' >> %{buildroot}%{_root_sysconfdir}/rpm/macros.%{scl}-config
 
if [ x"%{macrosdir}" != x"%{_root_sysconfdir}/rpm" ] ; then
    # scl-utils installs this to the sysadmin directory.  Use the directory reserved for packages
    mkdir -p %{buildroot}%{macrosdir}/
    mv %{buildroot}%{_root_sysconfdir}/rpm/macros.%{scl}-config %{buildroot}%{macrosdir}/macros.%{scl}-config
fi


%files
%files


%files runtime
%files runtime
%scl_files
%{scl_files}


%files build
%files build
%{_root_sysconfdir}/rpm/macros.%{scl}-config
%{macrosdir}/macros.%{scl}-config
 
# See the scldevel section for information on this optional section
#%%files scldevel
#%{macrosdir}/macros.%{scl_name}-scldevel
 


%changelog
%changelog
Line 238: Line 266:
- Initial package.
- Initial package.
</pre>
</pre>
{{admon/question|scldevel package?|What's the purpose of the scldevel package?  I looked through some sample SCLs and only found one for the python27 and python3 SCLs.  They just seem to contain macro files... Couldn't those go in the -build package instead?}}


{{admon/question|Needs more info|This section needs to explain everything that is non-standard in the template.  For instance, we need either explain all the scl macros used here or point to another section that explains them.  We need to list all the standard macros whose meaning is changed by making this an SCL package.  Probably need to have someone who hasn't gotten to know SCLs to review this and make sure there's nothing that we haven't left unexplained.}}
{{admon/question|Needs more info|This section needs to explain everything that is non-standard in the template.  For instance, we need either explain all the scl macros used here or point to another section that explains them.  We need to list all the standard macros whose meaning is changed by making this an SCL package.  Probably need to have someone who hasn't gotten to know SCLs to review this and make sure there's nothing that we haven't left unexplained.}}
{{admon/question|Constraints on defining macros|-build only place for overrides.  macros themselves defined in General SCL packages like fdr-python2.7-python-devel.}}


Things to note here:
Things to note here:


* The <code>-build</code> subpackage '''should''' include Requires: scl-utils-build.
* The <code>-build</code> subpackage '''must''' include Requires: scl-utils-build.
{{admon/question||Why is this a should and not a must?}}
{{admon/question||Why is this a should and not a must?}}
{{admon/question||marcela: it's needed in the main package, why it should be again in -build sub-package?}}
{{admon/question||marcela: it's needed in the main package, why it should be again in -build sub-package?}}
Line 257: Line 289:


* Among other things, the <code>%scl_install</code> macro creates a macro file <code>macros.%{scl}-config</code> (that ends up in the <code>%{scl}-build</code> package). This file is always located at <code>%{buildroot}%{_root_sysconfdir}/rpm/macros.%{scl}-config</code>. If you need to add some more macros specific for this SCL, add them into this file.
* Among other things, the <code>%scl_install</code> macro creates a macro file <code>macros.%{scl}-config</code> (that ends up in the <code>%{scl}-build</code> package). This file is always located at <code>%{buildroot}%{_root_sysconfdir}/rpm/macros.%{scl}-config</code>. If you need to add some more macros specific for this SCL, add them into this file.
{{admon/question||This needs to be expanded upon.  how do you pre-create the file?  How do you stop scl_install from creating it?}}
{{admon/question||This needs to be expanded upon.  how do you pre-create the file?  How do you stop scl_install from creating it? Marcela: It's generated automatically by scl-utils. It contains name of the collection.}}
* Unlike in specfiles of normal SCL packages (see below), the metapackage doesn't need to conditionalize SCL specific macros, as it can only be used as a part of SCL (for example, instead of <code>Requires: %{?scl_prefix}foo</code>, use just <code>Requires: %{scl_prefix}foo</code> - notice the missing questionmark, that makes the first macro conditional).
* Unlike in specfiles of normal SCL packages (see below), the metapackage doesn't need to conditionalize SCL specific macros, as it can only be used as a part of SCL (for example, instead of <code>Requires: %{?scl_prefix}foo</code>, use just <code>Requires: %{scl_prefix}foo</code> - notice the missing questionmark, that makes the first macro conditional).
* When user runs <code>yum install ruby193</code>, he expects that the whole SCL with all dependencies gets installed. Because of that, it should have <code>Requires</code> on all packages of that SCL, that are needed for the SCL to fulfil its purpose. E.g. <code>yum install ruby193</code> definitely has to install Ruby 1.9.3, but it needn't install collection packages that were used to build the interpreter (and that are not needed at Ruby runtime).
* When user runs <code>yum install ruby193</code>, he expects that the whole SCL with all dependencies gets installed. Because of that, it should have <code>Requires</code> on all packages of that SCL, that are needed for the SCL to fulfil its purpose. E.g. <code>yum install ruby193</code> definitely has to install Ruby 1.9.3, but it needn't install collection packages that were used to build the interpreter (and that are not needed at Ruby runtime).


=== SCL Prefix (proposal - not part of draft) ===
=== SCL scldevel subpackages ===
 
The scldevel subpackage is an optional strategy that creates generic macros for most of the macros that are used to build on each of these related SCLs.  This aids in building General SCL packages '''for other SCLs''' that are more easily ported to a similar SCL (for instance, an scl for the same language interpreter at a different version).  Both the creation of scldevel subpackages in the SCL metapackage and usage of the generic macros in General SCL Packages is optional.  Packagers simply use the specific macros defined in the individual packages if they choose not to use scldevel macros.
 
If a SCL maintainer chooses to implement scldevel in the metapackage, they will need to do this to create them:
 
# The packager knows that they will have a General SCL package that provides macros in a file (example: fdr-python2.7-python-devel provides python2_7python2_sitelib in %{_macrodir}/macros.python.%{scl_name})
# They create the scldevel package which has a generic macro that maps to this: (%global scl_python2_sitelib %fdr_python_2_7python_sitelib in %{_macrosdir}/macros.%{scl_name}-scldevel )
# They modify the build package so that it Requires: %{name}-scldevel and modify the macros there to define in terms of the scldevel macros: (%global python2_sitelib %scl_python2_sitelib in %{_macrosdir}/macros.%{scl_name}-build)
 
Package maintainers that decide to use the generic macros will:
# Be in a situation where they have a General SCL package which relies on macros provides by a different SCL than the one it belongs to.
# modify their spec files to use the generic macro name rather than the specific macro name.
# Add a Requires: other-scl-scldevel to their package.  {{admon/question|### BEEP>>>>|  this doesn't work because the macros are needed at srpm buildtime.  Have to create a new scl instead.}}
 
 
Macros in scldevel '''MUST NOT''' conflict with system package provided macros (for instance, standard rpm macros like %{_libdir} and macros provided by language interpreters like %{python3_sitelib}).  Macros in scldevel '''WILL CONFLICT''' by design with the macros provided by scldevel packages of similar SCLs (for instance, scl_python3_sitelib would be provided by both the fdr-python3.3 and the fdr-python3.4 packages).
 
There are some drawbacks to using scldevel packages.
* scldevel packages add an extra layer of complexity to SCLs.
* In order for the generic macros to help, the macros have to be coordinated between similar SCLs (fdr-python2.4 and fdr-python2.7 have to maintain the same set of generic macros with the same names).
* The generic macros may need to be added onto by other packages which are not essential to the SCL.  For instance, there might be a General SCL package that provides a %foo_pluginsdir macro.  The General SCL would need to have this macro added to the metapackage's scldevel subpackage in generic form so that people using the scldevel package are can make use of it.
 
=== SCL Prefix (approved but needs to integrate into the draft) ===


==== Proposal 1 ====
* Use /opt/$vendor/scls/$sclname/ for the main part of the scl package.  This loosely replaces /usr/ in a mainstream package
* Use /opt/$vendor/scls/$sclname/ for the main part of the scl package.  This loosely replace /usr/ in a mainstream package
** /var/opt/$vendor/scls/$sclname/ for the variable state.  This replaces /var/  (thus, there would be /var/opt/$vendor/scls/log, /var/opt/$vendor/lib, and so on)
** /var/opt/$vendor/scls/$sclname/ for the variable state.  This replaces /var/  (thus, there would be /var/opt/$vendor/scls/log, /var/opt/$vendor/lib, and so on)
** /etc/opt/$vendor/scls/$sclname/ for config files.  This replaces /etc/
** /etc/opt/$vendor/scls/$sclname/ for config files.  This replaces /etc/
Line 270: Line 324:
* Third party scls go in /var/opt/$vendor2/scls
* Third party scls go in /var/opt/$vendor2/scls


==== Proposal 2 ====
Need to specify that using /opt is a temporary violation of the FHS that we feel comfortable with because we expect the FHS to be updated: https://bugs.linuxfoundation.org/attachment.cgi?id=432
* Use %{_libdir}/scls/$sclname/
** /var/lib/scls/$sclname/
** /etc/scls/$sclname/
* SCL names '''must''' contain a vendor string.  So rht-ruby1.9.3 for a ruby package provided by Red hat.  fdr-ruby1.9.3 for one provided by the fedoraproject.
* Have spot register fdr with the Linux Assigned Names and Numbers Authority and use that as vendor in the package name http://www.lanana.org/lsbreg/providers/index.html
* Third party scls end up in %{_libdir}/scls/$sclname but we're willing to accept that since collisions are protected against by good vendors putting the vendor string in the package name.


==== Proposal 3 ====
== General SCL Package ==
* Use %{_prefix}/scls/$sclname/
{{admon/note|SCL spec files are separate from mainstream specs| In Fedora, the spec for mainstream packages and SCL packages are separate.  This makes it so that maintainers who care about SCLs can know about the semantics and special syntax of building SCLs while those who do not can ignore them.  This is similar to the strategy adopted for mingw packages.  SCL macros '''must not''' be present in mainstream packagesSCL Macros may, but are not required to, be conditionalized so that the packages may be built in an un-SCL form as well. The SCL package spec files are shared between SCLs since most of the scl syntax remains the same (although in different branches [if rel-eng can pull that off] so that per-SCL differences are possible)}}
** /var/lib/scls/$sclname/
** /etc/scls/$sclname/
* SCL names '''must''' contain a vendor stringSo rht-ruby1.9.3 for a ruby package provided by Red Hat.  fdr-ruby1.9.3 for one provided by the fedoraproject.
* Third party scls end up in %{_libdir}/scls/$sclname but we're willing to accept that since collisions are protected against by good vendors putting the vendor string in the package name.


== General SCL Package ==
This section sums up the steps needed to take in order to convert a normal specfile to SCL specfile step by step.
This section sums up the steps needed to take in order to convert a normal specfile to SCL specfile step by step. The resulting package '''should''' be buildable even without SCL - that means both without <code>%{scl}-build</code> in the buildroot and with it. Advantage of this approach is, that both non-SCL packages like <code>rubygem-foo</code> and SCL packages %{scl_prefix}rubygem_foo (depending on the SCL) can be built from the same SRPM, depending on the buildroot.


{{admon/tip|Automation|'''spec2scl''' was written to automate the task of converting specfiles to scl-enabled specfiles as much as possible. Install it by <code>yum install spec2scl</code> or use upstream version from https://bitbucket.org/bkabrda/spec2scl.}}
{{admon/tip|Automation|'''spec2scl''' was written to automate the task of converting specfiles to scl-enabled specfiles as much as possible. Install it with <code>yum install spec2scl</code> or use upstream version from https://bitbucket.org/bkabrda/spec2scl.}}


{{admon/caution|Hardcoded Paths|If the package you're converting for SCL contains hardcoded paths in upstream code, you will need to do some patching. Typical places to look for hardcoded paths are [[#Dealing_With_Shebangs|shebangs]] and configure scripts.}}
{{admon/caution|Hardcoded Paths|If the package you're converting for SCL contains hardcoded paths in upstream code, you will need to do some patching. Typical places to look for hardcoded paths are [[#Dealing_With_Shebangs|shebangs]] and configure scripts.}}
=== Bits and Bobs ===
{{admon/question|What is this?|Record some things about General SCL Packages that need to be worked on in this section}}
The scl_prefix has to be defined in order to get an SRPM that matches with the package actually being built.  Possibilities:
Add this to the top of the spec file:
<pre>
%global scl_prefix fdr-python2.4-
</pre>
I don't like this as it means each scl-spec file would have to modify this for use in other scls.
Modify %scl_package to set scl_prefix even if the scl_build package isn't installed.  Seems reasonable from a Fedora perspective as all the information is in the spec file already (scl_package is given the scl name as an argument so it seems okay.  But for people that want to share the spec file between scl builds and non-scl builds I'm not sure if this is going to work.
Modify fedpkg to define the scl_prefix when building in scl branches.  And document that invoking rpmbuild -bs manually requires using --define scl_prefix=fdr-python2.4-


=== Converting Tags and Macro Definitions ===
=== Converting Tags and Macro Definitions ===
Every SCL specfile '''must''' have <code>%scl_package</code> macro specified (like any other macro, it '''should''' be conditionalized). This macro does this:
Every SCL specfile '''must''' have <code>%scl_package</code> macro specified. This macro does this:
* Rewrites the standard path macros by prefixing them with <code>%{_scl_prefix}/%{scl}/root/</code>. E.g. <code>%{_datadir}</code> changes (depending on value of <code>%_scl_prefix</code>) from <code>/usr/share</code> to e.g. <code>/opt/fedora/myscl/root/usr/share</code>.
* Rewrites the standard path macros by prefixing them with <code>%{scl_basedir}/%{scl_vendor}/%{scl}/root/</code>. E.g. <code>%{_datadir}</code> changes from <code>/usr/share</code> to e.g. <code>/opt/fedoraproject/scls/myscl/root/usr/share</code>.
* Introduces some SCL specific macros (like <code>%pkg_name</code>, <code>%scl_prefix</code> or <code>%_root_*</code> set of macros, that contain values of the original RPM macros (e.g. <code>%_root_datadir</code> contains <code>/usr/share</code>).
* Introduces some SCL specific macros (like <code>%pkg_name</code>, <code>%scl_prefix</code> or <code>%_root_*</code> set of macros, that contain values of the original RPM macros (e.g. <code>%_root_datadir</code> contains <code>/usr/share</code>).
One of the important macros is <code>%pkg_name</code>, which represents the original package name - the <code>%name</code> macro stands for the name with SCL prefix during SCL build. Therefore it is a good practice to also define <code>%pkg_name</code> macro for non-SCL builds, to be able to use it consistently throughout the whole specfile.
One of the important macros is <code>%pkg_name</code>, which represents the original mainstream package name.  During a General SCL Package build the <code>%name</code> macro includes the name with SCL prefix so using <code>%pkg_name</code> instead becomes important. You may also define <code>%pkg_name</code> macro for non-SCL builds, to be able to use it consistently throughout the whole specfile.


So here is what the first two lines should look like:
So here is what the first two lines should look like:
<pre>
<pre>
%{?scl:%scl_package foo}
%{?scl:%scl_package ruby}
%{!?scl:%global pkg_name %{name}}
%{!?scl:%global pkg_name ruby}
</pre>
</pre>


Usual steps to adapt tag definitions for SCL builds are these:
Usual steps to adapt tag definitions for SCL builds are these:
* Name '''must''' be modified like this:
* Name '''must''' be modified like this:
<pre>
<pre>
-Name:          foo
-Name:          ruby
+Name:          %{?scl_prefix}foo
+Name:          %{?scl_prefix}ruby
</pre>
</pre>
* Requires and BuildRequires have to be considered carefully. These depend on what you are building/linking with and it is your decision as a packager. The only rule here is, that if building/linking with other SCL packages, their names must be also prefixed with conditionalized macro <code>%{?scl_prefix}</code> like this:
* Requires and BuildRequires have to be considered carefully. These depend on what you are building/linking with and it is your decision as a packager. The only rule here is, that if building/linking with other SCL packages, their names must be also prefixed with conditionalized macro <code>%{?scl_prefix}</code> like this:
Line 323: Line 382:
-BuildRequires:  rubygem-foo = 1.0.0
-BuildRequires:  rubygem-foo = 1.0.0
+BuildRequires:  %{?scl_prefix}rubygem-foo = 1.0.0
+BuildRequires:  %{?scl_prefix}rubygem-foo = 1.0.0
</pre>
</pre>affe
* Obsoletes, Conflicts and BuildConflicts '''must''' always be prefixed with <code>%{?scl_prefix}</code>. This is extremely important, as the SCLs are often used for deploying new packages on older systems (that may contain old packages, now obsoleted by the new ones), but they shouldn't Obsolete or Conflict with the non-SCL RPMs installed on the system (that's the idea of SCL :) ). For example:
* Obsoletes, Conflicts and BuildConflicts '''must''' either be deleted or prefixed with <code>%{?scl_prefix}</code>. This is extremely important as SCLs are supposed to create an isolated environment that doesn't affect the main system.  If the Obsoletes remained and the general SCL package was deployed to an older systems that may contains old packages mentioned in the <code>Obsoletes:</code> or <code>Conflicts:</code> then the general SCL package would cause those packages to be removed from the main system.  Examples of dealing with these:
<pre>
<pre>
-Obsoletes:      foobar < 1.0
-Obsoletes:      foobar < 1.0
# Add %{?scl_prefix}.  This would Obsolete the foobar package as built for the SCL.
+Obsoletes:      %{?scl_prefix}foobar < 1.0
+Obsoletes:      %{?scl_prefix}foobar < 1.0
# You can also simply remove the Obsoletes altogether when the mentioned package
# has never been built for the SCL
-Obsoletes:      baz < 1.0
</pre>
</pre>
* Provides tag '''must''' always be prefixed with <code>%{?scl_prefix}</code>. For example:
* Provides tag '''must''' always be prefixed with <code>%{?scl_prefix}</code>. For example:
Line 352: Line 415:
This applies not only to <code>%package</code> macro, but also for <code>%description</code> and <code>%files</code>.
This applies not only to <code>%package</code> macro, but also for <code>%description</code> and <code>%files</code>.


If the subpackages Require the main package, those lines will also need adjusting to use <code>%{pkg_name}</code>.


=== Inter-SCL Dependencies ===
=== Inter-SCL Dependencies ===
Line 395: Line 459:


=== Dealing With Automatic Provides/Requires and Filtering ===
=== Dealing With Automatic Provides/Requires and Filtering ===
As noted earlier, all SCL Provides and Requires on things provided by a General SCL Package  '''must''' be prefixed with <code>%{?scl_prefix}</code>.  This includes rpm's automatic provides and requires.  In general, the scripts that generate provides and requires do not understand how to handle SCLs.  Therefore they '''must''' be discarded and manually specified Provides and Requires used in their place.  To make things easier on other people packaging for SCLs, the discarded SCL Requires should be replaced with Requires on the required package names.  This may change if the scripts that extract the autodependencies are updated to understand how to extract dependencies from scls.  These guidelines will be updated if that happens.
{{admon/note|Valid as of scl-utils-20140127-1|For a period of time, scl-utils filtered out all of the General SCL Package's auto-provides.  As of scl-utils-20140127-1, this behaviour has been reverted so spec files need to filter this out themselves.}}
==== AutoProvides ====
Removing the Autoprovides is fairly straightforward:
<pre>
%global __provides_exclude_from %{_scl_root}
</pre>
That tells the autoprovides mechanism not to scan any files in the %_scl_root when looking for autoprovides.
==== AutoRequires ====
Filtering the autorequires is harder as you do want Requires on mainstream packages to be left intact.  At the moment, the best method is probably to build the package, check the autorequires, and then manually exclude every requirement from an SCL package in a list.  For instance, if you are getting libdb-4.3 and libpython2.4 from the SCL, you would need to filter both of those from the auto requires like so:
<pre>
%global __requires_exclude ^(libdb-4\\.3\\.so\\(\\)(\\(64bit\\))?|libpython2\\.4\\.so\\.1\\.0\\(\\)(\\(64bit\\)))?$
</pre>
Remember that when you filter out the autorequire you need to replace it with a manual Require on the package.  For instance,
<pre>
Requires: %{scl_prefix}db4
Requires: %{scl_prefix}python-libs
</pre>
==== TODO: ====
{{admon/question|Are the python autodeps updated?| The following section from bkarbrda's draft make it seem like the python autodep script has been updated to handle SCLs.  But no one else has mentioned any such script updates.  Need to look into this.  Nope.  These might be provided by the RHEL pythonX.Y rpms but they aren't present in Fedora.  They are in the python2.7-python-devel package.  It just checks if it was given parameters.  If so, it adds the parameter as a prefix  to python(abi)}}
RPM has some automatic Provides/Requires searching capabilities as well as filtering capabilities. For example all Python libraries have an automatically added <code>Requires: python(abi) = (version)</code> (and in SCL, the proper way to have this is <code>Requires: %{?scl_prefix}python(abi) = (version)</code>). The scripts that search for these dependencies must sometimes be rewritten for SCL, as the original RPM ones are not extensible enough (and in some cases, filtering is not usable). This is the example of rewriting Python provides and requires (the following lines can be placed into the <code>macros.%{scl}-config</code>):
RPM has some automatic Provides/Requires searching capabilities as well as filtering capabilities. For example all Python libraries have an automatically added <code>Requires: python(abi) = (version)</code> (and in SCL, the proper way to have this is <code>Requires: %{?scl_prefix}python(abi) = (version)</code>). The scripts that search for these dependencies must sometimes be rewritten for SCL, as the original RPM ones are not extensible enough (and in some cases, filtering is not usable). This is the example of rewriting Python provides and requires (the following lines can be placed into the <code>macros.%{scl}-config</code>):
<pre>
<pre>
%__python_provides /usr/lib/rpm/pythondeps-scl.sh --provides %{_scl_root} %{scl_prefix}
%global __python_provides %{_rpmconfigdir}/pythondeps-scl.sh --provides %{_scl_root} %{scl_prefix}
%__python_requires /usr/lib/rpm/pythondeps-scl.sh --requires %{_scl_root} %{scl_prefix}
%global __python_requires %{_rpmconfigdir}/pythondeps-scl.sh --requires %{_scl_root} %{scl_prefix}
</pre>
</pre>
The <code>pythondeps-scl.sh</code> is a file created from <code>pythondeps.sh</code> by adjusting some search paths.
The <code>pythondeps-scl.sh</code> is a file created from <code>pythondeps.sh</code> by adjusting some search paths.
Line 428: Line 524:
Sometimes the package ships with macro files, that go into <code>/etc/rpm</code> (in SCL terms, they go into <code>%{?scl:%{_root_sysconfdir}}%{!?scl:%{_sysconfdir}}</code>). This is fine, if two conditions are met:
Sometimes the package ships with macro files, that go into <code>/etc/rpm</code> (in SCL terms, they go into <code>%{?scl:%{_root_sysconfdir}}%{!?scl:%{_sysconfdir}}</code>). This is fine, if two conditions are met:
* The macro files '''must''' be renamed by appending <code>.%{scl}</code> to their name, so that they don't conflict with system files.
* The macro files '''must''' be renamed by appending <code>.%{scl}</code> to their name, so that they don't conflict with system files.
* The defined macros '''must''' be present in a form, that doesn't break any macros from non-SCL packages (with exception of macros from <code>%{scl}-build</code> subpackage, as mentioned below).
* All macros '''must''' be part of the SCL Build Package even if they are present in a different package in the mainstream repo.  This is because there is no way to keep macros from conflicting between SCLs so we need to place them in the package that is already known to be problematic if it is installed on a system that is building rpm packages.
The second rule means, that macros in macro files must be unexpanded or be properly conditionalized. This is fine:
* The defined macros '''should''' be present in a form, that doesn't break any macros from non-SCL packages (with the exception of macros from <code>%{scl}-build</code> subpackage, as mentioned below). Other SCL packaging systems try to make it so these macros can be present on a system that builds both mainstream rpms and packages for one-SCL.  If the macros are conditionalized in this way, then that is possible.
 
{{admon/question|Remove the third rule?|With the requirement that the macros must go into the SCL BUild package, the third rule has little positive effect for Fedora and makes writing macros much harder.  Consider dropping it as a requirement.  (I've already changed it from a Must to a should since it doesn't help Fedora packages.}}
 
The third rule means, that macros in macro files must be unexpanded or be properly conditionalized. This is fine:
<pre>
<pre>
# the %gem_docdir macro depends on a macro that may be redefined by a collection and thus is ok
# the %gem_docdir macro depends on a macro that may be redefined by a collection and thus is ok
Line 445: Line 545:
%python2_sitelib %(scl enable %scl '%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"')
%python2_sitelib %(scl enable %scl '%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"')
</pre>
</pre>
{{admon/question|Doesn't work with multiple scls?|Looking at this, I think that it breaks when there's multiple scls installed on the system.  For instance, if I have a python2.4 and a python2.6 SCL installed so that I can test applications written for deployment on both RHEL5 and RHEL6, I might get the macros for the scl that I was not intending.  Perhaps putting the macros into the %{scl}-build package is a better idea.}}


==== Macros and Depending Collections ====
==== Macros and Depending Collections ====

Latest revision as of 16:07, 26 June 2014

Software Collections (SCLs) give rpm packagers a means of packaging multiple versions of software for a single distribution. The version from the software collection does not interact with the system version. SCLs can provide backwards and forwards compat versions of a package compared to what's on the system.

Add text about SCL use
SCLs are used to create an alternate platform.

SCLs are used to create a different platform than that which is shipped by the system. The platform may then be shipped on top different OS's and different OS releases


Definitions

Term Definition
General
SCL A Software Collection
SCL Definition The SCL Definition documents the platform the SCL establishes including the backwards compatibility guarantees for end users. This is also the form that needs to be submitted to the FPC to approve the SCL.
Packages
SCL Metapackage (SRPM) SRPM package that defines the SCL. Each SCL has exactly one of these.
SCL Metapackage (RPM) RPM built from the SCL Metapackage SRPM. Defines the essential packages you get when you install the scl.
SCL Runtime Package Built from the SCL Metapackage SRPM. Contains the SCL filesystem and essential scripts (usually just enable) used to work with this SCL.
SCL Build Package Built from the SCL Metapackage SRPM. Contains macro definitions needed to build packages for the collection.
General SCL Package Any binary RPM built for use within an SCL. The ones that establish the platform are mentioned in the SCL Definition.
Mainstream Package Package containing the software in Fedora that is not part of an SCL.

SCL Approval

SCLs need to be approved on several levels:

  • The SCL itself needs approval. At the moment we're rather strict about what can be an SCL so that we limit the number and types of SCLs. At least in this phase of getting SCLs into Fedora we want to avoid a proliferation of SCLs with only minor differences. This may be relaxed in the future.
  • Each General SCL Package needs to be approved so that it meets the packaging guidelines.

SCLs are approved by the FPC. General SCL Packages are approved by packagers just like any other rpm package. The SCL Maintainers must own the General SCL Packages that are listed in the SCL Definition and thus both what packages are "officially" part of the SCL and the backwards compat guarantees for them. Additional General SCL packages not listed in the SCL Definition can be submitted and owned by any packager. They do not have to be specially approved by the SCL owners. They can not carry any official backwards compat guarantees.

Vendor
All General SCL packages MUST be part of an existing collection in Fedora/EPEL. They cannot extend a collection from another vendor, such as RHSCL or RHDTS, the metapackage must also be part of Fedora/EPEL. If you wish to build packages that add onto another vendor's SCL, you need to create a new SCL that has an interSCL dependency on the other vendor's SCL you want to extend.)
Need to make other vendor packages available for dep solving
Depending on other vendor SCLs may well mean that we need those SCLs for building and installing on end-user systems. Need to figure out how that is done.

SCL Criteria

There can be only one

For any given version of a software environment that satisfies the other requirements listed below there shall only be one SCL that provides the environment within Fedora. The version may overlap with the version provided by the mainstream Fedora package. Please remember to read the SCL Criteria: Version section to understand how version is being used here.

Size

The software ecosystem that an SCL targets must be of a moderate to large size. A language interpreter would satisfy this. A web framework would as well. In general, a single library would not. Use the standard backwards/forwards-compat packaging standards for those. (An exception to this would be alternate libc implementation, older boost versions and similar, very basic libraries).

In some cases, an application may also be deemed large enough to satisfy this requirement when the application itself is the major portion of an ecosystem. Servers like apache httpd which has an ecosystem of loadable modules and postgres which has programs linked against its client libraries are examples of this.

Version

SCLs are used to allow parallel versions of a software environment to be installed and used on the system. In Fedora we choose not to allow all versions to be parallel installed. Only when there is a compatibility break between versions do we have a new version packaged. As an example, we might allow an httpd2.4 scl and an httpd2.5 scl as modules written for httpd2.4 may need intrusive code changes to run with httpd2.5. We would not have an httpd2.5.0 scl and an httpd2.5.1 scl as httpd2.5.1 was intended as a bugfix release that does not require code changes (or even a recompile) in order for modules built for httpd2.5.0 to run on them.

Conflicts with non-SCLs and other SCLs

Portions of this may belong in the SCL Package section. Re-evaluate once we have more of the guideline revised

SCLs must not conflict with other packages on a filesystem level. They must follow the Conflicts Guidelines for use of the rpm Conflicts tag. They may Conflict with other packages in other ways but be sure to read the rest of this section to understand what you must do in those cases.

Conflicts between tcp and udp ports, plugins that provides a non-version-protected API (different versions of apache modules, for instance), and other non-filesystem conflicts may occur. These should not be marked using the rpm Conflicts tag as the person installing the packages can usually configure the packages to coexist with a bit of manual effort. A service in an SCL must not override a service started from a mainstream package. This is so that an already installed mainstream package does not get disabled by installing an SCL. At this time we do not make the same protection guarantee for SCLs being disabled by installing a mainstream package or a different SCL.

For standalone services like httpd, we implement this by not allowing those services to start by default. If a service should start by default when it is installed, it must be packaged as a mainstream package.

For add on modules (like mod_php), the service must either be configured to only load if the mainstream package is not installed or configured to not load at all without the user configuring it to do so.

Change Conflicts Guidelines
FPC should change the Conflicts guidelines to mention how to deal with Conflicting ports there. Conflicting ports were seen in case of mysql. The new mysql must be configured first, both can't run and be enabled by default.

Only for packages that aren't parallel installable

Some packages have accepted methods of being parallel installed. These packages must not be packaged as SCLs. Use the standard guidelines for parallel installing the packages instead. Taking a look at upstream, if and how other packages multiple versions of the software, existing Packaging Guidelines, and similar existing packages within the distribution can guide you in determining whether the package is parallel installable or not.

SCL Approval Process

SCLs are approved by the FPC in a process similar to that used for Fedora Changes. To make a request, copy the SCL Definition Template to a new page and fill it out. Then file an FPC ticket to have the FPC approve the SCL.

The SCL Defintion page documents the purpose of the SCL including the main package/stack that is being enabled by the SCL. SCLs must not target more than one main package/stack. (ie: rails3 and ruby1.9.3 would be two separate SCLs). Use inter-SCL dependencies for making one SCL depend on the platform provided by another.

Compatibility Guarantees

Each SCL can make compatibility guarantees. These set expectations for the user about what things they can expect to remain backwards compatible for the life of an SCL and what things they should use only if they are prepared to update their code if needed. The Compatibility Guarantees should list a set of the packages or libraries being provided by the SCL as things that won't change in a backwards incompatible manner. Compatibility Guarantees are per-SCL so an individual SCL may choose to list everything that they provide or or very little as having guaranteed backwards compat. At a minimum, the SCL must list a backwards compat guarantee that encompasses the version information encoded in the SCL metapackage's name. SCLs must also mention whether they are supporting the backwards compat guarantees that SCL's they depend on are making or if the SCL could change that. (For instance, if fdr-rails3 depends on fdr-ruby1.9.3 it must mention whether the SCL will always use ruby1.9.3 or if it could be targeted against a different version of ruby in the future.)

The SCL may choose to say they are going to maintain backwards compat even if that means backporting code or merely that they'll follow upstream's lead. For instance, an scl named fdr-httpd2.4 may choose to write the following guarantee:

  • httpd -- follow upstream's 2.4.x releases

which would allow the SCL to ship new 2.4.x releases even if upstream makes minor changes to the API. OTOH:

  • httpd -- maintain backwards compatibility with 2.4.0

means the SCL maintainer should evaluate whether a new httpd-2.4.x release has made any changes that would be backwards incompatible before rebasing to a new upstream minor release. The important thing here is that the Compatibility Guarantees set the user's expectations and then the SCL maintainer follows them.

SCL maintainers should make every attempt to balance their ability to keep up with evaluating and pushing changes while maintaining a stable platform for people developing on top of their SCL when writing what guarantees they are willing to make.

Security trumps compatibility
If a security issue needs to be fixed but that breaks backwards compatibility, the security issue must be fixed despite violating the guarantee. The SCL Definition must be updated to tell the Fedora versions and the version of the relevant packages from the SCL which needed a backwards incompatible change to fix the issue).

The Guarantees are written down in two places: The SCL Definition form has a section that needs to list the guarantees being made. The SCL metapackage's %description also needs to list the guarantees so that a user can query the information from their system.

May change from %description
There needs to be something associated with the SCL metapackage that lets the user discover what is guaranteed and what is not. Although %description is used here, we may be able to reimplement this as a virtual Provide or another tag at a later date.
Changing a Guarantee

Compatiblity guarantees can be changed but it is discouraged. This can happen, for instance, if the maintainer of an SCL wants to orphan it and someone else steps in to take over. The new maintainer may want to either make a larger or a smaller compatibility promise when they take over. Within a Fedora release, compatibility guarantees can be changed to cover more packages but you cannot make the guarantees cover less packages or change what versions the compatibility guarantee is made against. Those changes can only occur between Fedora releases.

Changing the compatibility guarantee requires:

  1. Changing the SCL Definition to list the new compatibility guarantee. The document must continue to document what the old compatibility guarantee was and the Fedora version and SCL metapackage version at which the compatibility guarantee changed.
  2. Updating the SCL metapackage with the new compatibility guarantee in its %description
  3. These changes must go through a re-review of the SCL.
  4. The changes need to be published in the release notes for the fedora release that they are introduced in.

SCL Retirement

Planned Retirement

SCL Maintainers need to commit to supporting an SCL explicitly every cycle. This must be done by the Change Proposals Submission Deadline. If an SCL does not have a commitment to be maintained, the SCL will be retired and not branched for the next release. This gives people who depend on an SCL from 9 months to a year (depending on when the Change Proposal Submission Deadline is in relation to the previous Fedora release's EOL date) to port their software away from the old SCL.

Security Retirement

If an SCL has open security bugs that apply to the SCL version in rawhide and have been open for 2 months or more then the SCL will be retired in rawhide. For the purposes of this requirement, a bug against the SCL is defined as a bug against one of the general SCL packages that is mentioned in the compatibility guarantee or one of the packages that those packages depend upon.

A similar but less far reaching policy also applies to general SCL packages not needed to satisfy the compatibility guarantee. If one of those packages has a security bug opened against it for 2 months or more, that general SCL package and things that depend on it will be retired in rawhide.

To be clear, just like normal packages, retiring of scls and scl general packages in already released Fedora is not to be done except for package renames. provenpackagers will need to step in to try to deal with the security issues if they are bad enough. They should also institute the Nonresponsive maintainer policy on the packager so that the SCL/General SCL package is removed at the next Fedora release.

How to Retire

When an SCL is going to be retired the fact that it is no longer available needs to be in the release notes for the Fedora Release where it no longer appears. We also need to notify people that the SCLs are being retired. Typically, this will happen in an email to announce@lists.fedoraproject.org right after the Change Proposal Submission Deadline.

EPEL
Should EPEL choose to allow SCLS, it will need to heavily modify this section as they don't have the natural 6 month boundaries to deprecate and retire things on. Might make sense for EPEL to consider creating cycles based on RHEL point releases or just pick some annual dates for themselves. Creating cycles could also be leveraged by EPEL to address the backwards-incompatibilities-in-the-main-repository problem that EPEL has traditionally suffered from.

Naming the SCL

The SCL and the SCL metapackage are known by the same name. This name must make it unique from non-scl packages and also from scl packages that could be released by other vendors. Thus it is prefixed with a vendor string. In Fedora, the prefix is fdr-. Following the prefix, the SCL must carry the name and version of the primary package being packaged. For instance, if this is to be ruby at version 1.9.3 then the SCL metapackage would be named fdr-ruby1.9.3. All SCL packages within the SCL must carry the SCL metapackage base name as a prefix. So, for instance, the ruby interpreter package contained within the metapackage would be named fdr-ruby1.9.3-ruby.

The SCL Name also contains the version. In general, this embedded version number follows the same rules as other compat packages. One difference to note is that SCLs are required to keep the dots in their embedded version as omitting the dots is considered a legacy practice and the greater number of packages with version in their name that scls introduce mean that we'll end up with ambiguous and conflicting versions more frequently (for instance, without dots upstream v8-3.14 and v8-3.14 would both want to take the name fdr-v8_314).

Choosing the version in the SCL name
The version in the SCL metapackage name should be decided upon based on some sort of backwards compatibility guarantee. Some upstreams will use a $MAJOR.$MINOR convention for backwards compatibility and should use just the $MAJOR.$MINOR components of the version in the name. Others, like the ruby example, make backwards compatibility guarantees at a different level and the version should reflect that upstream policy.
Get vendor prefix with LANANA
Need to get Fedora registered with LANANA http://www.lanana.org/lsbreg/instructions.html In process. licquia is our contact
Vendor and Non-Fedora-Provided SCls
This has ramifications both here and in the SCL Criteria section: One of SCLs benefits is that you could create a platform that people can target independent of underlying OS. This includes them building their own packages for the SCL. Unfortunately, this means that they need to be able to install that exact SCL for both OSes. This is going to mean that someone targeting the RHSCL SCLs will need access to install the RHSCL SCLs in Fedora. Similarly, if we wanted to extend the RHSCL SCLs in Fedora/EPEL (currently, by using a new Fedora SCL and inter-SCL dependencies) then we'd need the RHSCL SCLs present so that deps would be fulfillable. There's lots of issues around solving that.
This does not address intra-SCL dependencies. If a package is being built for two SCLs, what naming do we have to adopt to avoid collisions? ie: we have a fdr-python3.2 SCL and a fdr-mysql5.1 SCL. Some people writing code for fdr-python3.2 want to use an fdr-python3.2-mysql binding that targets the system version of mysql. Other people writing code want to use a mysql binding that targets the SCL version of mysql. What name does this second package need to have to avoid collisions?

Version and Release String

The SCL metapackage should use a simple integer starting with 1 as its version. SCL metapackages must use %{?dist} in their release string. There are no other special rules about how the version and release should be determined. As these strings are purely Fedora creations (there's no "upstream tarball" for the SCL), please use common sense when deciding how to increment these strings.

SCL Metapackage

Every SCL must have an SRPM-level metapackage. This metapackage must build three binary rpm packages:

  • One which has the same name as the SRPM metapackage. This RPM contains Requires to pull in the packages essential to this SCL.
  • An $SCLNAME-runtime subpackage. This package needs to be installed if an end user intends to use the scl.
  • An $SCLNAME-build subpackage. This package contains rpm macros needed for building packages into the SCL. If it is installed, a spec file with SCL macros inside will be built as a General SCL Package for $SCLNAME. If it is not installed, it will be built as a mainstream package.
It is not recommended to install $SCLNAME-build packages onto a live system. If you forget that it is installed (or this is a multiuser system and someone else builds a package) you may end up building General SCL Packages by mistake later. Use mock or a throw away virtual machine if you want to build locally.
Update this section for scldevel
with pointer to the scldevel section for more information
scl_prefix
Decided on: /opt/$vendor with pieces in /var/opt/$vendor and /etc/opt/$vendor


An example SCL metapackage follows:

%global scl ruby1.9.3
%global scl_vendor fedoraproject/scls

# Note: this must be invoked *after* defining scl_vendor
%scl_package %scl

%global install_scl 1

%global macrosdir %(d=%{_rpmconfigdir}/macros.d; [ -d $d ] || d=%{_root_sysconfdir}/rpm; echo $d)

Name: %{scl_name}
Version: 1
Release: 1%{?dist}
Summary: SCL that installs ruby1.9.3
License: GPLv2+
Url: # Put the URL to the completed SCL Defintion here
BuildRequires: scl-utils-build

Requires: %{name}-runtime = %{version}-%{release}
# List the packages essential for this SCL.
Requires: %{scl_prefix}ruby

%description
This is the main package for the %{scl} Software Collection.

[Provide some useful info about this SCL.]

[List the Compatibility Guarantees being made here]


%package runtime
Summary: Minimal runtime environment for %{scl} Software Collection
Requires: scl-utils

%description runtime
This package contains essential scripts to work with the %{scl} Software
Collection.

%package build
Summary: Configuration to build packages for the %{scl} Software Collection
Requires: scl-utils-build
Requires: %{name}-runtime = %{version}-%{release}
# Only include if you are creating an scldevel package.  See the scldevel section for details
#Requires: %{name}-scldevel

%description build
This package containts essential configuration macros to build packages
for the %{scl} Software Collection.

# See the scldevel section for information on this optional section
#%%package scldevel
#Summary: Package shipping development files for %scl
#
#%%description scldevel
#Package shipping development files, especially useful for development of
#packages depending on %{scl} Software Collection.


%prep
%setup -T -c

%build

%install
rm -rf %{buildroot}
mkdir -p %{buildroot}%{_scl_scripts}/root
cat >> %{buildroot}%{_scl_scripts}/enable << EOF
export PATH=%{_bindir}\${PATH:+:\${PATH}}
export LD_LIBRARY_PATH=%{_libdir}\${LD_LIBRARY_PATH:+:\${LD_LIBRARY_PATH}}
export MANPATH=%{_mandir}:\${MANPATH}
EOF
%{scl_install}
# Note: this echo can go away once a fix for this is deployed: https://bugzilla.redhat.com/show_bug.cgi?id=1084095
echo '%%scl_vendor %{scl_vendor}' >> %{buildroot}%{_root_sysconfdir}/rpm/macros.%{scl}-config

if [ x"%{macrosdir}" != x"%{_root_sysconfdir}/rpm" ] ; then
    # scl-utils installs this to the sysadmin directory.  Use the directory reserved for packages
    mkdir -p %{buildroot}%{macrosdir}/
    mv %{buildroot}%{_root_sysconfdir}/rpm/macros.%{scl}-config %{buildroot}%{macrosdir}/macros.%{scl}-config
fi

%files

%files runtime
%{scl_files}

%files build
%{macrosdir}/macros.%{scl}-config

# See the scldevel section for information on this optional section
#%%files scldevel
#%{macrosdir}/macros.%{scl_name}-scldevel


%changelog
* Fri Mar 30 2012 Bohuslav Kabrda <bkabrda@redhat.com> - 1-1
- Initial package.
scldevel package?
What's the purpose of the scldevel package? I looked through some sample SCLs and only found one for the python27 and python3 SCLs. They just seem to contain macro files... Couldn't those go in the -build package instead?
Needs more info
This section needs to explain everything that is non-standard in the template. For instance, we need either explain all the scl macros used here or point to another section that explains them. We need to list all the standard macros whose meaning is changed by making this an SCL package. Probably need to have someone who hasn't gotten to know SCLs to review this and make sure there's nothing that we haven't left unexplained.
Constraints on defining macros
-build only place for overrides. macros themselves defined in General SCL packages like fdr-python2.7-python-devel.

Things to note here:

  • The -build subpackage must include Requires: scl-utils-build.
Why is this a should and not a must?
marcela: it's needed in the main package, why it should be again in -build sub-package?
  • The enable script contains certain environment variable overrides so that programs and CLI interaction will use the files provided by the SCL. These are commonly overridden variables:
    • PATH=%{_bindir}\${PATH:+:\${PATH}} to run SCL binaries
    • LD_LIBRARY_PATH=%{_libdir}\${LD_LIBRARY_PATH:+:\${LD_LIBRARY_PATH}} to properly link against SCL shared objects
    • MANPATH=%{_mandir}:\${MANPATH} to be able to diplay manpages present in the SCL
    • PKG_CONFIG_PATH=%{_libdir}/pkgconfig\${PKG_CONFIG_PATH:+:\${PKG_CONFIG_PATH}} or PKG_CONFIG_PATH=%{_datadir}/pkgconfig\${PKG_CONFIG_PATH:+:\${PKG_CONFIG_PATH}} to enable using pkg-config files
    • XDG_DATA_PATH=%{_datadir}\${XDG_DATA_PATH:+:\${XDG_DATA_PATH}} to use systemtap tapsets from the SCL
  • SCL metapackages must not be noarch. The filesystem directories contained in the metapackage may contain arch-specific paths.
  • Among other things, the %scl_install macro creates a macro file macros.%{scl}-config (that ends up in the %{scl}-build package). This file is always located at %{buildroot}%{_root_sysconfdir}/rpm/macros.%{scl}-config. If you need to add some more macros specific for this SCL, add them into this file.
This needs to be expanded upon. how do you pre-create the file? How do you stop scl_install from creating it? Marcela: It's generated automatically by scl-utils. It contains name of the collection.
  • Unlike in specfiles of normal SCL packages (see below), the metapackage doesn't need to conditionalize SCL specific macros, as it can only be used as a part of SCL (for example, instead of Requires: %{?scl_prefix}foo, use just Requires: %{scl_prefix}foo - notice the missing questionmark, that makes the first macro conditional).
  • When user runs yum install ruby193, he expects that the whole SCL with all dependencies gets installed. Because of that, it should have Requires on all packages of that SCL, that are needed for the SCL to fulfil its purpose. E.g. yum install ruby193 definitely has to install Ruby 1.9.3, but it needn't install collection packages that were used to build the interpreter (and that are not needed at Ruby runtime).

SCL scldevel subpackages

The scldevel subpackage is an optional strategy that creates generic macros for most of the macros that are used to build on each of these related SCLs. This aids in building General SCL packages for other SCLs that are more easily ported to a similar SCL (for instance, an scl for the same language interpreter at a different version). Both the creation of scldevel subpackages in the SCL metapackage and usage of the generic macros in General SCL Packages is optional. Packagers simply use the specific macros defined in the individual packages if they choose not to use scldevel macros.

If a SCL maintainer chooses to implement scldevel in the metapackage, they will need to do this to create them:

  1. The packager knows that they will have a General SCL package that provides macros in a file (example: fdr-python2.7-python-devel provides python2_7python2_sitelib in %{_macrodir}/macros.python.%{scl_name})
  2. They create the scldevel package which has a generic macro that maps to this: (%global scl_python2_sitelib %fdr_python_2_7python_sitelib in %{_macrosdir}/macros.%{scl_name}-scldevel )
  3. They modify the build package so that it Requires: %{name}-scldevel and modify the macros there to define in terms of the scldevel macros: (%global python2_sitelib %scl_python2_sitelib in %{_macrosdir}/macros.%{scl_name}-build)

Package maintainers that decide to use the generic macros will:

  1. Be in a situation where they have a General SCL package which relies on macros provides by a different SCL than the one it belongs to.
  2. modify their spec files to use the generic macro name rather than the specific macro name.
  3. Add a Requires: other-scl-scldevel to their package.
### BEEP>>>>
this doesn't work because the macros are needed at srpm buildtime. Have to create a new scl instead.


Macros in scldevel MUST NOT conflict with system package provided macros (for instance, standard rpm macros like %{_libdir} and macros provided by language interpreters like %{python3_sitelib}). Macros in scldevel WILL CONFLICT by design with the macros provided by scldevel packages of similar SCLs (for instance, scl_python3_sitelib would be provided by both the fdr-python3.3 and the fdr-python3.4 packages).

There are some drawbacks to using scldevel packages.

  • scldevel packages add an extra layer of complexity to SCLs.
  • In order for the generic macros to help, the macros have to be coordinated between similar SCLs (fdr-python2.4 and fdr-python2.7 have to maintain the same set of generic macros with the same names).
  • The generic macros may need to be added onto by other packages which are not essential to the SCL. For instance, there might be a General SCL package that provides a %foo_pluginsdir macro. The General SCL would need to have this macro added to the metapackage's scldevel subpackage in generic form so that people using the scldevel package are can make use of it.

SCL Prefix (approved but needs to integrate into the draft)

  • Use /opt/$vendor/scls/$sclname/ for the main part of the scl package. This loosely replaces /usr/ in a mainstream package
    • /var/opt/$vendor/scls/$sclname/ for the variable state. This replaces /var/ (thus, there would be /var/opt/$vendor/scls/log, /var/opt/$vendor/lib, and so on)
    • /etc/opt/$vendor/scls/$sclname/ for config files. This replaces /etc/
  • Have spot register fedoraproject or fdr with the Linux Assigned Names and Numbers Authority and use that as $vendor http://www.lanana.org/lsbreg/providers/index.html
  • Third party scls go in /var/opt/$vendor2/scls

Need to specify that using /opt is a temporary violation of the FHS that we feel comfortable with because we expect the FHS to be updated: https://bugs.linuxfoundation.org/attachment.cgi?id=432

General SCL Package

SCL spec files are separate from mainstream specs
In Fedora, the spec for mainstream packages and SCL packages are separate. This makes it so that maintainers who care about SCLs can know about the semantics and special syntax of building SCLs while those who do not can ignore them. This is similar to the strategy adopted for mingw packages. SCL macros must not be present in mainstream packages. SCL Macros may, but are not required to, be conditionalized so that the packages may be built in an un-SCL form as well. The SCL package spec files are shared between SCLs since most of the scl syntax remains the same (although in different branches [if rel-eng can pull that off] so that per-SCL differences are possible)

This section sums up the steps needed to take in order to convert a normal specfile to SCL specfile step by step.

Automation
spec2scl was written to automate the task of converting specfiles to scl-enabled specfiles as much as possible. Install it with yum install spec2scl or use upstream version from https://bitbucket.org/bkabrda/spec2scl.
Hardcoded Paths
If the package you're converting for SCL contains hardcoded paths in upstream code, you will need to do some patching. Typical places to look for hardcoded paths are shebangs and configure scripts.

Bits and Bobs

What is this?
Record some things about General SCL Packages that need to be worked on in this section

The scl_prefix has to be defined in order to get an SRPM that matches with the package actually being built. Possibilities:

Add this to the top of the spec file:

%global scl_prefix fdr-python2.4-

I don't like this as it means each scl-spec file would have to modify this for use in other scls.

Modify %scl_package to set scl_prefix even if the scl_build package isn't installed. Seems reasonable from a Fedora perspective as all the information is in the spec file already (scl_package is given the scl name as an argument so it seems okay. But for people that want to share the spec file between scl builds and non-scl builds I'm not sure if this is going to work.

Modify fedpkg to define the scl_prefix when building in scl branches. And document that invoking rpmbuild -bs manually requires using --define scl_prefix=fdr-python2.4-

Converting Tags and Macro Definitions

Every SCL specfile must have %scl_package macro specified. This macro does this:

  • Rewrites the standard path macros by prefixing them with %{scl_basedir}/%{scl_vendor}/%{scl}/root/. E.g. %{_datadir} changes from /usr/share to e.g. /opt/fedoraproject/scls/myscl/root/usr/share.
  • Introduces some SCL specific macros (like %pkg_name, %scl_prefix or %_root_* set of macros, that contain values of the original RPM macros (e.g. %_root_datadir contains /usr/share).

One of the important macros is %pkg_name, which represents the original mainstream package name. During a General SCL Package build the %name macro includes the name with SCL prefix so using %pkg_name instead becomes important. You may also define %pkg_name macro for non-SCL builds, to be able to use it consistently throughout the whole specfile.

So here is what the first two lines should look like:

%{?scl:%scl_package ruby}
%{!?scl:%global pkg_name ruby}


Usual steps to adapt tag definitions for SCL builds are these:

  • Name must be modified like this:
-Name:           ruby
+Name:           %{?scl_prefix}ruby
  • Requires and BuildRequires have to be considered carefully. These depend on what you are building/linking with and it is your decision as a packager. The only rule here is, that if building/linking with other SCL packages, their names must be also prefixed with conditionalized macro %{?scl_prefix} like this:
-Requires:       bar
+Requires:       %{?scl_prefix}bar
  • For example, since ifconfig is a binary, we're going to use its system version, which means not using %scl_prefix with it. rubygem-foo, on the other hand, has to be in the path of the Ruby interpreter which will be part of this SCL, therefore it needs to be used with %scl_prefix. When depending on system packages, you should be very general in your requirements (avoid using versioned Requires on specific versions) and if you need a package that might be updated, you have to either add it to your SCL or be willing to rebuild the SCL when the system package updates.
 Requires        ifconfig
-Requires:       rubygem-foo = 1.0.0
+Requires:       %{?scl_prefix}rubygem-foo = 1.0.0
 BuildRequires:  ifconfig
-BuildRequires:  rubygem-foo = 1.0.0
+BuildRequires:  %{?scl_prefix}rubygem-foo = 1.0.0

affe

  • Obsoletes, Conflicts and BuildConflicts must either be deleted or prefixed with %{?scl_prefix}. This is extremely important as SCLs are supposed to create an isolated environment that doesn't affect the main system. If the Obsoletes remained and the general SCL package was deployed to an older systems that may contains old packages mentioned in the Obsoletes: or Conflicts: then the general SCL package would cause those packages to be removed from the main system. Examples of dealing with these:
-Obsoletes:      foobar < 1.0
 # Add %{?scl_prefix}.  This would Obsolete the foobar package as built for the SCL.
+Obsoletes:      %{?scl_prefix}foobar < 1.0
 # You can also simply remove the Obsoletes altogether when the mentioned package
 # has never been built for the SCL
-Obsoletes:      baz < 1.0
  • Provides tag must always be prefixed with %{?scl_prefix}. For example:
-Provides:       foo(bar)
+Provides:       %{?scl_prefix}foo(bar)
  • All the other tag definitions should be unchanged, unless they contain %{name} macro, which may need to be substituted for %pkg_name (for example in SourceN tag, where it may be a part of URL).
  • There are some additional tags you should add. The package must require %{scl}-runtime, unless it depends on another package that requires %{scl}-runtime.
    • Here is a rule of thumb: If packaging SCL with a language interpreter, like Ruby or Python, typically all other packages in the SCL depend on the interpreter. Therefore it is sufficient when only the interpreter runtime package requires %{scl}-runtime. Generally, every SCL package that can be installed on its own (without other packages from this SCL) should require the %{scl}-runtime package.
    • The line to add is:
%{?scl:Requires %{scl}-runtime}

Subpackages

The same rules as for normal tags apply for subpackages tags. The only thing needs to be changed:

If (and only if) a package define its name with -n, the name must be prefixed with %{?scl_prefix} like this:

-%package -n foo
+%package -n %{?scl_prefix}foo

This applies not only to %package macro, but also for %description and %files.

If the subpackages Require the main package, those lines will also need adjusting to use %{pkg_name}.

Inter-SCL Dependencies

Available from scl-utils version 20120613.

There are situations where packages from an SCL have to depend on packages from another SCL. While using simple [Build]Requires: rails323-rubygem-rails is possible, it is not the wisest thing to do, since it hardcodes the information about how %{?scl_prefix} gets expanded. If the expansion of this macro was changed (for example from rails323- to scl-rails-323, all the packages using such dependencies would need to be manually altered and rebuilt.

The scl-utils package therefore provides two ways of inter-SCL dependencies using macros. Note that both of them are in fact functions, so they need to be invoked without the curly braces to work properly:

  • %scl_require foo macro is used to depend on the whole SCL. This is used if you want to use a functionality of an SCL, no matter what packages that SCL will install along the way. For example, the rails323 SCL contains the whole Ruby on Rails stack in version 3.2.3. If packaging an SCL that requires the whole Rails stack, you can simply use %{?scl:Requires: %scl_require rails323}.
  • %scl_require_package foo pkg provides the ability to depend on a specific package from another SCL. For example, if your package needs rubygem-minitest package from rails323 SCL, you can simply use %{?scl:BuildRequires: %scl_require_package rails323 rubygem-minitest}. Installing the whole Rails stack in this case would be useless (and since rubygem-minitest is only a build dependency, it's not drawn in by default).

You can specify versioned Requires like this: %{?scl:BuildRequires: %{scl_require_package rails323 rubygem-minitest} = 1.0.0}

Note that both of these macros abstract from the way that %{?scl_prefix} expands. Therefore, if any changes were made, a simple rebuild with new versions of SCL utils would be sufficient.

When specifying inter-SCL dependencies, these macros must be used.

Inside RPM Scripts

It is not possible to describe the general process of rewriting %prep, %build, %install, %check and the %pre* and %post* scripts. There are however some general rules:

  • Substitute all occurencies of %name for %pkg_name. Most importantly, the %setup macro will need the -n argument for SCL builds (thanks to %pkg_name, we can use the same for non-SCL builds). Due to this, the %setup macro must always be used:
-%setup
+%setup -n %{pkg_name}-%{version}
  • If using a %_root_* macro to point to the original filesystem, you must conditionalize it, so that the package can be rebuilt for non-SCL use:
-mkdir -p %{_sysconfdir}
+mkdir -p %{?scl:%_root_sysconfdir}%{?!scl:%_sysconfdir}
  • If building SCL packages that depend on other SCL packages, you might need the scl enable functionality to link properly/run proper binaries, etc. It is not generally possible to say where this will be needed, but an example might be compiling against an SCL library or running an interpreted script with the interpreter in SCL:
+%{?scl:scl enable %scl - << \EOF}
 ruby foo.rb
 RUBYOPT="-Ilib" ruby bar.rb
 # more stuff
+%{?scl:EOF}
  • All hardcoded paths (such as /usr/share) must be replaced with proper macros (%{_datadir} in this case).

Files

The files section is usually OK as it is. The only adjustments needed are for the names of subpackages (see Subpackages) and possible of some path macros with their %_root_* alternatives (this is determined by what you do in the scripts, see Inside RPM Scripts.


Dealing With Automatic Provides/Requires and Filtering

As noted earlier, all SCL Provides and Requires on things provided by a General SCL Package must be prefixed with %{?scl_prefix}. This includes rpm's automatic provides and requires. In general, the scripts that generate provides and requires do not understand how to handle SCLs. Therefore they must be discarded and manually specified Provides and Requires used in their place. To make things easier on other people packaging for SCLs, the discarded SCL Requires should be replaced with Requires on the required package names. This may change if the scripts that extract the autodependencies are updated to understand how to extract dependencies from scls. These guidelines will be updated if that happens.

Valid as of scl-utils-20140127-1
For a period of time, scl-utils filtered out all of the General SCL Package's auto-provides. As of scl-utils-20140127-1, this behaviour has been reverted so spec files need to filter this out themselves.

AutoProvides

Removing the Autoprovides is fairly straightforward:

%global __provides_exclude_from %{_scl_root}

That tells the autoprovides mechanism not to scan any files in the %_scl_root when looking for autoprovides.

AutoRequires

Filtering the autorequires is harder as you do want Requires on mainstream packages to be left intact. At the moment, the best method is probably to build the package, check the autorequires, and then manually exclude every requirement from an SCL package in a list. For instance, if you are getting libdb-4.3 and libpython2.4 from the SCL, you would need to filter both of those from the auto requires like so:

%global __requires_exclude ^(libdb-4\\.3\\.so\\(\\)(\\(64bit\\))?|libpython2\\.4\\.so\\.1\\.0\\(\\)(\\(64bit\\)))?$

Remember that when you filter out the autorequire you need to replace it with a manual Require on the package. For instance,

Requires: %{scl_prefix}db4
Requires: %{scl_prefix}python-libs

TODO:

Are the python autodeps updated?
The following section from bkarbrda's draft make it seem like the python autodep script has been updated to handle SCLs. But no one else has mentioned any such script updates. Need to look into this. Nope. These might be provided by the RHEL pythonX.Y rpms but they aren't present in Fedora. They are in the python2.7-python-devel package. It just checks if it was given parameters. If so, it adds the parameter as a prefix to python(abi)

RPM has some automatic Provides/Requires searching capabilities as well as filtering capabilities. For example all Python libraries have an automatically added Requires: python(abi) = (version) (and in SCL, the proper way to have this is Requires: %{?scl_prefix}python(abi) = (version)). The scripts that search for these dependencies must sometimes be rewritten for SCL, as the original RPM ones are not extensible enough (and in some cases, filtering is not usable). This is the example of rewriting Python provides and requires (the following lines can be placed into the macros.%{scl}-config):

%global __python_provides %{_rpmconfigdir}/pythondeps-scl.sh --provides %{_scl_root} %{scl_prefix}
%global __python_requires %{_rpmconfigdir}/pythondeps-scl.sh --requires %{_scl_root} %{scl_prefix}

The pythondeps-scl.sh is a file created from pythondeps.sh by adjusting some search paths.

If there are some Provides/Requires, that you need to alter (for example pkg_config provides), there are two ways to do it.

  • Either with the following lines in macros.%{scl}-config (this will then apply to all packages built in the SCL):
%_use_internal_dependency_generator 0
%__deploop() while read FILE; do /usr/lib/rpm/rpmdeps -%{1} ${FILE}; done | /bin/sort -u
%__find_provides /bin/sh -c "%{?__filter_prov_cmd} %{__deploop P} %{?__filter_from_prov}"
%__find_requires /bin/sh -c "%{?__filter_req_cmd}  %{__deploop R} %{?__filter_from_req}"

# Handle pkgconfig virtual [provides/requires].
%__filter_from_req | %{__sed} -e 's|pkgconfig|%{?scl_prefix}pkgconfig|g'
%__filter_from_prov | %{__sed} -e 's|pkgconfig|%{?scl_prefix}pkgconfig|g'
  • Or in every single specfile that you want to filter Provides/Requires in, place following lines after tag definitions:
%{?scl:%filter_from_provides s|pkgconfig|%{?scl_prefix}pkgconfig|g}
%{?scl:%filter_from_requires s|pkgconfig|%{?scl_prefix}pkgconfig|g}
%{?scl:%filter_setup}
Filter your dependencies carefully
When using filters, you should consider carefully what automatic dependencies you actually want to change. For example, if the original package Requires: pkgconfig(foo) and Requires: pkgconfig(bar), and only foo is in the SCL, you don't want to filter the Requires for bar.

Filters don't work in EL-6, because rpm is using different filter mechanism, as mentioned in RHBZ#1001674. Consult EPEL:Packaging_Autoprovides_and_Requires_Filtering if you need to do dependency filtering in EPEL.

Dealing With Macro Files

Sometimes the package ships with macro files, that go into /etc/rpm (in SCL terms, they go into %{?scl:%{_root_sysconfdir}}%{!?scl:%{_sysconfdir}}). This is fine, if two conditions are met:

  • The macro files must be renamed by appending .%{scl} to their name, so that they don't conflict with system files.
  • All macros must be part of the SCL Build Package even if they are present in a different package in the mainstream repo. This is because there is no way to keep macros from conflicting between SCLs so we need to place them in the package that is already known to be problematic if it is installed on a system that is building rpm packages.
  • The defined macros should be present in a form, that doesn't break any macros from non-SCL packages (with the exception of macros from %{scl}-build subpackage, as mentioned below). Other SCL packaging systems try to make it so these macros can be present on a system that builds both mainstream rpms and packages for one-SCL. If the macros are conditionalized in this way, then that is possible.
Remove the third rule?
With the requirement that the macros must go into the SCL BUild package, the third rule has little positive effect for Fedora and makes writing macros much harder. Consider dropping it as a requirement. (I've already changed it from a Must to a should since it doesn't help Fedora packages.

The third rule means, that macros in macro files must be unexpanded or be properly conditionalized. This is fine:

# the %gem_docdir macro depends on a macro that may be redefined by a collection and thus is ok
%gem_docdir %{gem_dir}/docs

# the %python2_sitelib macro evaluates depending on whether we build for collection or not and thus is ok
%python2_sitelib %(%{?scl:scl enable %scl '}%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"%{?scl:'})

and this is not fine:

# the %gem_dir macro hardcodes collection path and thus would break non-SCL builds, which is not ok
%gem_dir /opt/fedora/ruby193/root/usr/share/gems

# the %python2_sitelib macro would run "scl enable" everytime, so if the collection would redefine %__python2, this would not work for non-SCL builds
%python2_sitelib %(scl enable %scl '%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"')
Doesn't work with multiple scls?
Looking at this, I think that it breaks when there's multiple scls installed on the system. For instance, if I have a python2.4 and a python2.6 SCL installed so that I can test applications written for deployment on both RHEL5 and RHEL6, I might get the macros for the scl that I was not intending. Perhaps putting the macros into the %{scl}-build package is a better idea.

Macros and Depending Collections

Now consider a situation where you need to create collection mypython that depends on python26 collection that defines macro %{__python2} in a way shown above. The macro will evaluate to /opt/<provider>/mycollection/root/usr/bin/python2, but the python2 binary is not in your collection, but in python26 - this means nothing can be built in depending collection now. When you need to follow this use-case, here is what you need to do:

  • In macros.python.python26 (part of python26-python-devel), define
%__python26_python2 /opt/<provider>/python26/root/usr/bin/python2
  • And in macros file in python26-build and any depending collection's %{scl}-build use:
%scl_package_override() {%global __python2 %__python26_python2}

This will redefine the %{__python2} macro only if such %{scl}-build package is present (which usually means that you want to build for the collection and don't mind breaking the macro for local non-SCL builds).

Macro %scl_package_override
This macro function is called at the end of %scl_package function. It is not defined in scl-utils-build, it is rather left to be defined by SCL packagers in %{scl}-build, so that they can override arbitrary macros only during collection package build. Due to the fact that %scl_package is always called from specfile, macros (re)defined by this function effectively override any macro defined in RPM macro files.

Dealing With Shebangs

Shebangs have two important aspects: they are processed by the automatic dependency processor and they point to a certain (possibly non-SCL location).

As one of its functions, the automatic dependency processor goes through the shebangs. And adds dependencies according to the binaries they point to. From the SCL point of view, there two types of shebangs:

  • /usr/bin/env foo
    • 'auto-dependency generation' point of view: The resulting package will depend on /usr/bin/env, which is not a problem.
    • 'pointing to non-SCL location' point of view: If the $PATH is redefined properly in the enable scriptlet, the foo binary is found in the SCL hierarchy, so not a problem either.
    • But you generally should avoid this. Packages that belong into collection should make sure that they invoke collection binaries, so they should hardcore full path to them in shebangs.
  • /usr/bin/foo
    • 'auto-dependency generation' point of view: The resulting package will depend on /usr/bin/foo (part of non-SCL package), but we may have wanted to depend on %{?_scl_root}/usr/bin/foo when building for SCL.
    • 'pointing to non-SCL location' point of view: The $PATH redefinition has no effect, the /usr/bin/foo binary is still used (which is probably not what was intended).
    • If you have some shebangs like this in the specfile and the should point to SCL locations when built for SCLs, you can use similar command to adapt them:
find %{buildroot} -type f | \
  xargs sed -i -e '1 s"^#!/usr/bin/foo"#!%{?_scl_root}/usr/bin/foo"'

Building Packages

The converted packages should be buildable both in SCL and non-SCL buildroots. This means, that after correct conversion of the specfile, build in non-SCL buildroot will produce standard system RPMs and build in buildroot containing %{scl}-build will produce SCL packages.

Before building in mock, you will want to create SRPMs. The process here is standard - if the SCL macroes are properly conditionalized, everything will work even without the SCL metapackage and scl-utils-build installed locally. However, for the build of the metapackage itself, you will need scl-utils-build, otherwise rpmbuild will be unable to parse the macros inside it.

For testing the SCL builds in mock (you will get the same environment in BREW), you have to create a standard config file with few adjustments (let's say that you are building SCL named ruby193):

  • Use 'install @build scl-utils-build ruby193-build' for config_opts['chroot_setup_cmd'].
  • Add a local repo with your builds of the SCL packages to the mock config file:
[ruby193]
name=ruby193
enabled=1
baseurl=file:///home/bkabrda/ruby193/repo
cost=0
metadata_expire=0

Testing non-SCL builds is done in normal mock configurations (obviously :) ).