User:Bkabrda/SCLGuidelinesDraft: Difference between revisions

Revision as of 08:34, 3 September 2013

Naming the SCL

Every SCL must have a name which does not match name of any other package. This is implied by the fact that the SCL metapackage's name is the same as name of the whole SCL. It is therefore not a good idea to name an SCL "ruby", when there is a "ruby" package - the name has to be altered. Packager should include something specific to the SCL (usually a specific version of included library) into it's name. Therefore "ruby193" is a good name for a collection that contains Ruby 1.9.3, etc.

Automation
spec2scl was written to automate the task of converting specfiles to scl-enabled specfiles as much as possible. Install it by yum install spec2scl or use upstream version from https://bitbucket.org/bkabrda/spec2scl.

SCL Metapackage

Every SCL must have a so called metapackage. This metapackage must have a runtime and build subpackages (the build subpackage is needed for Koji builds, but may be left out for local builds).

An example SCL metapackage follows:

%global scl ruby193
%scl_package %scl
%_scl_prefix /opt/myorganization

%global install_scl 1

Summary: Package that installs %scl
Name: %scl_name
Version: 1
Release: 1%{?dist}
License: GPLv2+
Requires: %{scl_prefix}js
Requires: %{scl_prefix}rubygem-sqlite3
Requires: %{scl_prefix}rubygem-rails
BuildRequires: scl-utils-build

%description
This is the main package for %scl Software Collection.

%package runtime
Summary: Package that handles %scl Software Collection.
Requires: scl-utils

%description runtime
Package shipping essential scripts to work with %scl Software Collection.

%package build
Summary: Package shipping basic build configuration
Requires: scl-utils-build

%description build
Package shipping essential configuration macros to build %scl Software Collection.

%install
rm -rf %{buildroot}
mkdir -p %{buildroot}%{_scl_scripts}/root
cat >> %{buildroot}%{_scl_scripts}/enable << EOF
export PATH=%{_bindir}\${PATH:+:\${PATH}}
export LD_LIBRARY_PATH=%{_libdir}\${LD_LIBRARY_PATH:+:\${LD_LIBRARY_PATH}}
export MANPATH=%{_mandir}:\${MANPATH}
EOF
%scl_install

%files

%files runtime
%scl_files

%files build
%{_root_sysconfdir}/rpm/macros.%{scl}-config

%changelog
* Fri Mar 30 2012 Bohuslav Kabrda <bkabrda@redhat.com> - 1-1
- Initial package.

Things to note here:

The -build subpackage should include Requires: scl-utils-build.
The enable scriptlet must contain certain path redefinitions, according to what packages SCL contains and what it is supposed to do. Most importantly, use these, if needed:
- PATH=%{_bindir}\${PATH:+:\${PATH}} to run SCL binaries
- LD_LIBRARY_PATH=%{_libdir}\${LD_LIBRARY_PATH:+:\${LD_LIBRARY_PATH}} to properly link against SCL shared objects
- MANPATH=%{_mandir}:\${MANPATH} to be able to diplay manpages present in the SCL
- PKG_CONFIG_PATH=%{_libdir}/pkgconfig\${PKG_CONFIG_PATH:+:\${PKG_CONFIG_PATH}} or PKG_CONFIG_PATH=%{_datadir}/pkgconfig\${PKG_CONFIG_PATH:+:\${PKG_CONFIG_PATH}} to enable using pkg-config files
- XDG_DATA_PATH=%{_datadir}\${XDG_DATA_PATH:+:\${XDG_DATA_PATH}} to use systemtap tapsets from the SCL
- If you use any of the above, that contains %{_libdir}, the metapackage must be arch specific, otherwise it may be noarch.
The macros.%{scl}-config in %{scl}-build package contains the %scl macro definition (name of the SCL). If you need to use some more macros for this SCL, add them into this file.
Unlike in specfiles of normal SCL packages (see below), the metapackage doesn't need to conditionalize SCL specific macros, as it can only be used as a part of SCL (for example, instead of Requires: %{?scl_prefix}foo, use just Requires: %{scl_prefix}foo - notice the missing questionmark, that makes the first macro conditional).
When user runs yum install ruby193, he expects that the whole SCL with all dependencies gets installed. Because of that, it should have Requires on all package of that SCL, that are essential for its runtime.

Converting Packages for SCL

This section sums up the steps needed to take in order to convert a normal specfile to SCL specfile step by step. The resulting package should be buildable even without SCL - that means both without %{scl}-build in the buildroot and with it. Advantage of this approach is, that both non-SCL packages like rubygem-foo and SCL packages %{scl_prefix}rubygem_foo (depending on the SCL) can be built from the same SRPM, depending on the buildroot.

Converting Tags and Macro Definitions

Every SCL specfile must have %scl_package macro specified (like any other macro, it must be conditionalized). This macro does this:

Rewrites the standard macros (like %_libdir, etc...).
Introduces some SCL specific macros (like %pkg_name, %scl_prefix or %_root_* set of macros, that contain values of the original RPM macros.

One of the important macros is %pkg_name, which represents the original package name - the %name macro stands for the name with SCL prefix during SCL build. Therefore it is a good practice to also define %pkg_name macro for non-SCL builds, to be able to use it consistently throughout the whole specfile.

So here is what the first two lines should look like:

%{?scl:%scl_package foo}
%{!?scl:%global pkg_name %{name}}

Usual steps to adapt tag definitions for SCL builds are these:

Name must be modified like this:

-Name:           foo
+Name:           %{?scl_prefix}foo

Requires and BuildRequires have to be considered carefully. These depend on what you are building/linking with and it is your decision as a packager. The only rule here is, that if building/linking with other SCL packages, their names must be also prefixed with conditionalized macro %{?scl_prefix} like this:

-Requires:       bar
+Requires:       %{?scl_prefix}bar

For example you can build and link the package with one SCL and one non-SCL package. The dependencies then look like this:

 Requires        spam
-Requires:       bar
+Requires:       %{?scl_prefix}bar
 BuildRequires:  spam-devel
-BuildRequires:  bar-devel
+BuildRequires:  %{?scl_prefix}bar-devel

Obsoletes, Conflicts and BuildConflicts must always be prefixed with %{?scl_prefix}. This is extremely important, as the SCLs are often used for deploying new packages on older systems (that may contain old packages, now obsoleted by the new ones), but they shouldn't Obsolete or Conflict with the non-SCL RPMs installed on the system (that's the idea of SCL :) ). For example:

-Obsoletes:      foobar < 1.0
+Obsoletes:      %{?scl_prefix}foobar < 1.0

Provides tag must always be prefixed with %{?scl_prefix}. For example:

-Provides:       foo(bar)
+Provides:       %{?scl_prefix}foo(bar)

All the other tag definitions should be unchanged, unless they contain %{name} macro, which may need to be substituted for %pkg_name (for example in SourceN tag, where it may be a part of URL).
There are some additional tags you should add. The package must require %{scl}-runtime, unless it depends on another package that requires %{scl}-runtime.
- Here is a rule of thumb: If packaging SCL with a language interpreter, like Ruby or Python, typically all other packages in the SCL depend on the interpreter. Therefore it is sufficient when only the interpreter runtime package requires %{scl}-runtime. Generally, every SCL package that can be installed on its own (without other packages from this SCL) should require the %{scl}-runtime package.
- The line to add is:

%{?scl:Requires %{scl}-runtime}

Subpackages

The same rules as for normal tags apply for subpackages tags. The only thing needs to be changed:

If (and only if) a package define its name with -n, the name must be prefixed with %{?scl_prefix} like this:

-%package -n foo
+%package -n %{?scl_prefix}foo

This applies not only to %package macro, but also for %description and %files.

Inter-SCL Dependencies

Available from scl-utils version 20120613.

There are situations where packages from an SCL have to depend on packages from another SCL. While using simple [Build]Requires: ruby193-rubygem-rails is possible, it is not the wisest thing to do, since it hardcodes the information about how %{?scl_prefix} gets expanded. If the expansion of this macro was changed (for example from ruby193- to scl-ruby-193, all the packages using such dependencies would need to be manually altered and rebuilt.

The scl-utils package therefore provides two ways of inter-SCL dependencies using macros. Note that both of them are in fact functions, so they need to be invoked without the curly braces to work properly:

%scl_require foo macro is used to depend on the whole SCL. This is used if you want to use a functionality of an SCL, no matter what packages that SCL will install along the way. For example, the ruby193 SCL contains the whole Ruby on Rails stack in version 3.2.3. If packaging an SCL that requires the whole Rails stack, you can simply use %{?scl:Requires: %scl_require ruby193}.
%scl_require_package foo pkg provides the ability to depend on a specific package from another SCL. For example, if your package needs rubygem-minitest package from ruby193 SCL, you can simply use %{?scl:BuildRequires: %scl_require_package ruby193 rubygem-minitest}. Installing the whole Rails stack in this case would be useless (and since rubygem-minitest is only a build dependency, it's not drawn in by default).

Note that both of these macros abstract from the way that %{?scl_prefix} expands. Therefore, if any changes were made, a simple rebuild with new versions of SCL utils would be sufficient.

When specifying inter-SCL dependencies, these macros must be used.

Inside RPM Scripts

It is not possible to describe the general process of rewriting %prep, %build, %install, %check and the %pre* and %post* scripts. There are however some general rules:

Substitute all occur

@@ Line 1: / Line 1: @@
 == Naming the SCL ==
 Every SCL '''must''' have a name which does not match name of any other package. This is implied by the fact that the SCL metapackage's name is the same as name of the whole SCL. It is therefore not a good idea to name an SCL "ruby", when there is a "ruby" package - the name has to be altered. Packager '''should''' include something specific to the SCL (usually a specific version of included library) into it's name. Therefore "ruby193" is a good name for a collection that contains Ruby 1.9.3, etc.
+{{admon/tip|Automation|'''spec2scl''' was written to automate the task of converting specfiles to scl-enabled specfiles as much as possible. Install it by <code>yum install spec2scl</code> or use upstream version from https://bitbucket.org/bkabrda/spec2scl.}}
 == SCL Metapackage ==
@@ Line 158: / Line 160: @@
 === Inside RPM Scripts ===
 It is not possible to describe the general process of rewriting <code>%prep</code>, <code>%build</code>, <code>%install</code>, <code>%check</code> and the <code>%pre*</code> and <code>%post*</code> scripts. There are however some general rules:
-* Substitute all occurencies of <code>%name</code> for <code>%pkg_name</code>. Most importantly, the <code>%setup</code> macro will need the <code>-n</code> argument for SCL builds (thanks to <code>%pkg_name</code>, we can use the same for non-SCL builds):
+* Substitute all occur
-<pre>
--%setup
-+%setup -n %{pkg_name}-%{version}
-</pre>
-* If using a <code>%_root_*</code> macro to point to the original filesystem, you '''must''' conditionalize it, so that the package can be rebuilt for non-SCL use:
-<pre>
--mkdir -p %{_sysconfdir}
-+mkdir -p %{?scl:%_root_sysconfdir}%{?!scl:%_sysconfdir}
-</pre>
-* If building SCL packages that depend on other SCL packages, you might need the <code>scl enable</code> functionality to link properly/run proper binaries, etc. It is not generally possible to say where this will be needed, but an example might be compiling against an SCL library or running an interpreted script with the interpreter in SCL:
-<pre>
-+%{?scl:scl enable %scl - << \EOF}
- ruby foo.rb
- RUBYOPT="-Ilib" ruby bar.rb
- # more stuff
-+%{?scl:EOF}
-</pre>
-* All hardcoded paths (such as <code>/usr/share</code>) '''must''' be replaced with proper macros (<code>%{_datadir}</code> in this case).
-=== Files ===
-The files section is usually OK as it is. The only adjustments needed are for the names of subpackages (see [[#Subpackages|Subpackages]]) and possible of some path macros with their <code>%_root_*</code> alternatives (this is determined by what you do in the scripts, see [[#InsideRPMScripts|Inside RPM Scripts]].
-=== Dealing With Automatic Provides/Requires and Filtering ===
-RPM has some automatic Provides/Requires searching capabilities as well as filtering capabilities. For example all Python libraries have an automatically added <code>Requires: python(abi) = (version)</code> (and in SCL, the proper way to have this is <code>Requires: %{?scl_prefix}python(abi) = (version)</code>). The scripts that search for these dependencies must sometimes be rewritten for SCL, as the original RPM ones are not extensible enough (and in some cases, filtering is not usable). This is the example of rewriting Python provides and requires (the following lines can be placed into the <code>macros.%{scl}-config</code>):
-<pre>
-%__python_provides /usr/lib/rpm/pythondeps-scl.sh --provides %{_scl_root} %{scl_prefix}
-%__python_requires /usr/lib/rpm/pythondeps-scl.sh --requires %{_scl_root} %{scl_prefix}
-</pre>
-The <code>pythondeps-scl.sh</code> is a file created from <code>pythondeps.sh</code> by adjusting some search paths.
-If there are some Provides/Requires, that you need to alter (for example <code>pkg_config</code> provides), there are two ways to do it.
-* Either with the following lines in <code>macros.%{scl}-config</code> (this will then apply to all packages built in the SCL):
-<pre>
-%_use_internal_dependency_generator 0
-%__deploop() while read FILE; do /usr/lib/rpm/rpmdeps -%{1} ${FILE}; done | /bin/sort -u
-%__find_provides /bin/sh -c "%{?__filter_prov_cmd} %{__deploop P} %{?__filter_from_prov}"
-%__find_requires /bin/sh -c "%{?__filter_req_cmd}  %{__deploop R} %{?__filter_from_req}"
-# Handle pkgconfig virtual [provides/requires].
-%__filter_from_req | %{__sed} -e 's|pkgconfig|%{?scl_prefix}pkgconfig|g'
-%__filter_from_prov | %{__sed} -e 's|pkgconfig|%{?scl_prefix}pkgconfig|g'
-</pre>
-* Or in every single specfile that you want to filter Provides/Requires in, place following lines after tag definitions:
-<pre>
-%{?scl:%filter_from_provides s|pkgconfig|%{?scl_prefix}pkgconfig|g}
-%{?scl:%filter_from_requires s|pkgconfig|%{?scl_prefix}pkgconfig|g}
-%{?scl:%filter_setup}
-</pre>
-Filters '''don't work in EL-6''', because rpm is using different filter mechanism. It will be solved by [https://bugzilla.redhat.com/show_bug.cgi?id=1001674 documenting] this behaviour.
-=== Dealing With Macro Files ===
-Sometimes the package ships with macro files, that go into <code>/etc/rpm</code> (in SCL terms, they go into <code>%{?scl:%{_root_sysconfdir}}%{!?scl:%{_sysconfdir}}</code>). This is fine, if two conditions are met:
-* The macro files '''must''' be renamed by appending <code>.%{scl}</code> to their name, so that they don't conflict with system files.
-* The defined macros '''must''' be present in a form, that doesn't break any macros from non-SCL packages (with exception of macros from <code>%{scl}-build</code> subpackage, as mentioned below).
-The second rule means, that macros in macro files must be unexpanded or be properly conditionalized. This is fine:
-<pre>
-%__python2 %{_bindir}/python
-%python2_sitelib %(%{?scl:scl enable %scl '}%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"%{?scl:'})
-</pre>
-and this is '''not''' fine:
-<pre>
-%__python2 /opt/rh/python27/root/usr/bin/python
-%python2_sitelib %(%{?scl:scl enable %scl '}%{__python2} -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())"%{?scl:'})
-</pre>
-Now consider a situation where you need to create collection <code>mypython</code> that depends on <code>python26</code> collection that defines macro <code>%{__python2}</code> in a way shown above. The macro will evaluate to <code>/opt/<provider>/mycollection/root/usr/bin/python2</code>, but the <code>python2</code> binary is not in your collection, but in <code>python26</code> - this means nothing can be built in depending collection now. When you need to follow this use-case, here is what you need to do:
-* In <code>macros.python.python26</code> (part of python26-python-devel), define
-<pre>
-%__python26_python2 /opt/<provider>/python26/root/usr/bin/python2
-</pre>
-* And in macros file in <code>python26-build</code> and any depending collection's <code>%{scl}-build</code> use:
-<pre>
-%scl_package_override() {%global __python2 %__python26_python2}
-</pre>
-This will redefine the <code>%{__python2}</code> macro only if such <code>%{scl}-build</code> package is present (which usually means that you want to build for the collection and don't mind breaking the macro for local non-SCL builds).
-=== Dealing With Shebangs ===
-Shebangs have two important aspects: they are processed by the automatic dependency processor and they point to a certain (possibly non-SCL location).
-As one of its functions, the automatic dependency processor goes through the shebangs. And adds dependencies according to the binaries they point to. From the SCL point of view, there two types of shebangs:
-* <code>/usr/bin/env foo</code>
-** 'auto-dependency generation' point of view: The resulting package will depend on <code>/usr/bin/env</code>, which is not a problem.
-** 'pointing to non-SCL location' point of view: If the <code>$PATH</code> is redefined properly in the enable scriptlet, the <code>foo</code> binary is found in the SCL hierarchy, so not a problem either.
-** But you generally should avoid this. Packages that belong into collection '''should''' make sure that they invoke collection binaries, so they should hardcore full path to them in shebangs.
-* <code>/usr/bin/foo</code>
-** 'auto-dependency generation' point of view: The resulting package will depend on <code>/usr/bin/foo</code> (part of non-SCL package), but we may have wanted to depend on <code>%{?_scl_root}/usr/bin/foo</code> when building for SCL.
-** 'pointing to non-SCL location' point of view: The <code>$PATH</code> redefinition has no effect, the <code>/usr/bin/foo</code> binary is still used (which is probably not what was intended).
-** If you have some shebangs like this in the specfile and the should point to SCL locations when built for SCLs, you can use similar command to adapt them:
-<pre>
-find %{buildroot} -type f | \
-  xargs sed -i -e 's"^#!/usr/bin/foo"#!%{?_scl_root}/usr/bin/foo"'
-</pre>
-== Building Packages ==
-The converted packages '''should''' be buildable both in SCL and non-SCL buildroots.
-Before building in mock, you will want to create SRPMs. The process here is standard - if the SCL macroes are properly conditionalized, everything will work even without the SCL metapackage and <code>scl-utils-build</code> installed locally. However, for the build of the metapackage itself, you will need <code>scl-utils-build</code>, otherwise <code>rpmbuild</code> will be unable to parse the macros inside it.
-For testing the SCL builds in mock (you will get the same environment in BREW), you have to create a standard config file with few adjustments (let's say that you are building SCL named <code>ruby193</code>):
-* Use <code>'install @build scl-utils-build ruby193-build'</code> for <code>config_opts['chroot_setup_cmd']</code>.
-* Add a local repo with your builds of the SCL packages to the mock config file:
-<pre>
-[ruby193]
-name=ruby193
-enabled=1
-baseurl=file:///home/bkabrda/ruby193/repo
-cost=0
-metadata_expire=0
-</pre>
-Testing non-SCL builds is done in normal mock configurations (obviously :) ).

Search