AxelThimm/kmdls

This scheme has been submitted to voting. See AxelThimm/kmdl/voting

Packaging kernel modules

Packaging kernel modules in current packaging systems like rpm and deb is a challenge because kernel modules are very tightly associated with a respective kernel package. Due to the lack of a stable kernel API new upstream kernels usually require rebuilds of such kernel modules. The lack of a stable kernel API is part of the kernel's development model and the reasons for not having a stable kernel API ever are outlined in the kernel's documentation in the file stable_api_nonsense.txt.

upgrade paths

The main difficulty with kernel module packages are the fact that we have two sets of versioning (aka epoch/version/release, EVR) involved:

• that of the module itself
• that of the kernel built for

The requirements to a random set of kernel module packages (for the same module) are as follows:

• kernel module packages built for different kernels (where kernel includes the kernel flavour, e.g. 2.6.xx-yy and 2.6.xx-yysmp

are considered different kernels in this context even though they were built from the same sources) do not interfere in any way.

• kernel module packages built for the same kernel and differing in the versioning (EVR) of the module itself should behave

like normal packages, e.g. an (rpm-)newer package should replace the older package (but 'only' for this kernel, other kernel modules from other kernels remain unaffected as outlined in the previous list item).

This means that the kernel module package needs to carry the information about which kernel package it has been built for for rpm and higher depsolvers to act according to the requirements above. We also need to disambiguate the packages for different kernels on the package filename level, e.g. the kernel package's EVR needs to enter the kernel module package's filename. Which part of the filename will be discussed in the next section.

The EVR of the kernel package in Fedora and RHEL is simply the uname -r of the kernel, so effectively we need to strategically place the kernel's uname -r into both the package's filename as well as in package relations within the package.

uname -r in name

The previous section showed that kernel module packages need to carry a two-dimensional versioning, but rpm and similar packaging tools have only one-dimensional versioning support. The split in epoch/version/release does not change that, at the end the EVRs are placed in an ordered relation in one dimension. The only way to escape rpm ordering is to have a different rpm names in the package. E.g. the ordering splits into equivalence classes where the representation of each class is the package's rpm name (which is only part of the filename).

In order to fullfil the requirements above we need to decide where to place the kernel package's EVR (the uname -r) in the package's filename. We can choose between

• the rpm name
• the rpm version
• the rpm release

Placing the uname -r in the version/release part means to merge together unrelated versioning information and creating an object that rpm will happily order into its one-dimensional upgrade path. That is not what we need! As outlined in the requirements some packages should not be comparable at all while others should follow conventional upgrade mechanics.

The way out is to

• properly disambiguate kernel module packages for different kernels in a way that rpm and higher level depsolvers

never relate the packages to each other.

• keep the kernel's EVR (the uname -r) and the module's cleanly separated! And use the module's EVR within a kernel for

the upgrade mechanics.

We can get both items solved by placing the uname -r into the rpm name of the package:

• kernel module packages are now completely independent across kernels.

• within a kernel the kernel modules are normally updated w/o any further action.

Introduction to kmdls

kmdls are an acronym for kernel module packages. They are being used at ATrpms for several years now.

The kmdl scheme is constituted out of two parts, an abstract macro interface and (several) implementations. The implementation may differ either to support different schemes (in fact even the current kernel module scheme could be supported) or to adapt to a local build system (for performance improvements).

kmdl interface

The kmdl interface design has the following key elements

• All parts of an external kernel module project are described in one specfile (therefore also only one src.rpm). This

includes userland and kernel-land build and install instructions. The kmdl macros provide conditionals to distinguish userland-only vs kernel-only parts.

• The specfile and src.rpm are kernel flavour agnostic. The kernel flavour is more or less the different .config files that

kernels are built against the same kernel source, for example smp, xen0, xenU, xen, hugemem, PAE, kdump etc. Therefore no changes in specfile/src.rpm are needed when new kernels or kernel flavours show up!

• userland subpackages follow a simple, conventional naming/versioning (they don't need to be double-versioned)

• The interface is abstract enough to work on RHEL and even ancient RHL distributions. While this is not an immediate goal of

Fedora it comes at no price.

A simplified sceleton specfile

The following is a real-life example of a very simple kernel module project that does not contain userland parts. Still the repective placeholders are in place to illustrate where explicit userland build/install instructions would be placed.

%kmdl arc4

Summary: ARC4 Cipher Algorithm
Name: arc4
Version: ...
Release: ...
...
%kmdl_parentdependencies

%package -n %kmdl_name
%kmdl_dependencies
Summary: Kernel module for the ARC4 Cipher Algorithm
Group: System Environment/Kernel

%description -n %kmdl_name
This package contains kernel drivers for the ARC4 Cipher Algorithm.

%kmdl_desc

...

%build

%if %{kmdl_userland}
%else

%kmdl_config
make -C %{kmdl_kernelsrcdir} SUBDIRS=<code>pwd</code> modules

%endif

%post -n %kmdl_name
%kmdl_install

%postun -n %kmdl_name
%kmdl_remove

...

%if %{kmdl_userland}

%files
%defattr(-,root,root,-)

%else

%files -n %kmdl_name
%defattr(-,root,root,-)
%{kmdl_moduledir}/crypto/arc4.*o

%endif

Explanation of example specfile/selected macros

• The package starts with declaring that it supports a kernel module package called "arc4".

• The rest of the main package preamble continues as usual.

• If the userland parts of the package requires the kernel module parts (true in 95% of all cases) %kmdl_parentdependencies

sets up the proper dependencies

• The kernel module subpackage name is %kmdl_name. The interface doesn't need to know about uname -r specifics, that's the

implementation's job. That also makes the specfiles easier to read.

• %kmdl_userland is a conditional macro indicating whether userland bits or kernel modules are built

• %kmdl_kernelsrcdir is the location of the kernel sources for the kernel we want to built the kernel modules for.

• %kmdl_moduledir is the installation directory for the kernel modules.

These are the basic macros needed to write simple kernel module specfiles. There are more macros for the more demanding kernel module projects. The implementation details are nicely hidden away in a KISS fashion - there is no hardwiring of any information that can change from release to release or distribution to distribution. RHEL and RHL can be supported by the same scheme (and in fact are supported at ATrpms).

kmdl implementation

We'll look at a kmdl implementation for recent Fedora Core distribution supporting uname-r-in-name idiom and kernel module subpackages' names compatible to the reference implementation at ATrpms ("foo-kmdl-uname -r" rpm names). Some of these macros are simplified and some omitted (RHEL and RHL support is stripped - the intention is an inductive introduction, not a reference implementation).

%_kernel (uname -r) and %kmdl_kernelsrcdir

%_kernel %(uname -r)
%kmdl_kernelsrcdir /lib/modules/%{_kernel}/build

%_kernel should never be used in the specfiles. It defaults to the running kernel's uname -r and is used to compute a default for kernel's source location in %kmdl_kernelsrcdir as well as the uname -r part of the kmdl's name (see below).

%kmdl

%kmdl() %{expand:%%global _kpkgname %1}

This macro set's the kernel module project's default name. The kmdl subpackage's name and several dependency releations are computed out of this name.

%kmdl_name*

%kmdl_name %{_kpkgname}-kmdl-%{_kernel}
%kmdl_nameof() %1-kmdl-%{_kernel}
%kmdl_namepure %{_kpkgname}-kmdl
%kmdl_namepureof() %1-kmdl

These macros compute rpm names of kmdl as well as a kernel agnostic name for the kmdls. The kmdls provide this name and the userland package depends on this name only. That way the userland dependencies never contain explit kernels!

There are two variants, one using the default name as provided by %kmdl, and one taking an argument. We need that to express relationsships to other kmdls. Real life examples are

Requires: %{kmdl_nameof iee80211} >= ...

in ipw* packages or Conflicts: between different v4l2 kernel module packages.

%kmdl_parentdependencies

%kmdl_parentdependencies Requires: %{kmdl_namepure}-%{evr}

This is the kernel-agnostic userland dependency mentioned above.

%kmdl_dependencies

%kmdl_dependencies\
Requires: /boot/vmlinuz-%{_kernel}\
Requires: /sbin/depmod, modutils >= 2.4.14\
BuildRequires: %_kernelcompiler\
Provides: %kmdl_namepure-%{evr}\
Provides: %kmdl_namepure-%_kernel = %{evr}\

This provides the kernel-agnostic dependency mentioned above as well as dependencies on the kernel built for. The dependency on /boot/vmlinuz-%{_kernel} is an element of compatibility to RHEL/RHL/FL, it could be replaced by kernel-devel (but then again, why should one drop RHEL/RHL support when it doesn't cost anything). This is mentioned to emphasize that kmdl implementations may differ.

%kmdl_install and %kmdl_remove

both are defined as

depmod -ae -F /boot/System.map-%{_kernel} %{_kernel} > /dev/null 2>&1 || :

But they are left different and abstract to allow future changes w/o specfile rewrites.

Summary of the kmdl scheme

The kmdl scheme is clean and KISS. It has proven to work reliably for several years now and in several more complex or corner case kernel module packages (like no userland to kmld dependencies wanted, several sub-kmdls per kernel out of one specfile, userland builds requiring presense of kernel sources).

It respects rpm's ordering by moving kernel modules for different kernels out of rpm's comparison algorithms using the uname-r-in-name idiom. kmdls for the same kernel are properly rpm-upgradable, there is no special handling needed in rpm or higher depsolvers, the scheme is completely rpm-conformant.

The userland/kmdl builds are organized in one specfile as subpackages which is the sane thing to do when the sources are a common tarball. This keeps maintenance low.

There are two items that are not perfect, though:

• kmdl names are ugly: The uname-r-in-name is difficult to get accustomed to at first. After recognizing that the uname-r

needs to be in the package filename somewhere and that it even serves a purpose in the rpm name one needs to get over it. This is not a beauty contest, it's a technical requirement!

• kmdls for new kernel packages are not automatically coinstalled by any of the high-level depsolvers. But it turns out that

adding such support is really trivial (less than 100 lines of python code for a yum plugin).

Comparison to current kernel module packaging scheme ("kmod")

See also AxelThimm/kmdls/kmods_vs_kmdls_at_a_glance

So what are the explicit differences between the kmdl scheme and the current kernel module scheme ("kmod")?

• The kmod scheme is broken on the rpm level. There is no way this can be fixed w/o using uname-r-in-name!!!.
  

As it merges the EVRs of the module and the kernel into one EVR rpm cannot know whether a package that is to be installed needs to remove other packages of the same name or not. It's even worse than that: rpm does have two different mode of operations, installing (-i) and upgrading (-U) to notify rpm whether old packages should be removed or not. But neither applies to a typical kernel module installation: Some kernel modules need to remain untouched' (these of different kernels) and some removed in the same transaction (these of the same kernel)

• The kmod breakage on rpm level is inherited to all rpm-depsolvers like yum/up2date/smart/apt/etc.
  

Even if the decision were to keep the rpm breakage and disallow direct usage of the rpm client, each of these depsolvers would need special workaround in form of patches or plugins to change their rpm compatible comparison. Otherwise these depsolvers will simply nuke running kernel modules (like the one for your graphics card) and/or fail on file conflicts or mixed module version coinstalls' (the latter if the file paths are changed). It turn out that this path is probably unfixable.
kmdl also needs special depsolver love otherwise new kernel upgrades will not coinstall kmdls for this kernel. But the consequences are not comparable: No running or old kernel is affected by the missing depsolver support. Manually coinstalling the kmdls for an arbitrary depsolver is 9 lines of bash!!! A yum plugin doing that automatically is trivial and already finished. In general:
depsolver support for kmdls is both just nice-to-have and easier to implement than kmod support.

• The kmod scheme uses pairs of specfiles/src.rpms
  

If a kernel module project publishes patches or such are cut from a SCM these need to be applied to both src.rpms. Same is true for changelog entries. Keeping sources/changelogs/versioning in two src.rpms is error-prone and increases maintenance. No one would create new src.rpm per subpackage in other packaging scenarios for the same reason.

• The kmod specfiles contain both kernel and flavour references
  

This creates a need to keep different specfiles for different releases and to update the specfiles/src.rpms upon each kernel rpm release (Note: It was discussed on the list, that this due to the buildsystem).

• The kmod scheme provides a nicer looking filename/rpm name for kernel modules packages.

• $1 in %post and all the other scriplets is broken in kmod, works in kmdl.

• very good support for custom kernel packaging in kmdls, for example the kernel-suspend2 packages at ATrpms.

The first two items alone are blockers for using the kmod scheme. The only benefit the kmod scheme has is that the uname-r part of the kernel is moved to the end of the filename making the package look nicer, but at a very high cost.

Cascades of workarounds for kmod (pandora's box)

As explained before the kmod scheme merges the two evrs of the kernel and the module into one. For rpm therefore all kernel module packages seem like having the same name and being part of one upgrade path. At first the authors of kmod considered kmod packages to be like kernels, in fact that's why they chose this broken design. But this is not the case as shown before, kernels are always coinstalable packages while kernel modules are not.

Let's see what the set of current yum workarounds for kmod support look like and why it is still not enough and probably is not fixable at all:

1.#1 kmods as they stand w/o any further special handling would only install for the latest kernel. This is because they all share the same name even when they belong to different kernels. Since the merged evr prefers module version over kernel's the preferred package (the one rpm considers newest) is the one which is the latest in module version and then for packages with the same module version the one with the newest kernel.
Note that there are two issues:

• There is only one latest package for all module/kernel versions.
• This latest package nukes all others
  

Note that only this issue is further addressed. At this point so much still does not work that the issue of only one kernel support is overseen. But it takes revenge later on. 1.#2 The requirement to allow some kind of coinstallation instead of nuking and the initial (false) thought of kmods being treatable like kernels, e.g. always coinstall, lead to yum adding support for kmods that is the same as that for kernels. This means that

• yum considers these packages as coinstallable
• yum always coinstalls (only) the latest package

1.#3 But then the issue of coinstallating packages that should had been (partially) upgraded came up. This was attempted to be "fixed" by a plugin, the so called fedorakmod plugin. It works in the following way:

• Let yum compute the set of changes to perform. As described in the workaround before yum's in-core support for kmods leads yum to

try to coinstall the latest package for the latest kernel (and for the latest kernel only, the revenge is near).

• Before yum commits the transaction the plugin removes the packages that should had been upgraded.

1.#4 Now we check what the current status quo is. yum will only coinstall/upgrade kernel modules for the very latest kernel!!!
This is because yum's coinstallation support is doing just that. It was a copy of the kernel package support (in fact kmods are treated exactly like kernel packages in yum). For kernel packages yum semantics properly assume that coinstalling should really only happen for the latest kernel. Coinstalling older kernel packages that the user removed from his system makes no sense. This is not true for kmods! You do want to support the last kernel, too, currently you cannot even have a security update for anything but the latest kernel ever once you start offering kmods for the next kernel.
The current setup only allows supporting one kernel!
But it will get worse:
The current setup disables or nukes old kernel modules!
So even though there may exist kernel module updates for older kernels in the repo they get discarded. Now if a kernel module depends on a userland package (like a firmware package), and this gets updated w/o the kernel module being updated, this kernel module will either not work, or if the dependencies were strict, will get uninstalled due to missing dependencies.
There are at least two real-life examples where this happens: ipw2x00 and the firmwares and ipw3945 and the ipw3945d daemon. In these cases the old kernel would get it's wireless modules deprived.
Now consider the case that 3w-9xxx or a qlaxxx module were affected, the old kernel would become unbootable, if the root fs lives on such a filesystem. 1.#5 Being used to the cascades of workarounds one will by now know what comes next. In order to fix this breakage yum would have to have special in-core handling for kmods. At first this may look simple to do - just tag those packages as not only-coinstall-the-latest, but coinstall-them-all, so that the old kernels still get their kernel module updates. 1.#6 Of course this is not as easy. Now yum sees and wants to install too many kmods including old ones that are already superseeded on disk.. No problem, we're having workaround parties: Have the plugin partially undo yum's package choices. E.g. if the installed package for a kernel is newer remove the package from the transaction set. 1.#7 Finally we arrive at the next bug station. Undoing a package selection is generally nearly impossible. Consider a yum install foo operation. foo may pull in lots of foo depedencies, may override bar and conflict with baz. Therefore yum install foo induces a larger package change. It even gets works when there are more install arguments or a yum update. Posteriori you cannot really know whether bar was pullen in only due to being a dependency of foo, it could had been a dependency of something else, too.
Real life examples are kmdls that require (different) firmware packages, require userland daemons, require other kmdls or conflict with them. This mostly affects wireless and v4l2 kernel projects, but also lm_sensors/i2c. 1.#8 At this point in order to rectify this the plugin needs to learn more about yum's resolving and even in a way yum itself doesn't know, e.g. inverting a package resolve operation. This seems like a dead-end situation.

Now we need to make a reality check:

• The kmod and yum-plugin method is a can of bugs. The more bugs you fix the more and worse come up. This comes not

unexpected (although for some it does): it is a propagation of the basic design flaw outlined in a previous section.

• The whole system is unmaintainable due to subtle bugs creeping up from all cracks.
• It is already twice the code that is needed for kmdl support and will grow to several times this size if the bugs above are to be

addressed.

• In contrast kmdl full support for yum is only 99 lines of python. It doesn't have any design bugs, and already works with

ATrpms' kmdls which probably exhibit all typical as well as all possible corner cases.

• Whatever the fedorakmod plugin will continue to try fixing, the kmod scheme will remain severely broken wrt rpm. So even if

yum were working properly we still have rpm to worry about.

• All other depsolvers are kmod-broken, either they are based on yum and share the above kmod issues, or have their own

plugin mechanism which will exhibit the same set of kmod-bugs as above for yum. For kmdls only the trivial coinstall support is missing.

We need to conclude that kmod's design peculiarities break rpm and depsolvers and that even partially fixing this breakage comes at a large cost as well as being possibly unfixable.

Various

The following lists some (non-)issues that reappear frequently on the list.

debuginfo

Ville Skyttä raised the issue of debuginfo packages. These are properly generated in the kmdl scheme w/o any specfile magic. See http://www.redhat.com/archives/fedora-packaging/2006-August/msg00053.html for details

kabi

There are efforts to introduce kernel abi metrics that will make the life of IHVs easier. But these do not really affect any of this discussion. Still we need to examine what the kabi projects are delivering to see what their use is and how and whether this interacts with packaging kernel modules.

about kabi

The kernel api is not stable. This is even by design/politics and will not change. Therefore the kernel abi is also changing with its api. This increases workload of IHV that need to support many different kernels with many different builds.

Since upstream development will never have a stable api/abi, a frozen kernel may be considered abi-stable. So if a distributor decides to only apply security fixes to a kernel he can argue that his kernel has an abi.

But kernels - even in RHEL - get updated beyond only security updates. Every quaterly RHEL update currently introduces new wireless extension updates changeing api and abi and breaking all wifi kernel modules, not only on binary level, but on source, too. So for wireless drivers no kabi would be of help. But what about the grpahics drivers? If a kernel update didn't touch vm and other needed subsystems then the graphics driver could be recycled.

Therefore a kabi metrics system is created. This creates hashes out of symbols and whole subsystems. If the hashes are equal then the kernel module can be reused, otherwise not. module-init-tools can then check foreign kernel modules for compliance with the running kernel's 'kabi'.

In this sense the 'kabi' is a set of hashes generated out of the kernel's exported symbols/signatures and can be used to check reusability of foreign kernel modules

But the term kabi is also used for a distribution specific abi labelling, we'll call this term the kabi-version to keep it apart from the above 'kabi'. The set of kernels within quaterly RHEL updates with only security updates and therefore compatible kabi could be given a kabi-version of 5.0. The next update would have a kabi-version of 5.1 and so forth. Now kernel module packages could instead of being disambiguated as foo-kmdl-2.6.18-12.EL5 be called foo-kmdl-5.0 and be valid for 2.6.18-12.EL5, 2.6.18-12.0.1.EL5 and 2.6.18-12.0.2.EL5 (the security updates of RHEL5U0).

This lowers the IHV's costs for offering external kernel module for RHEL as you only need to do so for every quarterly update.

Why this is not relevant

• The per kernel (or kabi) disambiguation is still needed. The identification tag that will seperate builds for different kernels or

kernel abis will still need to be part of the rpm name.

• Fedora kabi-version will change with each kernel update. It only adds a fake kabi-version on each update.
• The costs of rebuilding kmdls is inexistent in Fedora, and as said the kabi would change on every kernel update, so there wouldn't be

any less builds.

• A unified scheme for FC and RHEL shouldn't be based on kabi-versions. The Fedora one would fluctuate too much and users would only be

confused.

Why kabi is even an argument in favour of kmdls

• As explained at the very begining the kmdl design is an interface/implementation design. Even if there would be a replacement of

uname-r with a kabi value the specfile and src.rpm would remain the same. Only the macro implementation would change. kmdls are therefore a flexible future-proof solution.
In fact the same specfile/src.rpm could generate uname-r-in-name on FC and a kabi variant in RHEL. Not that this is something I recommend, but it shows the flexibility of the approach.

too late

Thorsten Leemhuis argues that it is too late to change the kmod standard (while suggesting even more radical changes himself) (note from thl: "too late to fix it before FC6 / RHEL5"). It is never too late to fix something that is that broken. You wouldn't ship a kernel that doesn't boot, too.

kmod inertia

Thorsten Leemhuis also argues that there are too many kmod packages to start changing them. Currently there is exatcly one such package in Fedora Core and Extras. Some more are planned, most notably GFS for Fedora Core and RHEL. It is important to not let the flawed design make it's way into Fedora Core or RHEL!