Changes/Enable Spec File Preprocessing

Enable spec file preprocessing

Summary

This change should enable an opt-in spec file preprocessor in Fedora infrastructure for the benefit of packagers. The preprocessor allows some very neat tricks that were impossible before, for example generate changelog and release automatically from git metadata or pack the entire dist-git repository into an rpm-source tarball (effectively allowing unpacked repos to live in DistGit).

Owner

• Name: Michal Novotný
• Email: clime@fedoraproject.org

Current status

• Targeted release: Fedora 34
• Last updated: 2020-12-23
• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

There is a recently added feature into mock: the rpkg preprocessor which, if enabled, introduces an intermediate step just before srpm building. This step consists of running the spec file through a text preprocessing engine that includes an already present library of macros designed specifically for rpm spec file generation from git metadata. This library is called rpkg-macros. The macros there allow packagers to have their %changelog, Release, Version, VCS tag, or even Source fields automatically generated from dist-git repository data and metadata. The library can be easily extended in future to support more packager use-cases or even a completely new library can be developed that doesn't look at git metadata at all and instead, for example, analyses already present tarball content to render spec file based on upstream information. This doesn't mean it will happen but the framework is generic enough to support that. There is also support for user-defined macros that are loaded on-demand from a file placed alongside the package sources, maintained by packager. This feature wouldn't be enabled by this change from start but it's an example of freedom that the preprocessing framework is able to provide. Enabling this change should be very easy, basically adding:

config_opts['plugin_conf']['rpkg_preprocessor_enable'] = True

into mock configuration of Koji builders and using at least mock 2.7. Some very minor change may be also needed in Koji regarding the spec file lookup.

Even if the change is enabled on the infrastructure level like this, the packager will still need to opt-in to use the preprocessor. The opting-in is done by placing rpkg.conf file into the package top-level directory with the following content:

[rpkg]
preprocess_spec = True

When this is done by a packager, the preprocessor will be finally enabled for the given package.

Alongside, there is an ongoing work to add the preprocessor support into the rpkg python library so that a packager can easily work with the spec files containing the preprocessor (rpkg) macros: https://pagure.io/rpkg/pull-request/530

Also, the following pull requests were open to add support for preprocessing into python version of spectool and into the dnf builddep command:

• https://pagure.io/rpmdevtools/pull-request/73
• https://github.com/rpm-software-management/dnf-plugins-core/pull/420

The macros are supported since epel6 until the most latest Fedora (preproc, rpkg-macros, and rpm-git-tag-sort packages are needed). The spec preprocessing step in mock happens in a target chroot just before the srpm build.

Benefit to Fedora

This change offers solution to some long-standing issues in Fedora around packaging (i.e. automatic release and changelog generation) while also offering some interesting future options (for example unpacked dist-git repos). The big advantage of this approach is that it is explicit. Spec file stays the source of truth and by looking inside one, you will be able to determine how the text will expand for a certain git repository state.

Scope

• Proposal owners:

For the very basic support, probably a small patch in Koji is needed to be able to lookup not only .spec files but also .spec.rpkg files (the .spec.rpkg extension explicitly states that the spec file is a template). Also the rpmdevtools/rpmdev-bumpspec script should be tweaked to be compatible with spec files using the macros.

• Other developers:

Some optional help with rpmdevtools/rpmdev-bumpspec changes would be welcome.

• Release engineering: #9910 (a check of an impact with Release Engineering is needed)

Enabling the rpkg_preprocessor plugin in mock config for Koji builders.

• Policies and guidelines:

The new macro support should be mentioned or even described in the packaging guidelines. We should decide if the full power of the rpkg-macros library should be allowed from start (i.e. even unpacked repos).

• Trademark approval: N/A (not needed for this Change)

• Alignment with Objectives: N/A

Upgrade/compatibility impact

Because of the opt-in nature on packager side, there should be no compatibility issues.

How To Test

Once the feature is enabled, one can test it by providing the rpkg.conf file with the required content in a package repository and use some rpkg macro in the spec file: e.g.

Name: {{{ git_dir_name }}}

to generate the name of the package from the repository name (this should actually produce the original text as package names should be the same as the repository basenames).

To try it out first without committing to dist-git, one can use rpkg command-line tool from https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/ or even fedpkg's koji scratch build after the work in the pyrpkg library is finished.

One can also currently use Copr's SCM "rpkg" build method where the macros are enabled but the rpkg-macros there are in version 2 whereas this change is about introducing the version 3 rpkg-macros. However, while there are some differences between v2 and v3, the idea and most of the working is the same.

User Experience

This change is intended for packagers. It should help to make a bit of their work easier and offer them some new interesting options.

Dependencies

N/A

Contingency Plan

Packagers can opt-out on individual basis by removing the rpkg.conf file or just setting the preproces_spec property to False. On infrastructure level, the rpkg_preprocessor plugin could be disabled again.

Documentation

- Mock's rpkg_preprocessor plugin

- rpkg-macros reference (the library of macros ready to be used in spec files)

Release Notes

Currently N/A