More Go packaging

This is an enhancement proposal to PackagingDrafts/Go. It builds on the hard work of the Go SIG and reuses the rpm automation of PackagingDrafts/Go when it exists, and produces compatible packages.

The proposal builds the Go integration inside rpm, and factors out common tasks. It tries to limit spec content to items where the packager adds actual value, and makes it easy to adapt to upstream changes. It uses rpm facilities to auto-compute Requires and Provides. Note that Go has not standardized on a common component tool yet, therefore the auto-produced dependencies are highly granular and lacking versionning constrains.

This proposal achieves a drastic reduction of Go spec sizes, up to 90% in some cases, not counting changelog lines. It often removes hundreds of high-maintenance lines per spec file. It increases quality by enabling stricter checks in the factored-out code, without relying on packagers to cut and paste the correct snippets in their spec files. It aggressively runs all the unit tests found in upstream projects. It does not try to rely on bundled Go code, to hide upstream renamings, to avoid rebasing packages when their dependencies change.

This proposal relies heavily on the Forge-hosted projects packaging automation proposal since the Go ecosystem uses almost exclusively modern software publishing services.

The proposal has been tested in Rawhide and EL7 over a set of ~ 140 Go packages. This set is a mix of current Fedora packages, bumped to a more recent version, rewrites of Fedora packages, and completely new packages.

Links

• FPC ticket
• go-srpm-macros RFE with the necessary technical files
• packaging@lists.fedoraproject.org discussion
• COPR repo
• Forge-hosted projects packaging automation draft this proposal depends on

Benefits and limitations

Benefits

• drastically shorter spec files, up to 90% in some cases, often removing hundreds of lines per spec.
• simple, packager-friendly spec syntax
• automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming.
• handling of import path changes, through trivial compatibility package creation and application of the official renaming process.
• working Go autoprovides. No forgotten provides anymore.
• working Go autorequires. No forgotten requires anymore.
• strict automated directory ownership (used by autorequires and autoprovides).
• centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages lacking guidelines. No more broken guidelines no one notices.
• easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing.
• guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014.
• guidelines-compliant bootstraping.
• systematic use of the Go switches defined by the Go maintainer. Easy to do changes followed by a mass rebuild.
• flexibility, do the right thing transparently by default, leave room for special cases and overrides.
• no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency.
• centralized Go macros that can be audited and enhanced over time.
• aggressive leverage of upstream unit tests to detect quickly broken code.
• no reliance on external utilities to compute code requirements. No more dependencies that do not match the shipped Go code.
• no maze of variable indirections. No more packages everyone is afraid of touching.
• no maze of cargo-culted and bitrotting shell code. No more packages everyone is afraid of touching.
• compatibility with existing packages (though many are so obsolete they need complete replacement)

Limitations

• very granular requires/provides, due to the lack of a common packaging format for Go projects.
• no automated version constrains on requires, due to the lack of a common packaging format for Go projects.
• can not choose the correct commit for the packager, due to the lack of release discipline in many Go projects. Need periodic bumping of all Go packages, followed by a mass rebuild, to avoid getting stuck in the past.
• does not eliminate dependency loops, caused by the lack of release discipline in many Go projects. Use bootstraping. Mass rebuilds need two stages.
• does not build shared libraries, due to their lack of adoption by most Go projects. Updating a Go component requires the rebuild of all its users. However, this project facilitates the creation of a coherent baseline of Go code, that can be converted to shared libraries later.
• gives up on many conventions of current Fedora Go packaging, as they were an obstacle to the target ease of packaging.

Testing the proposal

Technical files

The files are proposed for inclusion in go-srpm-macros. You also need macros-forge.srpm as proposed for inclusion in redhat-rpm-config.

In rpmbuild and spectool

Drop the files in the following locations, or rebuild go-srpm-macros as proposed in the RFE.

• macros-* files: in /usr/lib/rpm/macros.d/
• go.attr: in /usr/lib/rpm/fileattrs/
• go.prov and go.req: in /usr/lib/rpm/

If your version of redhat-rpm-config does not include macros-forge.srpm you can add it to go-srpm-macros.

Alternatively, grab it from COPR.

In mock

You need to add the files to the go-srpm-macros package and make it available in a repository mock can access.

In EL7

You need to add the following file:

• macros.golang-compiler: provided by go-compilers-golang-compiler in Fedora devel.

You need to add go-srpm-macros to the default mock chroot:

 config_opts['chroot_setup_cmd'] = 'install @buildsys-build go-srpm-macros'

Activating autodeps on old-style Go packages

To activate automated Provides and Requires on old-style Go packages you need to change the following lines in go.attr:

%__go_provides          %{_rpmconfigdir}/go.prov --goipath "%{?goipath}%{!?goipath:%{?import_path}}" --prefix "%{buildroot}" --gopath "%{gopath}" --version "
%__go_requires          %{_rpmconfigdir}/go.req  --goipath "%{?goipath}%{!?goipath:%{?import_path}}" --prefix "%{buildroot}" --gopath "%{gopath}" --version "

Those packages have never been tested for automated dependencies, the result may not be what you wish (for example, if the package installs example code with broken constrains).

Naming

Source packages (src.rpm)

• Packages dedicated to the furniture of Go code to other projects, with eventually some ancillary utilities, MUST use a Go-specific name derived from the upstream Go package import path. This name is automatically computed in %{goname} by %gometa.
• Packages that provide an application such as etcd MUST be named after the application. End users do not care about the language their applications are written in.
• Packages that provide connector code in multiple programming languages SHOULD also be named in some neutral non Go-specific way.

Go code packages: %{goname}-devel

In a dedicated source package

Packages that ship Go code in %gopath should be named %{goname}-devel. If your source package is already named %{goname}, that is easily achieved with:

%package devel
…
%description devel
…
%files devel

In a generic source package

If your source package is named something else, you can use:

%package -n %{goname}-devel
…
%description -n %{goname}-devel
…
%files -n %{goname}-devel

Separate code packages

And, finally, if you wish to ventilate the project Go code in multiple packages, you can compute the corresponding names with:

%global goname1 %gorpmname importpath1
…
%package -n %{goname1}-devel
…
%description -n %{goname1}-devel
…
%files -n %{goname1}-devel

Do remember that for Go each directory is a package. Never separate the .go files contained in a single directory in different packages (unit tests excepted).

Implementation: %gorpmname

%gometa uses the %gorpmname macro to compute the main %{goname} from %{goipath}.

%gorpmname can produce collisions
%gorpmname tries to compute human-friendly and rpm-compatible naming from Go import paths. It simplifies them, removes redundancies and common qualifiers. As a result it is possible for two different import paths to produce the same result. In that case, feel free to adjust this result manually to avoid the collision. And please report the case.

Compatibility Go code packages: %{oldgoname}-devel-compat

When a project changes its import path, it is possible to generate a temporary compatibility package to keep old code working during the transition. This package SHOULD be named %{oldgoname}-devel-compat, with %{oldgoname} generated the following way:

%global oldgoipath github.com/Sirupsen/logrus
%global oldgoname  %gorpmname %{oldgoipath}

Compatibility package creation is detailed in the Handling renamings chapter.

Go utilities: xxx-utils

Ancillary Go utilities (not full applications) should be shipped in xxx-utils packages. The rules are the same as for %{goname}-devel packages. The xxx prefix should be %{goname} or a simplified project identifier.

Go example code: %doc

Example code is usually shipped as %doc in the corresponding %{goname}-devel package. You can also produce a separate -devel package dedicated to the example import path.

Walkthrough

This chapter will present a typical Go spec file step by step, with comments and explanations.

Spec preamble: %{goipath}, %{forgeurl} and %gometa

A Go package is identified by its import path. A Go spec file will therefore start with the %{goipath} declaration. Don't get it wrong, it will control the behaviour of the rest of the spec file.

 %global goipath     google.golang.org/api

If you’re lucky the Internet hosting of the Go package can be automatically deduced from this variable (typically by prefixing it with https://). If that is not the case, you need to declare explicitly the hosting URL:

 %global forgeurl    https://code.googlesource.com/google-api-go-client/

If rpmbuild complains later your hosting service is unknown of %forgemeta, please extend this macro.

The %{forgeurl} declaration is followed by Version, %{commit} and %{tag}. Use the combination that matches your use-case. The rules are the same as in Forge-hosted packaging.

%global commit      3a1d936b7575b82197a1fea0632218dd07b1e65c

Commits vs releases
You SHOULD package releases in priority. Please reward the projects that make an effort to identify stable code states. Only fall back to commits when the project does not release, when the release is more than six months old, or if you absolutely need one of the changes of a later commit. In the later cases please inform the project politely of the reason you needed to give up on their official releases. Promoting releases is a way to limit incompatible commit hell.

The code versioning information is followed by a clear project description that will be reused in the various rpm packages produced from this spec file.

 %global common_description %{expand:
 A human-friendly multi-line project description.}

Then you need to call the %gometa macro to put it all together

 %gometa

Its behavior is similar to %forgemeta, with some Go-specific enhancements. This macro will attempt to compute and set the following variables if they are not already set by the packager:

• goname: an rpm-compatible package name derived from %{goipath}
• gosource: an URL that can be used as SourceX: value
• gourl: an URL that can be used as URL: value

… and via its use of %forgemeta:

• forgesource: an URL that can be used as SourceX: value
• forgesetupargs: the correct arguments to pass to %setup for this source; used by %forgesetup and %forgeautosetup
• archivename: the source archive filename, without extentions
• archiveext: the source archive filename extensions, without leading dot
• archiveurl: the url that can be used to download the source archive, without renaming
• scm: the scm type, when packaging code snapshots (commits or tags)

If the macro is unable to parse the %{forgeurl} value the packager should set at least %{archivename} and %{archiveurl} before calling it.

The macro also accepts the following optional parameters:

• -u <url>: ignore %{forgeurl} even if it exists and use <url> instead; note that the macro will still end up setting <url> as %{forgeurl} if it manages to parse it
• -s: silently ignore problems in %{forgeurl}, use it if it can be parsed, ignore it otherwise
• -v : be verbose and print every variable the macro sets
• -i: Print some info about the state of variables the macro may use or set at the end of the processing

Source package metadata: %{goname}, %{gourl} and %{gosource}

Then you can declare the usual rpm headers, using the values computed by %gometa.

Name:    %{goname}     See also Naming
Version: 0             If zero, because the project does not release. Otherwise it should have been declared before calling %gometa.
Release: 0.X%{?dist}   %gometa uses %forgemeta to compute the correct %{dist} value for snapshots.
Summary: Supplementary Google APIs Client Library for Go
License: BSD           See also Fedora licensing guidelines.
URL:     %{gourl}
Source:  %{gosource}

“%{gourl}“ or “%{forgeurl}“ use as “URL:”
You do not have to use %{gourl} or %{forgeurl} as URL: values if the packaged project has a better customized home page. They are a convenience, nothing more.

The headers are followed by the BuildRequires of the project unit tests and binaries. They are usually identified by trying to build an rpm from the spec file and noting compilation errors.

BuildRequires: golang(golang.org/x/text/secure/bidirule)
BuildRequires: golang(golang.org/x/net/context)
BuildRequires: golang(golang.org/x/oauth2)
BuildRequires: golang(golang.org/x/sync/semaphore)
BuildRequires: golang(github.com/google/go-cmp/cmp)
BuildRequires: golang(google.golang.org/genproto/googleapis/bytestream)

%{goname}-devel package metadata

If you’re producing a Go code package, the following should be sufficient:

 %package devel
 Summary: %{summary}
 BuildArch: noarch
 
 Obsoletes: golang-google-golang-api-devel < 0.100
 If the corresponding import path was provided by another package in the past. See also the corresponding guidelines.
 Replacing Go -devel packages does not require providing the old package name because they are accessed via golang() dependencies.

 %description devel
 %{common_description}
 
 This package contains the source code needed for building packages that import
 the %{goipath} Go namespace.

%{goname}-utils package metadata

If you are producing a utilities package the declaration is similar:

%description utils
%{common_description}
 
This package contains the command-line utilites provided by the
%{goipath} Go namespace.

%prep: %forgesetup

Assuming you followed the preamble instructions, preparation is reduced to:

%prep
%forgesetup

Followed by eventual patching the usual rpm way.

Vendoring
You MUST remove the bundled code eventually shipped by upstream in the vendor directories.

rm -fr vendor

%build: %gobuildroot and %gobuild

If you need to build some Go binaries, use the following pattern:

%gobuildroot
%gobuild -o _bin/something %{goipath}/cmd/something

%gobuildroot set ups the build environment and creates _bin. Most Go projects ship commands in a cmd subdirectory.

%install: %{gofindfilter} and %goinstall

The installation phase is reduced to:

 %install
 gofiles=$(find . %{gofindfilter} -print)
 %goinstall $gofiles

You can add more flags after %{gofindfilter}, use your own filter, apply the same pattern to a specific subdirectory.

%goinstall will install Go files in the correct place with default permissions and generate the corresponding devel.file-list. If you wish to generate a separate list (for example when producing separate code packages), just pass the -f flag to the macro:

 gosubfiles=$(find subdir %{gofindfilter} -print)
 %goinstall -f subdir.file-list $gosubfiles

If you build some binaries in %build you also need to deploy them:

install -m 0755 -vd  %{buildroot}%{_bindir}
install -m 0755 -vp _bin/* %{buildroot}%{_bindir}/

%check: %gochecks and %{gotest}

%check
# No test files in integration-tests/storage
# . transport/http, transport/grpc, transport, option, internal --> undefined: google.DefaultCredentials
%gochecks . transport/http transport/grpc transport option internal integration-tests/storage examples

The %gochecks macro will walk through all the project sub-directories containing .go files and try to execute %{gotest} there.

%gochecks

It is an opinionated and aggressive behavior designed to detect problems as soon as possible. Due to the pervasive use of rapid-changing commits in the Go ecosystem with little release engineering running every possible unit test is a MUST.

You can disable testing in a specific sub-directory by passing it as argument to %gocheck

%gochecks subdir

To disable testing in the root directory use .

 %gochecks .

You can also use wildcards

 %gochecks '_examples/*'

Common reasons to exclude a directory from testing:

• the directory does not actually contain any Go code,
• this is an example directory we do not care about,
• the unit tests contained in the directory want to access the network,
• the unit tests depend on a specific server running in the build environment,
• the unit tests depend on another Go package which is not packaged yet (please package it!),
• the Go compiler is crashing (please report the Fedora bug!)
• upstream confirms the tests should not be ran,
• there is some other problem you’re currently investigating with upstream

Remember to add the dependencies needed by the unit tests as BuildRequires in source package metadata. They should be of the following form:

 BuildRequires: golang(missing_import_path_Go_complains_about)

Be very careful to note down why you are disabling a particular unit test, with the eventual bug report URL. Do try to enable it again later if the reason is not definitive.

Dependency loops are not a good reason to disable unit tests definitely.

%files

The %files section of a %{goname}-devel package is reduced to reading the file list produced by %goinstall, and adding documentation and licensing files

%files devel -f devel.file-list
%license LICENSE
%doc *\.md AUTHORS CONTRIBUTORS NOTES TODO examples/

The %files section of a %{goname}-utils package will be as succinct:

%files -n utils
%license LICENSE
%doc cmd/foo/README.md
%{_bindir}/*

Handling dependency loops

Dependency loops are quite frequent in Go at unit test level. The correct way to handle them is to apply bootstrapping guidelines to disable one of the tests involved in the loop, with the corresponding BuildRequires:

%{?_with_bootstrap: %global bootstrap 1}
…
# foo.net/something is causing a loop with golang-foo-something
if ! 0%{?bootstrap}
  BuildRequires: golang(foo.net/something)
%endif
…
%check
%if ! 0%{?bootstrap}
  %gochecks
%else
  %gochecks foo-something
%endif

Handling renamings

Go packages MUST always be named after their current import path. An upstream renaming triggers the package renaming process if the source package is named %{goname} and %gorpmname computes a different %{goname} from the new %{goipath}. In that case the usual obsoletion rules SHOULD be kept in the main %{goname}-devel package.

In any case, you MAY generate an %{oldgoname}-devel-compat package during the import path change transition period, with the following pattern.

Symlinks do not fix code
It is very dangerous to keep this kind of workaround for a long period. Code that still calls your project by its old name is not adjusted to new changes, and will eventually fail in mysterious ways. Compatibility packages are transitory and MUST be removed after two Fedora releases.

The following only works from within an updated package. Automated dependency computation requires access to the new directory structure during the build process.

%global goipath     github.com/sirupsen/logrus
…
# Compatibility glue
%global oldgoipath github.com/Sirupsen/logrus
%global oldgoname  %gorpmname %{oldgoipath}
…
%package -n %{oldgoname}-devel-compat
Summary:   %{summary}
BuildArch: noarch

%description -n %{oldgoname}-devel-compat
%{common_description}

This package contains compatibility glue for code that still imports the
%{oldgoipath} Go namespace.
…
%install
…
%goinstall $gofiles
…
install -m 0755 -vd %{buildroot}%{gopath}/src/github.com/Sirupsen
ln -ns %{gopath}/src/%{goipath} %{buildroot}%{gopath}/src/%{oldgoipath}
Or alternatively
ln -s ../sirupsen/logrus %{buildroot}%{gopath}/src/%{oldgoipath}
…
%files -n %{oldgoname}-devel-compat
%dir %{gopath}/src/github.com/Sirupsen
%{gopath}/src/%{oldgoipath}

This is sufficient to generate a working compatibility package:

rpm -qp --provides golang-github-sirupsen-logrus-devel-compat-1.0.3-11.fc28.noarch.rpm
golang(github.com/Sirupsen/logrus) = 1.0.3-11.fc28
golang(github.com/Sirupsen/logrus/hooks) = 1.0.3-11.fc28
golang(github.com/Sirupsen/logrus/hooks/syslog) = 1.0.3-11.fc28
golang(github.com/Sirupsen/logrus/hooks/test) = 1.0.3-11.fc28
golang-github-sirupsen-logrus-devel-compat = 1.0.3-11.fc28

$ rpm -qp --requires golang-github-sirupsen-logrus-devel-compat-1.0.3-11.fc28.noarch.rpm
golang(github.com/sirupsen/logrus) = 1.0.3-11.fc28
golang(github.com/sirupsen/logrus/hooks) = 1.0.3-11.fc28
golang(github.com/sirupsen/logrus/hooks/syslog) = 1.0.3-11.fc28
golang(github.com/sirupsen/logrus/hooks/test) = 1.0.3-11.fc28
…

Note that in this example, creating the compatibility symbolic link required the creation of an intermediate directory, that was assigned to the compatibility package:

install -m 0755 -vd %{buildroot}%{gopath}/src/github.com/Sirupsen
ln -ns %{gopath}/src/%{goipath} %{buildroot}%{gopath}/src/%{oldgoipath}
…
%files -n %{oldgoname}-devel-compat
%dir %{gopath}/src/github.com/Sirupsen
%{gopath}/src/%{oldgoipath}

Putting it all together

%{?_with_bootstrap: %global bootstrap 1}
%global goipath     github.com/performancecopilot/speed
Version:            2.0.0
%global project     performancecopilot-speed
 
%global common_description %{expand:
A Go implementation of the Performance Co-Pilot (PCP) instrumentation.}
 
%gometa
 
Name:    %{goname}
Release: 10%{?dist}
Summary: Supplementary Go libraries implementing PCP instrumentation
License: MIT
URL:     %{gourl}
Source:  %{gosource}
 
BuildRequires: golang(github.com/codahale/hdrhistogram)
BuildRequires: golang(github.com/edsrzf/mmap-go)
# break go.uber.org/zap →
#       github.com/go-kit/kit/log →
#       github.com/performancecopilot/speed
%if ! 0%{?bootstrap}
  BuildRequires: golang(go.uber.org/zap)
%endif
 
%description
%{common_description}
 
%package   devel
Summary:   %{summary}
BuildArch: noarch
 
%description devel
%{common_description}
 
This package contains the source code needed for building packages that import
the %{goipath} Go namespace.
 
%package -n %{project}-utils
Summary:   %{summary}, command-line utilities
 
%description -n %{project}-utils
%{common_description}
 
This package contains the command-line utilites provided by the
%{goipath} Go namespace.

%prep
%forgesetup
rm -fr vendor
 
%build
%gobuildroot
%gobuild -o _bin/mmvdump %{goipath}/mmvdump/cmd/mmvdump
 
%install
gofiles=$(find . %{gofindfilter} -print)
%goinstall $gofiles
 
install -m 0755 -vd        %{buildroot}%{_bindir}
install -m 0755 -vp _bin/* %{buildroot}%{_bindir}/
 
%check
%if ! 0%{?bootstrap}
  %gochecks 'example*'
%else
  %gochecks . 'example*'
%endif
 
%files devel -f devel.file-list
%license LICENSE
%doc *\.md examples
 
%files -n %{project}-utils
%license LICENSE
%{_bindir}/*
 
%changelog
…