From Fedora Project Wiki

Revision as of 10:52, 12 February 2018 by Nim (talk | contribs)

This is an enhancement proposal to PackagingDrafts/Go. It builds on the hard work of the Go SIG and reuses the rpm automation of PackagingDrafts/Go when it exists, and produces compatible packages.

The proposal builds the Go integration inside rpm, and factors out common tasks. It tries to limit spec content to items where the packager adds actual value, and makes it easy to adapt to upstream changes. It uses rpm facilities to auto-compute Requires and Provides. Note that Go has not standardized on a common component tool yet, therefore the auto-produced dependencies are highly granular and lacking versioning constrains.

This proposal achieves a drastic reduction of Go spec sizes, up to 90% in some cases, not counting changelog lines. It often removes hundreds of high-maintenance lines per spec file. It increases quality by enabling stricter checks in the factored-out code, without relying on packagers to cut and paste the correct snippets in their spec files. It aggressively runs all the unit tests found in upstream projects. It does not try to rely on bundled Go code, to hide upstream renamings, to avoid rebasing packages when their dependencies change.

This proposal relies heavily on the Forge-hosted projects packaging automation proposal since the Go ecosystem uses almost exclusively modern software publishing services.

The proposal has been tested in Rawhide and EL7 over a set of ~ 450 Go spec files, both rewrites of Fedora packages, and completely new packages. This set is not completely self hosting and integrates with existing Fedora Go packages.

Links

Benefits and limitations

Benefits

  • drastically shorter spec files, often removing hundreds of lines per spec, up to 90% in some cases.
  • simple, packager-friendly spec syntax
  • automated package naming derived from the native identifier (import path). No more packages names without any relation with current upstream naming.
  • handling of import path changes, through trivial compatibility package creation and application of the official renaming process.
  • working Go autoprovides. No forgotten provides anymore.
  • working Go autorequires. No forgotten requires anymore.
  • strict automated directory ownership (used by autorequires and autoprovides).
  • centralized computation of source URLs (via Forge-hosted projects packaging automation). No more packages with partial guidelines conformance. No more broken guidelines no one notices. No more GitHub-specific patterns that can not be used for other hosting systems.
  • easy switch between commits, tags and releases (via Forge-hosted projects packaging automation). No more packages stuck on commits when upstream starts releasing.
  • guidelines-compliant automated snapshot naming, including snapshot timestamps (via Forge-hosted projects packaging automation). No more packages stuck in 2014 no one notices. No more mysterious failures due to those obsolete packages.
  • guidelines-compliant bootstraping.
  • systematic use of the Go switches defined by the Go maintainer. Easy to do changes followed by a mass rebuild.
  • flexibility, do the right thing transparently by default, leave room for special cases and overrides.
  • no bundling (a.k.a. vendoring) due to the pain of packaging one more Go dependency.
  • centralized Go macros that can be audited and enhanced over time.
  • aggressive leverage of upstream unit tests to detect quickly broken code.
  • no reliance on external utilities to compute code requirements. No more dependencies that do not match the shipped Go code.
  • no maze of variable indirections. No more packages everyone is afraid of touching.
  • no maze of cargo-culted and bitrotting shell code. No more packages everyone is afraid of touching.
  • compatibility with existing packages (though many are so obsolete they need complete replacement)

Limitations

  • very granular requires/provides, due to the lack of a common packaging format for Go projects.
  • no automated version constrains on requires, due to the lack of a common packaging format for Go projects.
  • no BuildRequires automation, they would need the following rpm and mock RFEs.
  • can not choose the correct commit for the packager, due to the lack of release discipline in many Go projects. Need periodic bumping of all Go packages, followed by a mass rebuild, to avoid getting stuck in the past. See also Go developer guidance.
  • does not eliminate dependency loops, caused by the lack of release discipline in many Go projects. Use bootstraping. Mass rebuilds need five stages: everything not needing bootstrap, then everything needing bootstrap (in bootstrap then full mode), then everything again to make sure nothing is built against a now removed API. See also Go developer guidance.
  • does not build shared libraries, due to their lack of adoption by most Go projects. Updating a Go component requires the rebuild of all its users. However, this project facilitates the creation of a coherent baseline of Go code, that can be converted to shared libraries later.
  • gives up on many conventions of current Fedora Go packaging, as they were an obstacle to the target packaging efficiency.

Testing the proposal

Technical files

The files are proposed for inclusion in go-srpm-macros. You also need macros-forge which is included in recent versions of redhat-rpm-config.

In rpmbuild and spectool

Drop the files in the following locations, or rebuild go-srpm-macros as proposed in the RFE.

  • macros-* files: in /usr/lib/rpm/macros.d/
  • go.attr: in /usr/lib/rpm/fileattrs/
  • go.prov and go.req: in /usr/lib/rpm/

If your version of redhat-rpm-config does not include macros-forge you can add it to go-srpm-macros.

Alternatively, grab it from COPR.

In mock

You need to add the files to the go-srpm-macros package and make it available in a repository mock can access.

In EL7

You need to add the following file:

  • macros.golang-compiler: provided by go-compilers-golang-compiler in Fedora devel.

You need to add go-srpm-macros to the default mock chroot: 

 config_opts['chroot_setup_cmd'] = 'install @buildsys-build go-srpm-macros'

Naming

Source packages (src.rpm): application name, otherwise %{goname}

  • Source packages dedicated to the furniture of Go code to other projects, with eventually some ancillary utilities, MUST use a Go-specific name derived from the upstream Go package import path. This name is automatically computed in %{goname} by %gometa.
  • Source packages that provide a well-known application such as etcd MUST be named after the application. End users do not care about the language their applications are written in. But do not name packages after an obscure utility binary that happens to be built by the package!
  • Source packages that provide connector code in multiple programming languages SHOULD also be named in some neutral non Go-specific way.

Go code packages: %{goname}-devel

In a source package dedicated to the furniture of Go code

Packages that ship Go code in %gopath should be named %{goname}-devel. If your source package is already named %{goname}, that is easily achieved with: 

%package devel
[…]
%description devel
[…]
%files devel -f devel.file-list

In a another kind of source package

If your source package is named something other than %{goname}, you SHOULD use: 

%package -n %{goname}-devel
[…]
%description -n %{goname}-devel
[…]
%files -n %{goname}-devel -f devel.file-list

Separate code packages

And, finally, if you wish to ventilate the project Go code in multiple packages, you can compute the corresponding names with: 

%global goname1 %gorpmname importpath1
[…]
%package -n %{goname1}-devel
[…]
%description -n %{goname1}-devel
[…]
%files -n %{goname1}-devel -f %{goname1}.file-list

See also the Generating multiple Go code packages chapter.

Do remember that for Go each directory is a package. Never separate the .go files contained in a single directory in different packages (unit tests excepted).

Implementation: %gorpmname

%gometa uses the %gorpmname macro to compute the main %{goname} from %{goipath}.

%gorpmname can produce collisions
%gorpmname tries to compute human-friendly and rpm-compatible naming from Go import paths. It simplifies them, removes redundancies and common qualifiers. As a result it is possible for two different import paths to produce the same result. In that case, feel free to adjust this result manually to avoid the collision. And please report the case.

Import path compatibility packages: compat-%{oldgoname}-devel

When a project can be referenced under multiple import paths, due to forks, renamings, rehostings, organizational changes, or confusing project documentation, it is possible to generate compatibility sub-packages to connect code that uses one of the other import paths to the canonical one.

The canonical import path SHOULD always be the one referenced in the project documentation. However some projects do not document import path changes, and rely on HTTPS redirections (for example https://github.com/docker/docker → https://github.com/moby/moby). Such a redirection is a sufficient indicator the canonical import path has changed (but please make sure with upstream).

Never defer renamings
Packagers MUST NOT use import path compatibility sub-packages to alias the canonical import path to one of the previous namings. Packagers MUST apply upstream renaming choices to the main %{goipath} spec variable and everything that derives from it, such as %{goname}. Deferred renamings introduce friction with upstream and other packagers.

Import path compatibility packages SHOULD be named %{oldgoname}-devel-compat, with %{oldgoname} generated the following way: 

%global oldgoipath github.com/Sirupsen/logrus
%global oldgoname  %gorpmname %{oldgoipath}

Use oldgoipath1, oldgoipath2… and oldgoname1, oldgoname2… as needed.

Import path compatibility package creation is detailed in the Handling renamings chapter.

Go code compatibility packages: compat-%{goname}-<id> and compat-%{goname}-<id>-devel

Go projects can change their API. Packaging this change will break projects that rely on the old API.

It is not always possible to patch or upgrade the code of the broken packages, to a state compatible with the new API. This situation, when it occurs, requires the packaging of the old API, in a Go code compatibility package. The naming of such packages differ from non-compatibility naming the following ways:

  • for all names derived from %{goname}: replace %{goname} with compat-%{goname}-<id>
  • if the source package %{name} is not derived from %{goname}: use compat-<usualname>-<id>
  • for all other subpackages which naming is not derived from %{goname} or %{name}: prefix with compat-, add <id> at the most appropriate place.

<usualname> is whatever you'd use to name the source package if it was not a compatibility package.

<id> is:

  • %{version} if it matches a release,
  • %{tag} if it matches a tagged state (eventually massaged to be compatible with rpm naming constrains), and
  • %{shortcommit} otherwise, with %{shortcommit} computed the following way:
%global shortcommit  %{lua:print(string.sub(rpm.expand("%{?commit}"), 1, 7))}

Go code compatibility package creation is detailed in the Handling API changes chapter.

Go example code: %doc

Example code is usually shipped as %doc in the corresponding %{goname}-devel package. You MAY also produce a separate -devel package dedicated to the example import path.

Walkthrough

This chapter will present a typical Go spec file step by step, with comments and explanations.

Spec preamble: %{goipath}, %{forgeurl} and %gometa

Usual case

A Go package is identified by its import path. A Go spec file will therefore start with the %{goipath} declaration. Don't get it wrong, it will control the behaviour of the rest of the spec file. 

 %global goipath     google.golang.org/api

If you’re lucky the Internet hosting of the Go package can be automatically deduced from this variable (typically by prefixing it with https://). If that is not the case, you need to declare explicitly the hosting URL: 

 %global forgeurl    https://code.googlesource.com/google-api-go-client/

The %{forgeurl} declaration is followed by Version, %{commit} and %{tag}. Use the combination that matches your use-case. The rules are the same as in Forge-hosted packaging. 

%global commit      3a1d936b7575b82197a1fea0632218dd07b1e65c
Commits vs releases
You SHOULD package releases in priority. Please reward the projects that make an effort to identify stable code states. Only fall back to commits when the project does not release, when the release is more than six months old, or if you absolutely need one of the changes of a later commit. In the later cases please inform the project politely of the reason you needed to give up on their official releases. Promoting releases is a way to limit incompatible commit hell.

The code versioning information is followed by a clear project description that will be reused in the various rpm packages produced from this spec file. 

 %global common_description %{expand:
 A human-friendly multi-line project description.}

Then you need to call the %gometa macro to put it all together: 

 %gometa

Its behavior is similar to %forgemeta, with some Go-specific enhancements. This macro will attempt to compute and set the following variables if they are not already set by the packager:

  • goname: an rpm-compatible package name derived from %{goipath}
  • gosource: an URL that can be used as SourceX: value
  • gourl: an URL that can be used as URL: value

… and via its use of %forgemeta:

  • forgesource: an URL that can be used as SourceX: value
  • forgesetupargs: the correct arguments to pass to %setup for this source; used by %forgesetup and %forgeautosetup
  • archivename: the source archive filename, without extentions
  • archiveext: the source archive filename extensions, without leading dot
  • archiveurl: the url that can be used to download the source archive, without renaming
  • scm: the scm type, when packaging code snapshots (commits or tags)

When Fedora has no knowledge of the hosting service

If the hosting service is unknown of Fedora or does not permit automation, %gometa will be unable to completely process %{goipath}, and the eventual %{forgeurl}). In that case it is possible to set the non or mis-computed variables manually before calling the macro. This way the packager will still benefit from the rest of the automation.

To determine the variables that need manual presetting, call %gometa with at least -i, and optionally -v or -p switches. The minimal set is usually %{archivename} and %{archiveurl}, but it may take more if the hosting service diverges a lot from usual conventions.

%gometa accepts the following optional parameters:

  • -u <url>: ignore %{forgeurl} even if it exists and use <url> instead; note that the macro will still end up setting <url> as %{forgeurl} if it manages to parse it
  • -s: silently ignore problems in %{forgeurl}, use it if it can be parsed, ignore it otherwise
  • -p: override -s, ignore fallbacks, do error reporting about problems. This is especially useful since%gometa calls forgemeta in silent mode by default.
  • -v : be verbose and print every variable the macro sets
  • -i: Print some info about the state of variables the macro may use or set at the end of the processing

Do consider extending %forgemeta if you think the problem hosting service will be of use in other spec files.

Source package metadata: %{goname}, %{gourl} and %{gosource}

Then you can declare the usual rpm headers, using the values computed by %gometa. 

Name:    %{goname}     Do read Naming
Version: 0             If zero, because the project does not release. Otherwise it should have been declared before calling %gometa.
Release: 0.X%{?dist}   %gometa uses %forgemeta to compute the correct %{dist} value for snapshots.
Summary: Supplementary Google APIs Client Library for Go
License: BSD           See also Fedora licensing guidelines.
URL:     %{gourl}
Source:  %{gosource}
“%{gourl}“ or “%{forgeurl}“ use as “URL:”
You do not have to use %{gourl} or %{forgeurl} as URL: values if the packaged project has a better customized home page. They are a convenience, nothing more.

The headers are followed by the BuildRequires of the project unit tests and binaries. They are usually identified by trying to build an rpm from the spec file and noting compilation errors. 

BuildRequires: golang(golang.org/x/text/secure/bidirule)
BuildRequires: golang(golang.org/x/net/context)
BuildRequires: golang(golang.org/x/oauth2)
BuildRequires: golang(golang.org/x/sync/semaphore)
BuildRequires: golang(github.com/google/go-cmp/cmp)
BuildRequires: golang(google.golang.org/genproto/googleapis/bytestream)

%{goname}-devel package metadata

Usual case

If you’re producing a Go code package, the following should be sufficient: 

 %package devel
 Summary: %{summary}
 BuildArch: noarch
 
 %description devel
 %{common_description}
 
 This package contains the source code needed for building packages that import
 the %{goipath} Go namespace.

When building binaries

If the produced package builds any binary, the subpackages used to ship those SHOULD be required from the devel subpackage that ships the corresponding code. For a simple Go package that ships its binaries in its main package that means changing the above to: 

 %package devel
 Summary: %{summary}
 BuildArch: noarch
 
 Requires: %{name} = %{version}-%{release}

 %description devel
 %{common_description}
 
 This package contains the source code needed for building packages that import
 the %{goipath} Go namespace.

When replacing another Go code package

If the corresponding import path was provided by another package in the past, and you chose to avoid creating an import path compatibility package, the devel package should also include obsoletion rules. For example, and assuming the versioning of the new package is consistent with the versioning of its ancestor: 

 %package devel
 Summary: %{summary}
 BuildArch: noarch
 
 Obsoletes: ancestor-package-devel < %{version}-%{release}

 %description devel
 %{common_description}
 
 This package contains the source code needed for building packages that import
 the %{goipath} Go namespace.

See also the section on handling renamings.

%prep: %forgesetup, removing vendoring

Assuming you followed the preamble instructions, preparation is reduced to: 

%prep
%forgesetup

Followed by eventual patching the usual rpm way.

Vendoring
You MUST remove the bundled code eventually shipped by upstream in the vendor directories.
rm -fr vendor

%build: %gobuildroot and %gobuild

Build everything that can be built
Packagers SHOULD build everything a project allows to build even if they have no use for the produced binaries. Building projects is an effective way to check if the Go packages in Fedora are compatible with each other. Do not wait till the code is imported with hundreds of others in another project, it will be a lot harder to debug.

However, when creating a new Go package, it may be more efficient to start by making unit tests work, before attempting to build project binaries. Unit tests can be a lot more granular, focused, and easier to interpret, than a general build failure.

General principle

Building Go binaries takes some form of the following commands: 

%build
%gobuildroot
%gobuild -o _bin/binary_name binary_import_path

%gobuildroot set up the build environment and creates the _bin directory.

Do not use custom commands or third-party Makefiles as they won't apply the distribution-wide settings defined by the maintainers of the Fedora Go compiler.

Go build failures are not immediate
Contrary to what happens in many other languages, Go build problems do not cause an immediate abort. The error that actually caused a build failure, may be reported hundreds of lines, before the final abortion.

Usual case

Many Go projects are nice enough to put the code corresponding to binaries in a cmd subdirectory. Building them only requires the simple and generic: 

%build
%gobuildroot
for cmd in cmd/* ; do
  %gobuild -o _bin/$(basename $cmd) %{goipath}/$cmd
done

Exceptions

However, many Go conventions are informal and not applied in a systematic way. Other projects may not apply this convention, use %{goipath} as a single project-wide binary location, use custom naming such as plugins instead of cmd, diverge in other ways.

Careful reading of the project documentation and custom build recipes, or even asking directly upstream, is necessary to identify what can be built in those cases.

%install: %{gofindfilter} and %goinstall

Usual case

The installation phase is reduced to: 

%install
gofiles=$(find . %{gofindfilter} -print)
%goinstall $gofiles

%goinstall will install Go files in the correct place with default permissions and generate the corresponding devel.file-list.

You can add more flags after %{gofindfilter}, use your own filter, apply the same pattern to a specific subdirectory.

Common pitfalls; making %goinstall process more or less

Identify any form of example, testing or integration directory and strip it from the installation list. Example and testing code frequently contains imports, that will complexify the package Go dependencies, as computed by rpm. If they can not be satisfied, they will even render the package completely uninstallable.

For example, if the packaged project includes a non-standard tutorial example directory, you should filter it with: 

%install
gofiles=$(find lib/go %{gofindfilter} \! -ipath '*/tutorial/*'  -print)
%goinstall $gofiles

Filter those files, instead of deleting them. You will need them in %check or as %doc.

Also, make sure the project code does not need files with unusual extensions, that are ignored by the default find invocation. Such files need to be passed as parameters to %goinstall if you want a result usable by other packages. 

%install
gofiles=$(find . %{gofindfilter} -print)
%goinstall $gofiles special/*.foo

Generating multiple Go code packages

If you wish to generate a separate list (for example when producing separate code packages), just pass the -f flag to the macro: 

gosubfiles=$(find subdir1 %{gofindfilter} -print)
%goinstall -f %{goname1}.file-list $gosubfiles

In this example %{goname1} is the name produced from the %{goipath}/subdir1 import path.

The Go language does not make any provision for splitting the files located in a single directory, among several packages. Therefore, golang(…) provides must not be duplicated for the system to work. When using %goinstall multiple times, the ownership of each directory will be attributed to the first invocation, that requires creating this directory. This ownership will then be used to determine which subpackage, should be tagged with the associated golang(…) provides.

Invocation order matters
As a consequence, splitting code among multiple subpackages, requires careful thought, and testing, of %goinstall ordering. Packagers SHOULD NOT try to correct non-working ordering, by forcing dependencies or directory ownership manually.

Installing binaries

If you build some binaries in %build you also need to deploy them: 

install -m 0755 -vd        %{buildroot}%{_bindir}
install -m 0755 -vp _bin/* %{buildroot}%{_bindir}/

Installing import path aliases

Symbolic links can be used to alias import paths used in previous versions of the project: 

install -m 0755 -vd %{buildroot}%{gopath}/src/%(dirname %{oldgoipath})
ln -s %{gopath}/src/%{goipath} %{buildroot}%{gopath}/src/%{oldgoipath}

The whole aliasing pattern is described in the Handling renamings chapter.

Collecting documentation files: %gocollectmd

Some projects pepper their subdirectories with .md files. Identifying those to tag them as documentation can be terribly inconvenient for packagers (see also the Go developer guidance chapter). The %gocollectmd convenience macro will walk the project subdirectories and copy those .\md files to the project root. 

 %gocollectmd

%check: %gochecks and %{gotest}

Usual case

The %gochecks macro will walk through all the project sub-directories containing .go files and try to execute %{gotest} there. 

%check
%gochecks

It is an opinionated and aggressive behavior designed to detect problems as soon as possible.

Excluding testing in specific directories

%gochecks permits excluding specific directories from tests. It's an opt-out macro, it only permits removing directories from testing. 

%check
# No test files in integration-tests/storage
# We do not care about example unit tests
# . transport/http, transport/grpc, transport, option, internal → undefined: google.DefaultCredentials
%gochecks . transport/http transport/grpc transport option internal integration-tests/storage examples

To exclude a specific sub-directory (not recursively) pass it as argument to %gochecks 

%gochecks subdir

To disable testing in the root directory use “.” 

 %gochecks .

To exclude a whole sub-tree (recursively) pass its root as argument to %gochecks with the -R or --root switch 

%gochecks -R subtree1 --root 'sub/something/*/tree'

To perform fine-grained exclusion using egrep-style regexes use the -r or --regex switch. The regex will usually start with ./ or .*/. 

%gochecks --regex  '.*/(.*[._-])?test(([._-])?case)?(s)?/.*' -r …

You can mix and match as many exclusions as needed. Remember, %gochecks will test every directory except for the ones excluded so invoking it multiple times with a different exclusion is useless.

When to exclude

Thou shall test
Due to the pervasive use of rapid-changing commits in the Go ecosystem with little release engineering packagers MUST try to execute as many unit tests as possible.

Common reasons to exclude a directory from testing:

  • the directory does not actually contain any Go code,
  • this is an example directory we do not care about,
  • the unit tests contained in the directory want to access the network,
  • the unit tests depend on a specific server running in the build environment,
  • the unit tests depend on another Go package which is not packaged yet (please package it!),
  • the Go compiler is crashing (please report the Fedora bug!)
  • upstream confirms the tests should not be ran,
  • there is some other problem you’re currently investigating with upstream

Remember to add the dependencies needed by the unit tests as BuildRequires in source package metadata. They should be of the following form: 

 BuildRequires: golang(missing_import_path_Go_complains_about)

Be very careful to note down why you are disabling a particular unit test, with the eventual bug report URL. Do try to enable it again later if the reason is not definitive.

Dependency loops are not a good reason to disable unit tests definitely.

%files

Usual case

The %files section of a %{goname}-devel package is reduced to reading the file list produced by %goinstall, and adding documentation and licensing files 

%files devel -f devel.file-list
%license LICENSE
%doc *\.md AUTHORS CONTRIBUTORS NOTES TODO examples/

When shipping binaries

Binaries are usually shipped in the main package, which is required from the %{goname}-devel packages that ship the associated code. This package MUST include legal files and documentation associated with those binaries You do not need to duplicate those files in %{goname}-devel packages. 

%files
%license LICENSE
%doc cmd/foo/README.md
%{_bindir}/*

Handling dependency loops

Usual case

Loops are unfortunately quite frequent in Go at unit test level. The correct way to handle them is to apply bootstrapping guidelines to disable one of the tests involved in the loop, with the corresponding BuildRequires: 

%{?_with_bootstrap: %global bootstrap 1}
…
# foo.net/something is causing a loop with golang-foo-something
if ! 0%{?bootstrap}
  BuildRequires: golang(foo.net/something)
%endif
…
%check
%if ! 0%{?bootstrap}
  %gochecks
%else
  %gochecks directory-with-tests-that-needs-foo-something
%endif

When building binaries

When the loop involves a binary built in the %build section, it needs to be excluded in a similar way:

  • to white-list binaries from bootstrapping use:
%build
%if ! 0%{?bootstrap}
  cmds=cmd/*
%else
  cmds="cmd/ovrouter cmd/proxy" White-listed binaries
%endif
  
%gobuildroot  
for cmd in $cmds ; do
  %gobuild -o _bin/$(basename $cmd) %{goipath}/$cmd
done
  • to black-list binaries from bootstrapping use:
%build
%if ! 0%{?bootstrap}
  cmds=$(ls -1 cmd)
%else
  cmds=$(ls -1 cmd | grep -v '^\(certdump\|certexpiry\|certverify\)$') Black-listed binaries
%endif
 
%gobuildroot
for cmd in $cmds ; do
   %gobuild -o _bin/$cmd %{goipath}/cmd/$cmd
done

Choosing what to bootstrap

Keep bootstraping minimal
While it is tempting to solve loops by bootstrapping every unit test and binary build, remember that we live in an imperfect world, and the Go ecosystem is very young. Packages may be stuck in bootstrapped mode quite a long time. Packagers SHOULD limit bootstraping exclusions to the minimum necessary.

A dependency cycle, that only involves unit tests in one of the affected packages, can be fixed by bootstraping the tests that contribute to the cycle in that package.

A dependency cycle, with mutual production code dependency, requires bootstraping both ends.

When the part that requires the other end is isolated in directories, which are not themselves required by the other end, it is also possible to reduce the problem to the first case, and avoid one of the bootstraps, by splitting those directories in a separate code package. This operation can, however, be quite complex.

Handling renamings

General rules

Go code packages MUST always be named after their current canonical import path. An upstream renaming triggers the package renaming process if the source package is named %{goname} and %gorpmname computes a different %{goname} from the new %{goipath}.

In any case, you MAY generate one or several import path compatibility packages during the import path change transition period, with the following pattern.

Should you decide to create an import path compatibility package, it SHOULD include the usual obsoletion rules. Otherwise, those rules SHOULD be added to the main Go code package. Providing the previous Go code package name is usually not necessary since Go code packages are only accessed through golang(…) provides which are automatically computed.

Example

The following only works from within the renamed package spec file. Automated dependency computation requires access to the rest of the project code during the build process. 

%global goipath     github.com/sirupsen/logrus
[…]
%global oldgoipath github.com/Sirupsen/logrus
%global oldgoname  %gorpmname %{oldgoipath}
[…]
%package -n compat-%{oldgoname}-devel
Summary:   %{summary}
BuildArch: noarch
 
Obsoletes: %{oldgoname}-devel < %{version}-%{release}
 
%description -n compat-%{oldgoname}-devel
%{common_description}
 
This package contains compatibility glue for code that still imports the
%{oldgoipath} Go namespace.
[…]
%install
[…]
%goinstall $gofiles
[…]
install -m 0755 -vd %{buildroot}%{gopath}/src/%(dirname %{oldgoipath})
ln -s %{gopath}/src/%{goipath} %{buildroot}%{gopath}/src/%{oldgoipath}
[…]
%files -n compat-%{oldgoname}-devel
%dir %{gopath}/src/%(dirname %{oldgoipath})
%{gopath}/src/%{oldgoipath}

This is sufficient to generate a working compatibility package: 

rpm -qp --provides compat-golang-github-sirupsen-logrus-devel-1.0.3-11.fc28.noarch.rpm
golang(github.com/Sirupsen/logrus) = 1.0.3-11.fc28
golang(github.com/Sirupsen/logrus/hooks) = 1.0.3-11.fc28
golang(github.com/Sirupsen/logrus/hooks/syslog) = 1.0.3-11.fc28
golang(github.com/Sirupsen/logrus/hooks/test) = 1.0.3-11.fc28
golang-github-sirupsen-logrus-devel-compat = 1.0.3-11.fc28

$ rpm -qp --requires compat-golang-github-sirupsen-logrus-devel-1.0.3-11.fc28.noarch.rpm
golang(github.com/sirupsen/logrus) = 1.0.3-11.fc28
golang(github.com/sirupsen/logrus/hooks) = 1.0.3-11.fc28
golang(github.com/sirupsen/logrus/hooks/syslog) = 1.0.3-11.fc28
golang(github.com/sirupsen/logrus/hooks/test) = 1.0.3-11.fc28

Limitations and pitfalls

Symlinks do not fix code
This pattern is not sufficient to handle a project that also made API changes. Packages that rely on the old API must be patched or upgraded to a code state which is compatible with the new API. If that is not possible, they require the creation of a complete Go code compatibility package.

Also note that in this example, creating the compatibility symbolic link required the creation of an intermediate directory, that was then assigned to the compatibility package: 

%files -n %{oldgoname}-devel-compat
%dir %{gopath}/src/%(dirname %{oldgoipath})
%{gopath}/src/%{oldgoipath}

The import path compatibility package SHOULD own all the intermediary directory levels, between %{gopath}/src/ and %{gopath}/src/%{oldgoipath}, that are not already owned by the main Go code package.

Import path patching
This pattern is not sufficient to handle accesses to %{oldgoipath} from the renamed package itself, and from packages that test the effective import path is the one they expect. In both cases, packagers MUST patch the Go code importing the incorrect import path, to use the new %{goipath}. Packagers SHOULD NOT copy the same code in two different locations, eventually patching import paths, to avoid fixing the packages that call the wrong import path, and check it has not changed.

Handling API changes

API changes may require the creation of Go code compatibility packages, unless the project changing its API uses semantic versioning and changes its import path with major releases.

Notification and handling responsibility

Due to lax release discipline in many Go projects, such changes can be difficult to detect by the packager of the project that changed its API (see also: Go developper guidance). The same lack of release discipline, the fast pace of Go development, the pervasive use of vendoring, mean many projects may migrate to the new API, before all the broken projects identify they have a problem, let alone fix it. And since Go projects do not use dynamic linking, the breakage may be deferred to the next rebuild of the other projects. This rebuild can occur months later, during a routine version bump or during an unrelated automated mass rebuild.

Ideally, the Go code compatibility package should be created by the packager of the project that changed its API. This packager knows the project best. However, this packager may fail to identify the breakage at the time of its occurrence, may not be available when the breakage is identified, may need projects that already require the new API state. Therefore:

  • a packager, that identifies an API change in his Go package, MUST notify golang@fedoraproject.org at least a week before pushing his changes to any release (and those releases SHOULD include fedora-devel). This grace period can be waived by FPC for security reasons.
  • a packager, that identifies an unannounced API change in another Go package, MUST notify golang@fedoraproject.org at once.
  • those notices SHOULD be copied to the maintainers of packages affected by the change, when they are known.
  • a packager MAY NOT require the rollback of another Go package to a previous code state. History goes forward, and Go software version handling is already messy enough. He MUST rely on Go code compatibility packages if his own packages can not be migrated to the new API.
  • a packager MAY NOT require the creation of a Go code compatibility package by the packager of the project that changed its API. This packager may have no need of the old API or be unavailable. Ultimately, the burden of making old API users work, rests of the packagers of those users.
  • a packager, that identifies that the code he packages, is broken by API changes in another project, SHOULD notify politely the upstream of the broken project of the API change, if he can not ascertain it is already aware of the change.

Go code compatibility package creation

The creation of Go code compatibility packages is almost identical to the creation of a normal Go package, with the following twists:

  • specific naming rules apply
  • compatibility packages MUST NOT include any Obsoletes rules
  • their subpackages that ship Go code MUST protect themselves with:
Conflicts: golang(%{goipath}) > %{version}-%{release}
Conflicts: golang(%{goipath}) < %{version}-%{release}

Replace %{goipath} with the root import path of each subpackage if you've decided to split the shipped code in several subpackages.

Therefore, most Go code compatibility packages can be trivially derived from the package that ships the most recent code state. The following precautions apply:

  • The creator of a Go code compatibility package MUST check the original BuildRequires are actually needed by its compatibility package. They may have changed between versions.
  • The creator of a Go compatibility package SHOULD check if unit tests disabled in the original package, can be re-enabled in this version.
Never mix two code states in the same source packages
Go packagers MUST NOT try to generate code compatibility packages from other packages. Mixing the BuildRequires, of two code states, on a diverging course, will quickly degenerate. Go packagers MUST create a separate spec and source rpm for each code state.

Go code compatibility package use

Once the Go code compatibility package is created, other packages can force its use in their BuildRequires with: 

BuildRequires: golang(provided-import-path) = compat-version

or 

BuildRequires: golang(provided-import-path)(tag=compat-tag)

or 

BuildRequires: golang(provided-import-path)(commit=compat-hash)

This kind of version locking is of course possible with normal Go packages. Their Provides are computed the same way. However, nothing ensures normal Go packages won't be updated later (and other packagers MUST NOT hinder those updates).

Compatibility Go code packages are the only ones frozen in time. Version locking on normal Go packages is only useful as a way to detect changes.

Putting it all together

%{?_with_bootstrap: %global bootstrap 1}
%global goipath     github.com/performancecopilot/speed
Version:            2.0.0
%gometa
  
%global common_description %{expand:
A Go implementation of the Performance Co-Pilot (PCP) instrumentation.}
   
Name:    %{goname}
Release: 10%{?dist}
Summary: Go libraries implementing PCP instrumentation
License: MIT
URL:     %{gourl}
Source:  %{gosource}
 
BuildRequires: golang(github.com/codahale/hdrhistogram)
BuildRequires: golang(github.com/edsrzf/mmap-go)
# break go.uber.org/zap →
#       github.com/go-kit/kit/log →
#       github.com/performancecopilot/speed
%if ! 0%{?bootstrap}
  BuildRequires: golang(go.uber.org/zap)
%endif
 
%description
%{common_description}
 
%package   devel
Summary:   %{summary}
BuildArch: noarch
 
Requires: %{name} = %{version}-%{release}

%description devel
%{common_description}
 
This package contains the source code needed for building packages that import
the %{goipath} Go namespace.
 
%prep
%forgesetup
rm -fr vendor
 
%build
%gobuildroot
%gobuild -o _bin/mmvdump %{goipath}/mmvdump/cmd/mmvdump
 
%install
gofiles=$(find . %{gofindfilter} -print)
%goinstall $gofiles
 
install -m 0755 -vd        %{buildroot}%{_bindir}
install -m 0755 -vp _bin/* %{buildroot}%{_bindir}/
 
%check
%if ! 0%{?bootstrap}
  %gochecks 'example*'
%else
  %gochecks . 'example*'
%endif
  
%files
%license LICENSE
%doc *\.md
%{_bindir}/*

%files devel -f devel.file-list
%doc examples
   
%changelog
[…]

Go developer guidance: making your software third-party-friendly

This section tries to identify best practices, that contribute to making a Go software project third-party-friendly. They were collected at the request of Go developers, and will hopefully be completed and clarified over time.

Most of them are not Go, Fedora or Linux specific. They decline in Go terms common software engineering golden rules. They are the result of the insight, gained trying to build and deploy in the best conditions, hundreds of Go projects. Each one has been identified after someone made an expensive mistake. Learn from the mistakes of others, do not wait to reproduce them! Lastly, a lot of them are really simple and easy to follow. Failure can be complex but failure causes are often depressingly basic.

Applying those rules will facilitate the appropriation of your software by third parties, resulting in a better reputation, a larger user-base, and increasing the probability someone will contribute positively to your project. The rules take a forward-thinking approach, eliminating technical debt and other problems before they get critical and expensive to solve. They will reduce your own software maintenance costs, increase your velocity, help the onboarding of new project members. They will ensure you reap the maximum benefit of publishing your project as free and open source software, regardless of eventual Linux distribution involvement.

Contrary-wise, not following those rules will increase friction and probability of future failure by third parties or you. Failure can take many forms: complete failure (the software can not be built or deployed), partial failure (the software is deployed, in a partial or dysfunctional state), lag (the software is only deployed a long time after you finished coding it, in old feature-poor versions, that are never updated afterwards). Failure increases the probability of people asking annoying and time-wasting questions in your issue tracker and communication channels. Failure usually results in a damage to your reputation. It does not matter if you are yourself able to navigate most of the pitfalls resulting from not following best practices. In a free and open source software world, you can not prevent others from trying to use your software their own way. Their failures will be shared with you just like successes would be.

Import path

  • avoid changing your import path
  • do not make your code accessible through multiple import paths
  • choose an import path in lower case. Some operating systems are case insensitive; others — not. You will eventually have contributors and users from both worlds, lower case ensures they work well together.
  • communicate clearly the canonical import path of your project at the top of its README.md, especially if your project page can be reached through URLs that differ from this canonical import path.
  • if you absolutely need to change the import path of your project start by changing the import paths inside your own code, do not rely on https redirections.
  • that includes testing and example code.
  • do not append a .git suffix to your import path
  • do not sprinkle your import path with go or golang qualifiers, people knows it's a Go project
  • version your import paths with each major release (gopkg.in is a good example of how to do that)

Directory structure

  • put the code that needs to be built into a binary in cmd/binary-name. Do not reuse other software binary names, someone will end up renaming your binaries to avoid conflicts and you will not like the result.
  • put all the assets needed by tests in a root-level _testdata directory (it can contain subdirectories).
  • put all your example code in a root-level _examples directory (it can contain subdirectories). Do not use creative unusual names such as “tutorial”.
  • do not pepper your subdirectories with documentation files. Keep the documentation at root level or in a doc root-level subdirectory if there is too much of it.
  • add a one-line summary and a least a paragraph describing what your project does at the top of your README.md.
  • if part of your project is generated code, document cleanly how to remove what you already generated and how to regenerate it from scratch. And try to keep the result in a a specific root-level subdirectory.
  • if your project includes other materials not required to build software depending on your project, (unit tests excepted), keep them in a dedicated root-level directory and document the directory use.
  • if your project includes materials, required to build software depending on your project, which are not .go, .h, .c or .s files, try to keep them near the project root and document clearly they exist.
  • do not hide vendor subdirectories deep inside your project tree. Always use a root-level subdirectory.
  • if your project provides wrappers, connection glue or anything similar to many many many other projects keep the code for each other project in a separate subdirectory (Go package). This way, in case of problem in the other project, it can be deactivated without impacting the rest of your code.

Licensing

  • choose licenses already vetted by Fedora or Debian
  • include a detached licensing file named LICENSE
  • use unmodified plaintext canonical licensing text, that state the license name at the top of the file. Do not break automated checks by “fixing” wording, casing or indenting.
  • if you absolutely want to add an extension to make Windows happy, use LICENSE.txt
  • if you absolutely want to name your licensing file something else, please do not use something.md
  • do not hide your licensing terms in README.md
  • do not refer to a license by name without providing its text

Releasing

  • do releases
  • use semantic versioning for your releases. Distributors and godep will thank you
  • do releases, even for minor fixes. If you haven't felt the need to touch a project for months its latest state should be released! Otherwise many will ignore your latest fixes and pester you about already fixed problems.
  • do releases, whenever you alter your API, extending it, modifying it, removing some. Document succinctly the changes in the release notes. Otherwise, other projects will fear your changes, start to depend on magic commit states, refuse to apply your latest improvements, and continue for a long time to ship code your are not proud of.
  • do releases, even for projects that can only be called through another project. Do not rely on this other project to set code state through vendoring (that's easy to do, just propagate the tip project version to the ancillary projects at release time, like kubernetes does)
  • if your project is in an SCM, use a different branch for every major version of your project. That will make it easy to push fixes to past major releases when the need arises (look at GC). That will usually result in a new minor release. If you maintain several major releases, in the same SCM, without branching, someone (that can be you), will end up attributing a commit to the wrong release.
  • if your project is in an SCM, tag your x.y.z release x.y.z, or vx.y.z, never vx.y.zprettybeta. Versions and version tags are not the right place to document a project maturity state. That's what release notes are for.
  • numbers are cheap, never reuse the same number for a pre-release and a final release, increase the minor version!

Using other projects from your code

  • use releases of the projects you depend on.
  • never depend on a project that already depends on you, directly or indirectly. That removes the directed acyclic graph property from the relationship graph. It will get you in trouble sooner or later. Vendoring can only workaround compilation problems. It does not prevent the mess that inevitably arises when two projects do not have a clear unidirectional relationship, and evolve in conflicting directions. Common governance can mitigate those but common governance has a cost in velocity and effort and effectively means the projects will have to go through a merge eventually (reverting relationship cycles is hard once they are entrenched).
  • if for some reason, one of the projects you depend on does not release, please ask nicely it to do so.
  • if for some reason you need a code state in tip which is not in a release, please inform the origin you'd like it to do a release, and switch to this release as soon as it is available.
  • never depend on a commit state somewhere between two releases.
  • document the required major versions of the projects you depend on somewhere easy to find. If a major version is only usable past as specific minor version, document it.
  • add a unit test that detects if the project you depend on is missing the part that requires being after this particular minor version.
  • do not write code or tests, that depend on example code.
  • do not write code or tests, that depend on the non-production code, assets or test data of another project. Those are not intended to be used externally.
  • identify clearly all the items sourced from third parties and not located in vendor in your documentation.
  • do remember, that material other than Go code, is also subject to copyright law and licensing terms.

Changing third party code or assets

  • never add changes to the projects you reuse, submit the changes to those projects
  • if you absolutely need to change a project you reuse, fork it cleanly with a new import path so distributors do not accidentally reinject the original project at build time.
  • do not publish forks that still reference themselves as the original project in code or documentation.
  • do document the origin and motivation or your forks.
  • don't forget to rebase your code on the original project if you don't have the energy to maintain your own fork. Document the fork is deprecated in its README.md.
  • rebase to the latest minor version of every project you depend on at release time. Do not let changes accumulate till rebasing becomes a major endeavor.

Testing

  • do not test projects that depend on you from your project. Contribute the testing code to those projects (otherwise that removes the directed acyclic property from the relationship graph).
  • do not ship not-working testing code (people will think their build is broken, and waste their time).
  • do not test that a project you depend on has a specific import path, it may rename itself in the future.
  • do not ship tests that depend on special parameters, go test should always just work.
  • do not ship tests that rely on being in a specific directory, as long as the test code in in GOPATH it should be testable regardless of the location.
  • do not ship tests that depend on specific timings, distributor build farms may be heavily loaded, causing the tests to fail in non-predictable ways.
  • make sure that all tests and test-only imports are exclusively located in *_test.go files.
  • do make sure tests of any kind are actually exclusively located in *_test.go files. Apply this convention even when they are located in a directory “obviously” associated with testing (it may not be obvious to others, or to automation).
  • if a test depends on a special preparation, ship a shell script that describes this preparation. Do not rely on years-old magical Ubuntu docker images.
  • if a test depends on network access, make it skip and log gracefully when this access is blocked (never FAIL).
  • if a test has specific requirements (hardware, some other software instance to connect to…), detect those, and skip and log if not met
  • never do go get from tests.
  • do not make your tests resolve directory names to their real paths, and complain the result is different from what you expected. Learn to work with Unix and Linux symbolic linking.
  • never stomp on GOPATH in tests.
Retrieved from "https://fedoraproject.org/w/index.php?title=More_Go_packaging&oldid=511027"