SIGs/bigdata/packaging/Sbt

Building packages that use sbt

This is a quick guide to getting Fedora packages built with sbt. Since sbt uses Ivy for dependency resolution, you'll either need to construct a local Ivy repository as part of the build (for F20) or rely on xmvn's improved support for Ivy metadata (in F21 and above after xmvn 2.0 lands). For now, this guide covers building packages on F20 with the climbing-nemesis script.

Eventually, I hope that the best practices around building Scala projects will coalesce into official Scala packaging guidelines, but for now it's sort of the wild west. Your feedback is welcome!

Differences with upstream sbt

The Fedora sbt package is primarily intended to be used for packaging sbt-based projects in Fedora. While you'll be able to use it to do general Scala development as well, some features that sbt users expect to have available (most notably, cross-building for incompatible Scala versions and launching different versions of sbt for different projects) have been removed since it is difficult or impossible to support them while meeting Fedora packaging guidelines.

How Fedora sbt resolves Ivy dependencies is specified by a properties file (see also sbt resolver specifications). For the Fedora sbt, this defaults to /etc/sbt/sbt.boot.properties, but you can override this by setting SBT_BOOT_PROPERTIES in your environment. Without overriding the properties file, the sbt script shipping in Fedora will allow you to specify the path to a local Ivy repository (set SBT_IVY_DIR) and the path to a cache of resolved artifacts that sbt needs to run (set SBT_BOOT_DIR). We'll discuss these more later in this document.

General tips and tricks

sbt build files are either written in a domain-specific language (*.sbt files) or are general Scala code that incorporates sbt as a library, and they typically either live in the root directory of a project's source tree or in the project directory. Because they aren't XML files, patches against sbt builds are typically less brittle than patches against Ant build files, and it's more straightforward to carry Fedora-specific build patches as sed commands.

sbt build files list dependencies in a particular format: GROUP % ARTIFACT % VERSION or GROUP % ARTIFACT % VERSION % CONFIG . You will also occasionally see dependencies of the form GROUP %% ARTIFACT % VERSION ; the double-percent sign indicates that the Scala version should be appended to the artifact name, as is common practice for Scala artifacts in Maven and Ivy repositories.

You'll want to patch the project/build.properties file, if there is one, to make sure that the project expects the version of sbt that is packaged in Fedora. You'll also want to make sure that the Scala version expected by this project is exactly the same as the one available in Fedora. If the project expects Scala 2.10.2, you'll need to patch the build files to have it look for 2.10.3 instead.

Any sbt plugins mentioned in your project's build are probably not available in Fedora. Fortunately, most builds will proceed just fine without them (at least if we're building for Fedora); a pattern that works for many projects is to remove the contents of project/plugins.sbt in %prep and then make a Fedora-specific patch eliminating any code from other build files that fails to compile once the plugins are removed. (Of course, packaging plugins is a great way to help out!)

Helpful macros

The first of these macros, remap_version, takes a group ID, an artifact ID, a version string, and a sbt build file and updates the file so that any dependency on the given artifact is for the version specified in the macro invocation and not on whatever version was in the build file.

%global remap_version() sed -i -e 's/"%{1}" %% "%{2}" %% "[^"]*"/"%{1}" %% "%{2}" %% "'%{3}'"/g' %{4}

remap_version_to_installed takes a group ID, an artifact ID, and a sbt build file and updates the file so that any dependency on the given artifact is for the version of that artifact installed on the build machine.

%global remap_version_to_installed() sed -i -e 's/"%{1}" %% "%{2}" %% "[^"]*"/"%{1}" %% "%{2}" %% "'$(rpm -q --qf "%%%%{version}" $(rpm -q --whatprovides "mvn(%{1}:%{2})" ))'"/g' %{3}

Using climbing-nemesis

The climbing-nemesis script uses Fedora's Maven metadata to construct and populate a local Ivy repository with artifacts you select. To use it, you'll need the script itself to be accessible from your build directory and to have added BuildRequires on python and maven-local. It has many options, all of which are documented in its on-line help; we will cover some of the most common ones here. The most basic climbing-nemesis invocation consists simply of a group ID, an artifact ID, and a path to the local Ivy repository, like this:

./climbing_nemesis.py com.h2database h2 ivy-local

The above will install com.h2database:h2 into ivy-local, creating Ivy metadata from Maven metadata and making a symbolic link from the Ivy repository to the local, RPM-managed artifact. If you'd like to override the version number that xmvn-resolve decides is appropriate for the artifact, you can do so with the --version flag. Since packages occasionally have broken Maven metadata, it can be convenient to override the version string by default; this macro is helpful in that case:

# The first argument is a group ID, the second is an artifact ID
# Wrap the whole invocation in curly braces if you need to pass additional arguments!
%global climbing_nemesis() ./climbing-nemesis.py %{1} %{2} ivy-local --version $(rpm -q --qf "%%%%{version}" $(rpm -q --whatprovides "mvn(%{1}:%{2})" ))

By default, climbing-nemesis will install all dependencies that aren't listed under test or optional configurations. Because we sometimes don't use every capability implied in a POM file and because Maven metadata sometimes has incorrect dependency information, you may wish to remove or override dependencies. You can remove dependencies with the --ignore flag, add extra dependencies with the --extra-dep flag, and override the required version of a dependency by doing both, like this:

%{climbing_nemesis cglib cglib-full ivy-local} --ignore asm --extra-dep asm:asm:$(rpm -q --qf "%%{version}" $(rpm -q --whatprovides "mvn(asm:asm)" ))

Some Fedora Java packages don't have Maven metadata at all. In those cases, we can explicitly specify where to find a JAR file and what version it is:

./climbing-nemesis.py --jarfile %{_javadir}/foofactory.jar com.foobar foofactory ivy-local --version 42.1

You can also use the --pomfile and --ivyfile options to take metadata from descriptor files pulled down from upstreams.

Some Scala packages use additional Ivy metadata. This will be supported in xmvn 2.0, but for now, we need to specify it explicitly using the --meta option to climbing-nemesis, for example --meta e:scalaVersion=2.10. You can also use the --scala VERSION option to append _VERSION to the artifact name.