Scala and SBT
Overview
Scala combines a lot of excellent features (functional-style pattern matching, an expressive type system, closures, etc.) with JVM compatibility and a very interesting developer ecosystem. Fedora has included a package for Scala itself for some time (although the Fedora Scala package is currently broken). However, Fedora doesn't include any of the Scala ecosystem projects. The main obstacle to having Scala ecosystem projects in Fedora is that many projects use sbt, the Simple Build Tool, but there is no native Fedora sbt package. If sbt were available on Fedora, it would greatly facilitate packaging other Scala packages for Fedora. We are taking steps to make Scala and sbt available in Fedora; see here for more background.
Building Scala
The latest stable release of the Scala language is 2.10. It isn't possible to build 2.9 on Fedora 19 or later. Some interesting ecosystem projects (in particular, Spark) don't officially support 2.10, although 2.10 is mostly source-compatible with version 2.9.
A revised Scala package for F19 and up is available from Fedora testing repositories.
Challenges
We had to overcome several challenges in order to build Scala for Fedora:
- The upstream build uses ant and relies on maven-ant-tasks in order to pull down dependencies. For a variety of reasons, maven-ant-tasks is a poor fit for Fedora, but patching ant build.xml files is tedious and brittle. In the future, it would be best to have a more automated approach to ant build patching (a la the functionality available to packagers via maven pom macros). Another possibility is to maintain a separate, leaner build file for Fedora. Gil Cattaneo developed such a script (currently missing documentation generation and some POM files); Will made a first pass at using it for packaging.
- Scala depends on a forked version of JLine that is roughly API-compatible with JLine 2.5. In prior Fedora releases, it was possible to simply replace references to the Scala JLine with references to the system JLine. However, the Scala JLine API is incompatible with stock JLine 2.7 and up, so Scala requires a more involved patch in order to work with Fedora 19 and later.
- A bug in the Scala optimizer interacts poorly with JLine (either version), so it is necessary to build Scala with optimizations turned off.
- The test suite and scala-partest artifact have missing dependencies in Fedora; getting them to run locally (viz., outside of a build) required some pruning to remove spurious failures (e.g. here, here, and here). (In particular, the binary-compatibility test checked files compiled by an optimizing scalac against files compiled by our scalac; since we need the optimizer turned off, this exposed a problem like this one.)
Limitations
- We'd like to have the test suite and scala-partest artifact in a build and run the former as part of %check.
Missing dependencies
These are immediate dependencies for the test suite (there may be other transitive dependencies):
| Project | State | Review BZ | Packager | Notes |
|---|---|---|---|---|
| com.googlecode.java-diff-utils:diffutils | Review | 1013039 | willb | check out the packaging repo; the open issue is removing some cut-and-paste code reuse to comply with bundling policy |
| org.ops4j.pax.exam:pax-exam-container-native | Available | version 2.5.0 | ||
| org.ops4j.pax.exam:pax-exam-junit4 | Available | version 2.5.0 | ||
| org.ops4j.pax.exam:pax-exam-link-assembly | Available | version 2.5.0 | ||
| org.ops4j.pax.url:pax-url-aether | Available | version 1.4.0; Gil has a pax-url spec | ||
| com.typesafe:mima-reporter_2.9.2 | Available | version 0.1.5 (needs Scala to build; used for binary-compatibility testing) |
Building sbt
We currently have sbt 0.13.1 under review in Fedora. It includes a patch to work with Ivy 2.3.0. Having sbt available in Fedora is currently the main obstacle to getting interesting parts of the Scala ecosystem into Fedora. The rest of this section will discuss some of the pitfalls involved in packaging sbt.
One of the big obstacles to building sbt is its reliance on Ivy repositories with metadata. We have both short-term and longer-term solutions to this problem: a patch to make sbt work with bare Ivy repositories, a tool to make a temporary Ivy repository from installed Fedora packages, and improved Fedora support for Ivy. In the future (F21+), it will be possible to use the latter as part of a much simpler build process.
We need to build sbt 0.13.x — and not sbt 0.12.x — because we use Scala 2.10, and we need to build sbt 0.13.1 in particular because sbt's bootstrap process depends on a particular version of the Scala compiler and 0.13.1 is the first version that works with (and requires) Scala 2.10.3. As a consequence, Scala and sbt in Fedora should necessarily move more slowly than other packages; we'll need to ensure that everything we want to build will work with a new compiler or sbt before upgrading any pieces.