Scala and sbt
Overview
Scala combines a lot of excellent features (functional-style pattern matching, an expressive type system, closures, etc.) with JVM compatibility and a very interesting developer ecosystem. Fedora has included a package for Scala itself for some time (although the Fedora Scala package is currently broken). However, Fedora doesn't include any of the Scala ecosystem projects. The main obstacle to having Scala ecosystem projects in Fedora is that many projects use sbt, the Simple Build Tool, but there is no native Fedora sbt package. If sbt were available on Fedora, it would greatly facilitate packaging other Scala packages for Fedora. We are taking steps to make Scala and sbt available in Fedora; see here for more background.
If you just want to get started helping out, check out these tips for packaging with sbt and our current packaging wishlist.
Building Scala
The latest stable release of the Scala language is 2.10. It isn't possible to build 2.9 on Fedora 19 or later. Some interesting ecosystem projects don't officially support 2.10, although 2.10 is mostly source-compatible with version 2.9. (Since version 0.9.0, Spark supports Scala 2.10.)
A revised Scala package for F19 and up is available from Fedora testing repositories.
Challenges
We had to overcome several challenges in order to build Scala for Fedora:
- The upstream build uses ant and relies on maven-ant-tasks in order to pull down dependencies. For a variety of reasons, maven-ant-tasks is a poor fit for Fedora, but patching ant build.xml files is tedious and brittle. In the future, it would be best to have a more automated approach to ant build patching (a la the functionality available to packagers via maven pom macros). Another possibility is to maintain a separate, leaner build file for Fedora. Gil Cattaneo developed such a script (currently missing documentation generation and some POM files); Will made a first pass at using it for packaging.
- Scala depends on a forked version of JLine that is roughly API-compatible with JLine 2.5. In prior Fedora releases, it was possible to simply replace references to the Scala JLine with references to the system JLine. However, the Scala JLine API is incompatible with stock JLine 2.7 and up, so Scala requires a more involved patch in order to work with Fedora 19 and later.
- A bug in the Scala optimizer interacts poorly with JLine (either version), so it is necessary to build Scala with optimizations turned off.
- The test suite and scala-partest artifact have missing dependencies in Fedora; getting them to run locally (viz., outside of a build) required some pruning to remove spurious failures (e.g. here, here, and here). (In particular, the binary-compatibility test checked files compiled by an optimizing scalac against files compiled by our scalac; since we need the optimizer turned off, this exposed a problem like this one.)
Limitations
- We'd like to have the test suite and scala-partest artifact in a build and run the former as part of %check.
Missing dependencies
These are immediate dependencies for the Scala test suite (there may be other transitive dependencies). We anticipate that getting the Scala test suite running under mock will be a nontrivial effort.
| Project | State | Review BZ | Packager | Notes |
|---|---|---|---|---|
| com.googlecode.java-diff-utils:diffutils | Review | 1013039 | willb | check out the packaging repo; the open issue is removing some cut-and-paste code reuse to comply with bundling policy |
| org.ops4j.pax.exam:pax-exam-container-native | Available | version 2.5.0 | ||
| org.ops4j.pax.exam:pax-exam-junit4 | Available | version 2.5.0 | ||
| org.ops4j.pax.exam:pax-exam-link-assembly | Available | version 2.5.0 | ||
| org.ops4j.pax.url:pax-url-aether | Available | version 1.4.0; Gil has a pax-url spec | ||
| com.typesafe:mima-reporter_2.9.2 | Available | version 0.1.5 (needs Scala to build; used for binary-compatibility testing) |
Other Scala projects to package
These are immediate build and test dependencies for other projects that use Scala and sbt. In some cases, we've been able to work around their omission, but having these available would make it possible to package more Scala projects and would bring the packages we have available more in line with the upstream builds.
Please take some of these and package them, if you're so inclined!
| Project | State | Review BZ | Packager | Notes |
|---|---|---|---|---|
| scalacheck | in testing | 1056335 | willb | property-based testing (à la QuickCheck) |
| chill | Scala extensions for Kryo | |||
| specs2 | willb has a preliminary package | |||
| scalatest | Note that the current version (2.0) is more recent than that required by most ecosystem projects. | |||
| Ooyala job server | job server for Spark-as-a-service | |||
sbt-ghpages
|
sbt plugin to generate GitHub pages-style documentation
| |||
sbt-site
|
sbt plugin to generate project websites
| |||
sbt-git
|
sbt plugin to access git functionality
| |||
sbt-release
|
sbt plugin to automate the release process (including generating dependency metadata, etc.)
| |||
sbt-unidoc
|
sbt plugin to generate unified API documentation across projects
| |||
sbt-osgi
|
sbt plugin to generate unified OSGi bundles
|
Building sbt
We currently have sbt 0.13.1 under review in Fedora. It includes a patch to work with Ivy 2.3.0. Having sbt available in Fedora is currently the main obstacle to getting interesting parts of the Scala ecosystem into Fedora. The rest of this section will discuss some of the pitfalls involved in packaging sbt.
One of the big obstacles to building sbt is its reliance on Ivy repositories with metadata. We have both short-term and longer-term solutions to this problem: a patch to make sbt work with bare Ivy repositories, a tool to make a temporary Ivy repository from installed Fedora packages, and improved Fedora support for Ivy. In the future (F21+), it will be possible to use the latter as part of a much simpler build process.
We need to build sbt 0.13.x — and not sbt 0.12.x — because we use Scala 2.10, and we need to build sbt 0.13.1 in particular because sbt's bootstrap process depends on a particular version of the Scala compiler and 0.13.1 is the first version that works with (and requires) Scala 2.10.3. As a consequence, Scala and sbt in Fedora should necessarily move more slowly than other packages; we'll need to ensure that everything we want to build will work with a new compiler or sbt before upgrading any pieces.