Spark packaging
Background
See Scala packaging for details on Fedora support for the Scala toolchain; briefly, we have version 2.10.3 of Scala in F19, F20, and Rawhide and sbt 0.13.1 is under review in Fedora as well.
There is no official release of Spark that works with Scala 2.10.3 (or SBT 0.13.x), but the 0.9 branch of the upstream Spark repository works with Scala 2.10.
Current Status
With some light Fedora-specific patching, we are able to build and run the Scala 2.10 branch of Spark locally in Fedora against our sbt and Scala. Once sbt is available, other things should follow pretty quickly. Detailed dependency information is below.
Most of the Scala packages build with tests disabled (due to unavailable test dependencies) or with varying degrees of modification to the upstream build process (due to varying dependency versions or unavailable sbt plugins). If you're looking for an easy way to get involved, packaging some of these missing dependencies would be a great place to start. See the list under Scala packaging or find willb on IRC for more information
Dependencies
Spark requires Scala and SBT in order to build. (Note that there is a Maven build option as well, but it requires artifacts that need to be built with SBT.)
The easiest and most up-to-date place to see the dependency list is in the Spark repository itself, but here we will call out some notable dependencies that aren't already in Fedora:
| Project | State | Review BZ | Packager | Notes |
|---|---|---|---|---|
| sbt | under review | sbt-package BZ | willb | |
| Not necessary any more with this patch (either carried or integrated into upstream) | willb | |||
| json4s | available to package | NB: this will either require a bundling exception (since it has a forked lift-json, which we won't be using) or a surgical patch | ||
| akka | gil and willb are looking at this | gil | ||
| Squeryl | awaiting a reviewer | 1057770 | willb | dependency of lift; no longer strictly necessary but still nice to have for the Scala ecosystem |
| scalaz | awaiting a reviewer | 1055809 | willb | dependency of lift-json; no longer strictly necessary but still nice to have for the Scala ecosystem |
| metrics | available in F20 | 861502 | gil | Coda Hale's metrics (Java/Maven build). |