SIGs/bigdata/packaging/Mesos

This is an summary as a part of the Big Data SIG of the issues uncovered during Mesos packaging.

Resolved Issues

• Removal of hard 3rdparty dependencies and replaced with autoconf checks where possible.
• Updated #define CHECKS on library interface dependencies
  • http_parser
  • Zookeeper
• Updates to follow standard conventions around include dir (e.g. process->libprocess)
• Updates to follow standard #ifndef __LIB_HEADER__ checks (some were missing, or not prefixed _LIB_).
• System integration files. (inside package)
• env deployment overloads moved from /var/mesos to /etc/sysconfig/mesos, more appropriate for env settings
• libmesos now a properly versioned library

Upstream Tracking

• https://issues.apache.org/jira/browse/MESOS-543
• Patches: https://issues.apache.org/jira/browse/MESOS-543

Packaging Breakdown

• stout
  • forked-repo: https://github.com/besser82/stout
  • rpm-spec: https://github.com/ignatenkobrain/stout-rpm
  • packaging BZ: https://bugzilla.redhat.com/show_bug.cgi?id=988545
• libprocess
  • forked-repo: https://github.com/ignatenkobrain/libprocess
  • rpm-spec: https://github.com/ignatenkobrain/libprocess-rpm
  • packaging BZ: https://bugzilla.redhat.com/show_bug.cgi?id=994152
• mesos
  • forked-repo: https://github.com/timothysc/mesos
  • rpm-spec: https://github.com/timothysc/mesos-rpm
  • https://bugzilla.redhat.com/show_bug.cgi?id=1010512

Open Issues

I've tried to list the open issues in priority order. All recommendations listed here should be construed as developer opinion, and the reader should choose whatever path makes the most sense.

Canonical source for dependencies

Both libprocess and stout are considered separable libraries with their own repos on github but are not setup as submodules within the main repository thus confusing "What is the definitive source?" and whether those libraries should be broken out separately or not? Currently I've tried to sync with the mesos repository and update the packages in fedora while supplying a proper autoconf check in mesos for the dependencies.

more autotools cleaning

Currently I have checks that will match the packages I've created, and will likely work if folks want to pull and rebuild the .srpm for their own koji setup, but there may be check mismatch on other distros. I also may have cheated in places by ensuring their proper rpm dependencies.

.jar(s) not installed?

I don't understand why mesos.jar is not installed.

JNI reaching into zookeeper utility classes

The reliance on internal utility classes inside of zookeeper can be quite problematic. Applications developers will usually supply a contract around the interface but the inner-workings are usually subject to churn. During the packaging it became apparent of the drift.

Recommendations:

• Remove the dependencies and try to stick with the interface where possible.
• Use a pImpl idiom || some factory mechanism and hide the JNI goo below such that autoconf can enable impl1 impl2.

Failing Tests

Given the stack changes there are approximately 2 dozen tests that have been temporarily disabled, many of them are Zookeeper integration issues. We will carry this patch in our packaging, but the afore mentioned issue should be addressed first.

Some tests also require DNS, which causes issues in a mock/koji env, but we can carry those patches too.

Boost / TR1 / C++11

There appears to be deps on tr1 but no checks for C++11. Adding checks where possible and leverage semantics C++11 where possible is preferred for long term maintenance. This applies to several elements in the code.. threading (pthread), variadic templates (tuple), etc.