GSOC 2013/Student Application Axilleas/Gitlab(463)

Short description

GitLab is an open source MIT licenced git repository management application. It is built on Ruby on Rails and is used by many companies as their internal git management repository. The reason it gained so much popularity is that it bares a strong resemblance to github's looks and feels. It is a project with great potential, under heavy development with a release cycle every month. That "release early, release often" strategy makes it possible to apply bug fixes quite regularly and add new features.

This proposal is aiming to package GitLab and all its dependencies for Fedora and EPEL.

What are the benefits

• Supporting more ruby packages would make Fedora the appropriate distribution for ruby development.

• Fedora is one of the biggest open source communities (if not the biggest) so far, with a very strong community base. If GitLab were to be supported officially, it would be the first distribution to do so and could be one of their supported platforms apart from Debian/Ubuntu.

• I have seen many installations on CentOS (forum, issue tracker, blogs), there is demand. rpm packages means "One click install" for Fedora, Red Hat, CentOS and other rpm based distributions. Huge profit, as sysadmins will not have to worry about the manual installation process.

• When the packaging is done GitLab could be deployed as an extra git service for fedorahosted.org. I expect many people to switch their git hosting from the bare gitweb to GitLab. It will enhance collaboration, encourage participation and make coding more fun as it will be an interactive process.

Relevant expertise

Hands on experience

• I have been using/testing GitLab since version 2.0.

• I actively participate in the forum talk, I contribute upstream and I am in direct contact with its developers.

• I have deployed it over 20 times, mostly on Debian, so I am familar with the process.

• I recently started creating gem specfiles.

Other related activities

I have spent several hours contributing not only upstream but I also wrote a wiki article for Archlinux as well as an installation script and recently I gave a small talk in greek about GitLab during a hackfest.

How I intend to implement my proposal

Gem hunting

The first step would be to compile a list of the gems GitLab is using and compare it against the ones already packaged in order to find those missing from the repos.

GitLab gems

Using the Gemfile.lock is a good starting point to compile the needed list. In it there are all the gems GitLab depends on. Many of them are duplicate, so using this little python script leaves us with about 200 gems. Of those 200, most are downloaded from rubygems.org, some are pulled from git repositories, whereas some are patched by GitLab.

However, those gems are not final, as the script doesn't take into account (yet) the development packages, which we don't necessarily need in order to have a working GitLab instance. On the other hand, some of them are needed by the test suite, so one must be careful of what to exclude. I will rework on that script and plan to write it in ruby as well.

Fedora gems

Next step is to find which of the gems above are already packaged for Fedora. I have crafted a little bash script, which searches the repositories of the current Fedora release (no testing/rawhide enabled) for gems. We come up with 387 total gems of which 93 are GitLab's dependencies. In particular:

 Gitlab uses 199 gems.
 Fedora has packaged 387 gems.
 There are 93 common gems.
 There should be packaged 106 gems.

Those 106 gems are not final, as there are also others which are not yet in Fedora, but they are packaged by their upstream. Also, there are many gems pending review in bugzilla.

Unfortunately, there is a caveat which needs to be addressed and that is version mismatch. More about this in #Caveats_and_possible_problems_I_may_encounter.

Packaging for Fedora/EPEL

As stated in the wiki, there are three basic categories of ruby packages: ruby gems, non-gem ruby packages, and applications written in ruby. Fortunately, all dependencies are ruby gems which makes it easier to package (details below). GitLab itself is a rails app which needs extra care.

To make matters easier, new packages should go only into Rawhide. Once Rawhide turns into Fxx, GitLab should be in some fixed stable version in the rest of Fxx life cycle. Once the packaging for Fedora is done, next step is EPEL6, either by editing the existing spec files and using conditions or keep a different version of .spec in each branch. There exists the katello koji which will be a good source of already packaged gems. They have many packaged in EPEL6 as well.

Gems

Gems contain three components:

• Code (including tests and supporting utilities)
• Documentation
• gemspec

They share the same structure so it should be fairly easier to package them.

foo/
├── bin/
│   └── foo
├── lib/
│   └── foo.rb
├── test/
│   └── test_foo.rb
├── README
├── Rakefile
└── foo.gemspec

(Taken from http://guides.rubygems.org/what-is-a-gem/)

Fortunately, there is a tool named gem2rpm which parses the info inside the gemspec and creates a spec file. This facilitates the update process and makes it easier to update the RPM when a new version of the Gem becomes available. This tool will be heavily used for the gem packaging.

On the other hand, there is no way to completely automate the conversion, so some fields will be filled by hand. According to rubygems, the required gemspec attributes are:

• files
• name
• platform=
• require_paths
• rubygems_version
• summary
• version

whereas attributes such as the license or the changelog are not mandatory and we will have to add it ourselves..

GitLab

Once gems are done, it is time to package GitLab itself. There is a new version every month, so Rawhide will always have the latest one. At this present time, GitLab is installed in the home directory of user 'git', but since the RPM packages must obey FHS rules it should be installed into %{_datadir}.

Service files

systemd

For newer versions of Fedora we will need systemd service files. That won't be a difficult task since there are already some unofficial services floating around. All they need is some refinement.

rc.d

An rc.d service file is already provided by upstream for CentOS.

MariaDB support

Since MariaDB will replace MySQL, GitLab must be tested against MariaDB. At this point there should not be any incompatibility. Upstream doesn't yet support it so this should be discussed with them.

Caveats and possible problems I may encounter

Version mismatch

One thing to consider before packaging, is the version mismatch between gems used by GitLab and those packaged/to be packaged for Fedora. Already, GitLab uses about 10 runtime packages pinned to a lower version than upstream. Although it is always a good idea to package the latest version, we might end up packaging a lower version in order to get GitLab to work. If the package is already in Fedora and we need an older version it would be better to test GitLab against the version in Fedora.

On the other hand, of the Ruby gems that are packaged, many of them are the wrong version, and for some of the Ruby gems that GitLab requires, it uses git snapshots of non-released versions, or versions that have been forked/patched by GitLab developers.

So, the first step prior to packaging, will be to compile a list with gems and their corresponding version. Maybe a table like this would be useful:

 Gem name | Upstream version | GitLab version | Fedora version

isfedoraruby and gemnasium are two good tools to find out about version mismatch.

Test suite in mock/koji

GitLab's test suite requires a running MySQL server instance, which means that the test suite can't run inside Koji. We will have to find a way to run the tests as they are the only way to test the package during buildtime. Same applies to other gems that share the same philosophy.

A rough timeline

There are almost 100 gems to be packaged. The concept is to use 2 months at most to finish packaging and then begin testing the whole application on a production server. I estimate to package roughly 2-3 gems/day including their bugzilla submission.

May 27 - June 17

During the bonding period I intend to thoroughly read the packaging documentation and familiarize with the tools that will be used throughout the project:

• gem2rpm
• rpm*
• mock
• fedpkg
• koji
• etc

A preliminary gem packaging process with hands-on use of the mentioned tools is also scheduled.

June 18 - August 2

June 18 - 25

• Compile the gem list and sort according to the following table:

 Gem name | Upstream version | GitLab version | Fedora version

That will make matters easier to work afterwards. I will also make sure to include the dependencies of each gem in order to prioritize which gems to build first to avoid conflicts.

• Reach any third party that are packaging for Fedora/EPEL (eg. Kastello people) and see if any of GitLab's gems are already packaged. Kastello has many packages ready for EPEL6, so it will be helpful later when packaging for EPEL begins.

June 26 - July 29

• Start packaging and submit to bugzilla.
• Test that the packages work in Rawhide.
• Make any corrections needed and repeat steps.

Incremental gem testing: For every packaged gem, test it against GitLab and ensure that the app works just like when using the unpackaged version.

After those 4 weeks, the packaging process should be around 60-70%.

July 29 - August 2

Wrap up things and get ready for the midterm evaluation. Modify current objectives based on what achieved so far.

August 3 - September 16

August 3 - 18

• Package remaining gems.

August 19 - 31

• Package GitLab application.
• Test against MariaDB.
• Get systemd service files ready.
• Get a test repo ready, since package review may take some time and packages may not yet be in Rawhide.
• Deploy in a production server (talk to infra team) and test current version of GitLab. May need to package some previous version if bugs causing malfunction detected.
• Look for brave Fedorian testers.

September 1 - 16

Should everything be up and running, next step is to get ready for the EPEL branch. This should't take much time as the specfiles would be ready and need minor changes.

• Make sure to contact any third party again and work together during the packaging process.
• GitLab needs some newer versions of basic packages that may not be included in CentOS/RHEL (eg. git, redis-server). We should find a way to address these problems.
• Deploy in a production server and test GitLab. Which version should be decided upon upstream stability.

September 23 - 27

At this point everything should be already packaged, tested and deployed on a production server. Four days are enough to submit final evaluations to Google.

Longterm commitment

It goes without saying that without longterm support the whole packaging process would have no meaning.

Team of maintainers

Maintaining almost 100 packages by oneself is quite a task, so there must be formed some sort of a team willing to take over some packages and help with the maintenance.

Nice to have

Puppet/Chef

Easy deployment on multiple servers is something a sysadmin would like to have. As far as GitLab is concerned, there are two puppet modules actively developed (unofficially), whereas there is a suggestion by a GitLab developer for an officially supported Cookbook.

Fedora openID

Right now GitLab provides signup/login through github/twitter/google auth accounts. It would be nice to see signups using Fedora's openID. This is a feature that could be implemented.

Other details

GitLab supports Ruby version >= 1.9.3 and the new 2.0, so there is no problem regarding the upcoming inclusion of ruby 2.0 in Fedora.

Next Steps

The actual idea (also proposed for GSoC 2012) is to package GitLab and then deploy it as a git service on fedorahosted. After talking to the infra team, it became clear that they first needed to see it packaged and then start deploying it. Another show-stopper was the lack of an important feature: public browserability. This is a long awaited and greatly requested feature that GitLab devs decided to implement just recently (May 1st). So, when this gets implemented, be it in 4,6,8 months, I would be more than happy to get involved and help the infra team deploy it for fedorahosted.