Dist Git Branch Proposal

This page is to track a proposal on changing how we have our branches structured currently in dist-git

Email Proposal

I'm working on fixing a few long standing bugs in fedpkg that have to do
with our branch structure
(https://bugzilla.redhat.com/show_bug.cgi?id=619979 and
https://bugzilla.redhat.com/show_bug.cgi?id=622592).

This has me examining our branch structure again and trying to remember
why exactly I chose it (obviously I did a poor job of documenting that...).

The original thought was to have top level branches that are named after
distribution releases, eg "f14", "f15", "el5".  Then we would force
branches of those branches use a naming structure of "f14/topic".  The
reason was so that our tooling could look at the name of the branch and
easily work back to the "f14" part.  This would work even if it was
"f14/user/fred/topic/mybranch"  or other such craziness.  When I went to
test this, I realized that git won't allow you to have both "f14" and
"f14/topic" as branches, because of the way the git metadata works on
the filesystem.  When I encountered this, I made "f14/master" become the
top level branch, and then "f14/somethingelse" could coexist.
Unfortunately I also wanted to keep things easy for users and tried to
maintain tooling that would allow you to just say "f14".  This didn't
get enough real world testing and in hindsight was a bad idea.  Things
go wrong quickly in git if your local branch name doesn't match the
remote branch name.

When thinking about the above, and the two bugs I'm working on, I
realized that we don't have any real strong need to be using "/" as a
delineator.  It makes some code easier, but makes other things more
complex and difficult.  So what if we changed it?

What I'm thinking about now is switching from / to - as a delineator.
This would improve a couple things.  First, we could achieve upstream
top level branch names that are short and simple: "f14", "f15",
"master".  We can have branches that build upon those names:
"f14-rebase", "f15-cve223", "f15-user-jkeating-private".  We could keep
the simple fedpkg tooling that allows users to just type "f14" and the
like to reference a branch, and now the local branch will match the name
of the remote branch.

As for the first two bugs I mentioned, it doesn't directly help them.
However I would feel better about telling people that their local
branches must follow a naming scheme of <release>-<something> and then
we could easily guess what release the local branch is for if it isn't
tracking a remote branch.  However the bug about what to do if there are
no remote branches is really not touched by any of this, it just got me
thinking about branches :)

What kind of user impact would this have?

My hope is that the impact would be minimal.  Git allows branch renames,
and can successfully rename "f14/master" to "f14".  All the history is
renamed.  We should be able to do this without an outage of the git server.

The ACL system will need a slight tweaking, and a regeneration of the
ACLs but that is fairly quick and minor to accomplish.

However there will be some issues client side.  We will not be able to
make use of git's symbolic-ref feature of "aliasing" a branch.  We
cannot make "f14/master" an alias for "f14", again because of the
filesystem layout issues.  These issues rear their head once again when
a client does a pull of an already checked out repo that had branches.
Because there was already a f14/master, when the client sees a new
branch just named "f14" it will fail to create it.  Git has a command
that will fix this "git remote prune origin".  That will remove the
local reference to f14/master and a subsequent pull sees the creation of
the "f14" branch happen successfully.  However, if a user had a local
branch of f14 or f14/master they will be left with mismatched
.git/config entries.  In this case it's easiest to delete the local
branch (git branch -d f14) and check it out again.  If there are changes
on the branch one can fix the config to point it to the right upstream
location.

Also we would need to get a new fedpkg into the hands of all the
developers that handles the new branchnames.  We could do a build that
handles both the oldnames and the new and have it out and available for
a reasonable period of time before we make the switch.

So, some pain, for some pretty good gain.  This time around I can setup
pkgs.stg with this branch configuration and builds of fedpkg to use it
for a more through testing before rolling it into production.

So please, tell me what you think!

Impact

This change will impact a few things.

Git Server

Various changes would happen on the git server

The repos

The git server is where we would process all the rename jobs. A script needs to be written in order to discover and rename the branches appropriately.

Some care will have to be taken when there exists release branches beyond f14/master. The other branches will need to get renamed to f14-<branch> before finally renaming f14/master to f14.

The ACL system

The ACL system will need to be tweaked to remove the trailing "/" from branch path names in the ACL files.

We can also modify the ACL system to detect push attempts that include a path with "<branch>/<something>" and instead of generically erroring out to the user we can print a link to a wiki page explaining the migration and what the end user must do.

Branch Creation Scripts

There are a couple branch creation scripts that will need to be updated to reflect no longer having "/master" in branch names.

fedpkg

The branch naming comes up surprisingly few times in fedpkg code. It is fairly trivial to go from "<foo>/master" to just "<foo>". The harder part will be putting in code that handles both scenarios at the same time, so that we can roll the new fedpkg into place well ahead of the git server work.

Testing

Currently we are using pkgs01.stg.phx2.fedoraproject.org as a testing environment for changing the various server side elements listed above with client side changes. At this time it is not appropriate for public testing, nor do we have a fedpkg build for client testing.

The test plan would be to sync pkgs.stg.fedoraprojec.org with some of the production pkgs.fedoraproject.org and then convert all the branches and scripts on pkgs.stg. Then provide a build of fedpkg that is configured to use this host and configured to use the new branch names to interested testers. Feedback will be processed and things potentially tweaked until we feel that the setup is sufficient.

Timeline

We would like to complete the migration prior to or just after FUDCon Tempe 2011, which means late January 2011. Date subject to change as this has not been proposed to FESCo yet.

Here is a quick todo list:

• Investigate changes requires a bit more using pkgs.stg -- IN PROGRESS
• Propose change to FESCo -- DONE and APPROVED
• Setup pkgs.stg for widespread testing -- DONE
• Publish a build of fedpkg appropriate for testing -- DONE
• Apply ACL patch to warn users on push attempt to old branch path -- DONE (locally, not in a package)
• Write client side script to update clone metadata -- DONE
• Solicit testing and feedback -- DONE
• Prepare production build of fedpkg -- DONE
• Submit updates of fedpkg -- DONE
• Prepare patch for genacls.sh to support new branch style -- IN PROGRESS
• Schedule time for production changeover of pkgs.fedoraproject.org -- DONE set for May 10
• Execute changeover
• Prepare and publish another build of fedpkg that removes code to deal with old branch style

Changeover Execution

On May 10 we will execute the changeover. Here are the steps involved:

• Start an outage of the git server
• Run rebranch.py
• Update gitolite to our custom build
• Update genacls.pkgdb for new branch style
• Update the reposconverted file on jkeating.fedorapeople.org
• End the outage

Start the Outage

First make sure an outage announcement has been sent!

All pushing makes use of the gitolite ACL system. To enact an outage, gitolite must be configured to deny everything. This can be accomplished by having

repo @pkgs
    R = @all

after the pkgs definition, and nothing else in the gitolite.conf file passed to gl-compile-auth.

There is a cron job that will overwrite this though, thus the cron job must be disabled. As jkeating, run crontab -e and comment out the cron job:

# Puppet Name: genacls
#0,10,20,30,40,50 * * * * /usr/local/bin/genacls.sh &>/dev/null

Puppet will overwrite this though so we must also disable puppet for the duration of the outage. This is accomplished by touching /var/lib/puppet/state/puppetdlock This will prevent any puppet runs from happening.

The outage is now in effect.

DONE

Run rebranch.py

rebranch.py is a script written to walk the directory of git repositories and rename the branches as necessary. This is a multithreaded script although it will be limited by disk I/O. In testing it took roughly 35 minutes to complete. Currently rebranch.py lives on Jesse Keating's laptop, although there is a copy of it on pkgs01.stg. It will be uploaded to jkeating's homedir on pkgs01 before the outage.

Running under the "rebranch" screen of jkeating

Update gitolite

A custom build of gitolite has been created that will watch for push attempts to the old style branch paths and direct users to Dist_Git_Branch_Redux. gitolite-1.5.7-2.1.el6.fedora.noarch.rpm currently exists in the infrastructure staging repo, and may be moved to the production repo during the outage so that it can be installed on pkgs01.

DONE

Update genacls.pkgdb

An update to genacls.pkgdb has been committed to the staging puppet repo. When the outage begins, this change can be cherry-picked to the production branch. However puppet is disabled on pkgs due to the outage. The patch can be manually applied to pkgs in order to generate a new set of acls. Once the patch has been applied to /usr/local/bin/genacls.pkgdb run /usr/local/bin/genacls.sh to generate a new set.

DONE

Repoconverted

There is a file at fedorapeople that tells fedpkg whether or not the repos have been converted. Edit this file to say True once the repos have been converted.

End the outage

The outage can now be ended. Replace gitolite.conf with a backup (or a newly generated copy) and re-run gl-compile-auth. Re-enable the cron job for this. Unlock puppet.

Once these things are done and the system is tested, an email announcing the end of the outage can be sent.

Proposal Owner

Jesse Keating