Infrastructure/VersionControl/dist-hg

Some notes regarding using mercurial as a replacement for CVS

Testing dist-hg

We are now ready for testers to take part in dist-hg evaluation. Please let JesseKeating know if you're going to start testing.

Tools Needed

Install mercurial and plague-client from Fedora Extras. Next get a clone of disthg-utils from the test server

hg clone http://publictest1.fedora.redhat.com/disthg-utils

Put the get-fedora-package script somewhere in your $PATH

Download files from http://people.redhat.com/jkeating/dist-hg/ and put them in your home directory with a '.' in front of them.

NOTE
Be sure to backup your existing .plague-client.cfg file for using Fedora Extras. A new file is needed to test dist-hg.

Then email JesseKeating for an account on the build system.

How to test

Use get-fedora-package to fetch a patch checkout from the dist-hg server. We'll use mock for an example:

This should populate a mock/ directory tree in your current working directory. Currently the only configured build target is fedora 6, so building from the FC-6 branch.

Here you can make changes to the spec file, bump the revision, etc... when you're ready to commit and push changes to the server:

Next, make tag and make build will send the job off to the build system. plague-client detail <jobid> should give you status information of your build.

Currently the plague instance will not email you anything. If somebody wants to configure that part of it, please let me know.

---

Conversion

Tools Used

To convert dist-cvs to HG repositories, I make use of:

• cvsps - In Extras.
• cvs-import - http://hg.beekhof.net/hg/cvs-import

Layout

The current layout I'm using is rougly the same as dist-cvs: <packagename>/<release-dirs>. I've omitted the rpms/ dir for now, as it may become an alias to getting all rpms. With CVS, the package dir was basically the module, and each release directory was just a directory, even though we call it a 'branch'. For HG, I've made it so that each release directory is it's own repo. This allows for "branch" confined logging and history, as well as making it easier to setup ACLs. Since hg doesn't allow a repo of repos very well (forest plugin didn't pan out), each package 'module' is a repo itself holding a file named 'branches' that is a list of all the release branches that exist for that package.

How to Convert

I had /cvs/extras restored from backup to the xen host under /srv/extras.

The conversion tools need both the CVSROOT __and__ a checkout of the content to do proper conversion. I setup a directory structure as such:

dist-cvs/
dist-hg/
checkout/
cvs-import/

• dist-cvs: holds CVSROOT and the rpms/<package> trees from the CVS server
• dist-hg: landing zone for converted modules
• checkout: scratch space used for cvs checkouts and conversion work files
• cvs-import: hg checkout of the cvs-import utility

Basic workflow:

cd checkout
cvs -d ~/dist-cvs co rpms/<packagename>
cd rpms/<packagename>/<releasedir>
TZ=UTC cvsps -u -A -q > ../../../cvsps.dump
cd ../../../
mkdir -p ~/dist-hg/<packagename>/<releasedir>
TZ=UTC ../cvs-import/hg-cvs-import ~/dist-cvs/ rpms/<packagename>/<releasedir> ~/dist-hg/<packagename> mapfile < cvsps.dump
mv ~/dist-hg/<packagename>/<releasedir>-HEAD/.hg* ~/dist-hg/<packagename>/<releasedir>/
echo "<releasedir>" >> ~/dist-hg/<packagename>/branches
rm -rf ~/dist-hg/<packagename>-HEAD*
rm mapfile

One would loop through each releasedir of each package (or at least devel/ FC-6/ FC-5/ FC-4/ FC-3/), hg init the <packagedir> and hg add/commit the branches file. I've converted all of extras package CVS as well as the common module.

Script used to convert:

for dir in <code>ls /cvs/extras/rpms/</code>;
do mkdir /srv/hg/extras/${dir};
cvs -d /cvs/extras/ co rpms/${dir};
for branch in rpms/${dir}/{FC-[3456] ,devel};
do
echo <code>basename ${branch}</code> >> /srv/hg/extras/${dir}/branches
pushd ${branch};
TZ=UTC cvsps -u -A -q > ../../../cvsps.dump;
popd;
TZ=UTC ~/sources/cvs-import/hg-cvs-import /cvs/extras/ ${branch} /srv/hg/extras/${dir} mapfile < cvsps.dump;
mkdir /srv/hg/extras/${dir}/<code>basename ${branch}</code>;
mv /srv/hg/extras/${dir}-HEAD/.hg* /srv/hg/extras/${dir}/<code>basename ${branch}</code>/;
rm -rf /srv/hg/extras/${dir}-*;
rm mapfile cvsps.dump;
done;
pushd /srv/hg/extras/${dir};
hg init;
hg add branches;
hg commit -m 'add the branches file';
rm branches;
popd;
rm -rf rpms/${dir}
done

Time needed to convert:

real    155m0.948s
user    84m18.080s
sys     41m5.246s

Space after conversion:

1.8G /srv/hg/extras

---

Server Setup

File system and Permissions

Mercurial can be accessed via http or ssh. In my test server, I created /srv/hg to serve up the repositories. I created a group 'fedora' and set group sticky and write bits on /srv/hg.

Web Configuration

HG web configuration is fairly simple. I've setup /srv/hg/ to be a hgwebdir, in that the repos within /srv/hg can be listed and browsed via http. This is controlled by two files, /srv/hg/hgwebdir.cgi and /srv/hg/hgweb.config. hgwebdir.cgi is a python cgi that calls hg to serve up the repos. hgweb.config is a mapping of names to actual repos paths. Currently I'm listing everything in /srv/hg as a repo by the directory name it has. This gets a bit noisy as each <package>/<releasedir> is listed. URL to view is http://publictest1.fedora.redhat.com One would assume that eventually this would get aliased as say hg.fedoraproject.org/fedora/ or packages/ or something similar.

---

Using dist-hg

Anonymous Access

Anonymous access is achieved via the http interface.

hg clone http://publictest1.fedora.redhat.com/extras/mock/fc-6 mock-fc-6

This would "clone" the FC-6 "branch" of mock as the local 'mock-fc-6' directory. This repo can then be used locally.

Write Access

Write or Commit access is achieved via ssh.

hg clone ssh://publictest1.fedora.redhat.com//srv/hg/extras/mock/fc-6 mock-fc-6

This would "clone" the FC-6 "branch" of mock as the local 'mock-fc-6' directory. This repo can be used locally until changes are happy, then 'hg push' would commit changes back to the server.

Note
the double slash before rpms is intentional. This tells hg to use an absolute path on the ssh server rather than a relative ( to your homedir ) path.

To checkout a package and all of its branches, I have cobbled together a method that uses a 'branches' file to iterate over. In <packagename> there is branches that has one line per release "branch". This script will pull down the set:

#!/bin/bash
#

HGSERVER="ssh://publictest1.fedora.redhat.com//srv/hg/extras"

hg clone ${HGSERVER}/${1}

for branch in <code>cat ${1}/branches</code>
do
hg clone ${HGSERVER}/${1}/${branch} ${1}/${branch}
done

hg clone ${HGSERVER}/common ${1}/common

Obviously this is a bit rough, but it helps to do the proof-of-concept. Eventually a plugin or something could be written for hg that will do this automatically. Call it with the package name as an argument. This also pulls down the common repo. Eventually script could be modified to take a list of 'branches' to pull down as well, instead of all.

---

Work Needed

These are some of the high level items that need to be investigated or implimented. If you wish to help, please contact JesseKeating.

• Integration of mercurial plugin for a repo of repos - hgforest

Seems a bust right now. Doesn't integrate into web view, nor seems to work with remote cloning.

• Makefile changes for hg commands

Most make targets have been converted and tested.

• Tagging hook to ensure unique tags across an entire <package>

Currently done for cvs with /cvs/extras/CVSROOT/tag-check for CVS.

• ACL framework
• Updated scripts for importing srpms (creating new <packages>)
• Updated scripts for "branching" for a release
• Integration of email notification scripts Notify Extension
• Modify plague to be able to pull an hg tag to build from.

Most plague work done, able to pull and build a package from dist-hg

• Configure plague to send emails regarding job completion/failure