Some notes regarding using mercurial as a replacement for CVS
Testing dist-hg
We are now ready for testers to take part in dist-hg evaluation. Please let JesseKeating know if you're going to start testing.
Tools Needed
Install mercurial and plague-client from Fedora Extras.
Next get a clone of disthg-utils from the test server
hg clone http://publictest1.fedora.redhat.com/disthg-utils
Put the get-fedora-package script somewhere in your $PATH
Download files from http://people.redhat.com/jkeating/dist-hg/ and put them in your home directory with a '.' in front of them.
Be sure to backup your existing .plague-client.cfg file for using Fedora Extras. A new file is needed to test dist-hg.
Then email JesseKeating for an account on the build system.
How to test
Use get-fedora-package to fetch a patch checkout from the dist-hg server. We'll use mock for an example:
This should populate a mock/ directory tree in your current working directory. Currently the only configured build target is fedora 6, so building from the FC-6 branch.
Here you can make changes to the spec file, bump the revision, etc... when you're ready to commit and push changes to the server:
Next, make tag and make build will send the job off to the build system. plague-client detail <jobid> should give you status information of your build.
Currently the plague instance will not email you anything. If somebody wants to configure that part of it, please let me know.
Conversion
Tools Used
To convert dist-cvs to HG repositories, I make use of:
- cvsps - In Extras.
- cvs-import - http://hg.beekhof.net/hg/cvs-import
Layout
The current layout I'm using is rougly the same as dist-cvs: <packagename>/<release-dirs>. I've omitted the rpms/ dir for now, as it may become an alias to getting all rpms. With CVS, the package dir was basically the module, and each release directory was just a directory, even though we call it a 'branch'. For HG, I've made it so that each release directory is it's own repo. This allows for "branch" confined logging and history, as well as making it easier to setup ACLs. Since hg doesn't allow a repo of repos very well (forest plugin didn't pan out), each package 'module' is a repo itself holding a file named 'branches' that is a list of all the release branches that exist for that package.
How to Convert
I had /cvs/extras restored from backup to the xen host under /srv/extras.
The conversion tools need both the CVSROOT __and__ a checkout of the content to do proper conversion. I setup a directory structure as such:
dist-cvs/ dist-hg/ checkout/ cvs-import/
- dist-cvs: holds CVSROOT and the rpms/<package> trees from the CVS server
- dist-hg: landing zone for converted modules
- checkout: scratch space used for cvs checkouts and conversion work files
- cvs-import: hg checkout of the cvs-import utility
Basic workflow:
cd checkout cvs -d ~/dist-cvs co rpms/<packagename> cd rpms/<packagename>/<releasedir> TZ=UTC cvsps -u -A -q > ../../../cvsps.dump cd ../../../ mkdir -p ~/dist-hg/<packagename>/<releasedir> TZ=UTC ../cvs-import/hg-cvs-import ~/dist-cvs/ rpms/<packagename>/<releasedir> ~/dist-hg/<packagename> mapfile < cvsps.dump mv ~/dist-hg/<packagename>/<releasedir>-HEAD/.hg* ~/dist-hg/<packagename>/<releasedir>/ echo "<releasedir>" >> ~/dist-hg/<packagename>/branches rm -rf ~/dist-hg/<packagename>-HEAD* rm mapfile
One would loop through each releasedir of each package (or at least devel/ FC-6/ FC-5/ FC-4/ FC-3/), hg init the <packagedir> and hg add/commit the branches file. I've converted all of extras package CVS as well as the common module.
Script used to convert:
for dir in <code>ls /cvs/extras/rpms/</code>;
do mkdir /srv/hg/extras/${dir};
cvs -d /cvs/extras/ co rpms/${dir};
for branch in rpms/${dir}/{FC-[3456] ,devel};
do
echo <code>basename ${branch}</code> >> /srv/hg/extras/${dir}/branches
pushd ${branch};
TZ=UTC cvsps -u -A -q > ../../../cvsps.dump;
popd;
TZ=UTC ~/sources/cvs-import/hg-cvs-import /cvs/extras/ ${branch} /srv/hg/extras/${dir} mapfile < cvsps.dump;
mkdir /srv/hg/extras/${dir}/<code>basename ${branch}</code>;
mv /srv/hg/extras/${dir}-HEAD/.hg* /srv/hg/extras/${dir}/<code>basename ${branch}</code>/;
rm -rf /srv/hg/extras/${dir}-*;
rm mapfile cvsps.dump;
done;
pushd /srv/hg/extras/${dir};
hg init;
hg add branches;
hg commit -m 'add the branches file';
rm branches;
popd;
rm -rf rpms/${dir}
done
Time needed to convert:
real 155m0.948s user 84m18.080s sys 41m5.246s
Space after conversion:
1.8G /srv/hg/extras
Server Setup
File system and Permissions
Mercurial can be accessed via http or ssh. In my test server, I created /srv/hg to serve up the repositories. I created a group 'fedora' and set group sticky and write bits on /srv/hg.
Web Configuration
HG web configuration is fairly simple. I've setup /srv/hg/ to be a hgwebdir, in that the repos within /srv/hg can be listed and browsed via http. This is controlled by two files, /srv/hg/hgwebdir.cgi and /srv/hg/hgweb.config. hgwebdir.cgi is a python cgi that calls hg to serve up the repos. hgweb.config is a mapping of names to actual repos paths. Currently I'm listing everything in /srv/hg as a repo by the directory name it has. This gets a bit noisy as each <package>/<releasedir> is listed. URL to view is http://publictest1.fedora.redhat.com One would assume that eventually this would get aliased as say hg.fedoraproject.org/fedora/ or packages/ or something similar.
Using dist-hg
Anonymous Access
Anonymous access is achieved via the http interface.
hg clone http://publictest1.fedora.redhat.com/extras/mock/fc-6 mock-fc-6
This would "clone" the FC-6 "branch" of mock as the local 'mock-fc-6' directory. This repo can then be used locally.
Write Access
Write or Commit access is achieved via ssh.
hg clone ssh://publictest1.fedora.redhat.com//srv/hg/extras/mock/fc-6 mock-fc-6
This would "clone" the FC-6 "branch" of mock as the local 'mock-fc-6' directory. This repo can be used locally until changes are happy, then 'hg push' would commit changes back to the server.
the double slash before rpms is intentional. This tells hg to use an absolute path on the ssh server rather than a relative ( to your homedir ) path.
To checkout a package and all of its branches, I have cobbled together a method that uses a 'branches' file to iterate over. In <packagename> there is branches that has one line per release "branch". This script will pull down the set:
#!/bin/bash
#
HGSERVER="ssh://publictest1.fedora.redhat.com//srv/hg/extras"
hg clone ${HGSERVER}/${1}
for branch in <code>cat ${1}/branches</code>
do
hg clone ${HGSERVER}/${1}/${branch} ${1}/${branch}
done
hg clone ${HGSERVER}/common ${1}/common
Obviously this is a bit rough, but it helps to do the proof-of-concept. Eventually a plugin or something could be written for hg that will do this automatically. Call it with the package name as an argument. This also pulls down the common repo. Eventually script could be modified to take a list of 'branches' to pull down as well, instead of all.
Work Needed
These are some of the high level items that need to be investigated or implimented. If you wish to help, please contact JesseKeating.
- Integration of mercurial plugin for a repo of repos - hgforest
Seems a bust right now. Doesn't integrate into web view, nor seems to work with remote cloning.
- Makefile changes for hg commands
Most make targets have been converted and tested.
- Tagging hook to ensure unique tags across an entire <package>
Currently done for cvs with /cvs/extras/CVSROOT/tag-check for CVS.
- ACL framework
- Updated scripts for importing srpms (creating new <packages>)
- Updated scripts for "branching" for a release
- Integration of email notification scripts Notify Extension
- Modify plague to be able to pull an hg tag to build from.
Most plague work done, able to pull and build a package from dist-hg
- Configure plague to send emails regarding job completion/failure