Dist Git Proposal

This proposal has been accepted by FESCo and implemented by Release Engineering. This page is a historical reference.

Overview

This proposal concerns the desire to replace Fedora's package source control, currently based on CVS, with git, and to improve the developer workflow at the same time.

Problem Space

Current package source control is in CVS, and designed in a way that can make the developer's life difficult. See Infrastructure/VersionControl#Current_Pain_Points for a list of current pain points with our package source control. See also Jesse Keating's FUDCon presentation on the issue.

Proposed Solution

We propose to convert the current package source control repositories into git format, turning the CVS release subdirs into git branches. We also propose to replace the Make and common/ dir system with a userland utility.

Scope

• CVS
• git
• fedpkg
• Koji
• Packaging workflow
• Wiki Pages

CVS

All available history from Fedora dist-cvs (which goes back to F-7) will need to be imported. Currently we are planning to use parsecvs in order to import the history. This requires read access to the ",V" files which live on the CVS server. CVS commits must be disabled prior to the final import run to prevent any lost changes.

git

Git is where the CVS history will wind up. As mentioned, parsecvs will be used to get content into the git repo. The layout of the git repos will be:

• One git repo per package
• Development (rawhide) happens on origin/master
• A ref alias 'origin/devel' which points to 'origin/master' to closer match what naming convention CVS used
• Remote branches for each Fedora/EPEL release (F-12, F-11, EL-5...)
• Commit ACLs per branch
• Builds come from commit hashes which are immutable, rather than tags based on n-v-r
• (Immutable) Tags applied to commit hashes after successful builds
• No more common/
• No more Makefile(s)
• No more branch files
• Optional "patches" file to list patches used, to be %included in .spec files

fedpkg

fedpkg is the working name for a new set of tools to replace the existing Make system we have in our package source control. Make is overly complicated for the type of things we wish to do, and prohibitive to adding features/functionality. In our re-design of the package source control we plan to replace Make with a userland application called fedpkg, which may be part of the fedora-packager package. The feature set includes, but is not limited to:

• Duplicate existing Make targets
• Provide utility to check out all active branches of a package in classic dist-cvs view with subfolders for the branches
• Determine branch and thus koji target by git repo context
• Provide exploded code view with git metadata to enable easy patch management (not required for initial roll out)
• Provide utility to clone/checkout from upstream source repo in order to rebase/send patches upstream easily (not required for initial roll out)
• Provide utility to locate/import new upstream source releases (not required for initial roll out)
• Provide utility to check out all packages of a given branch (not required for initial roll out)
• Provide utility to init a local repo that matches remote repo layout and contents for dealing with in-review packages (not required for initial roll out)

koji

Koji will have to learn how to operate the new source control:

• How to clone and checkout to the provided hash sum
• How to create an srpm from that checkout
• How to prevent builds of hashes for the wrong target (eg don't allow a build in dist-f12-updates-candidate from a hash that doesn't exist in the F-12 branch)

Packaging Workflow

As noted above, the packaging workflow will have some subtle changes. Maintainers will no longer tag the source archive before they attempt a build. Builds can be done using a given git commit hash as the reference point. This will allow maintainers to attempt builds multiple times, fixing things as they go, without having to continuously bump n-v-r and enter complete new %changelog sets. Once a build has been successful, an external tool would apply a n-v-r tag to the hash string used to build.

Another change to packaging workflow could be to patch management. With fedpkg's ability to provide an exploded view of either an upstream clone, or a unpack and git init of the upstream tarball, plus our patches applied on top as git commits, we can now use an intelligent patch management system to manage the changes we make to upstream code. To facilitate a better work flow, packages with patches should convert to a standard where each patch is the output of 'git format-patch'. This allows the patches to have useful context around the change itself, rather than just a raw diff. This also allows us to export all the changes we've made to upstream source as a series of format-patch files and write them to the above mentioned patches file automatically. This coupled with rpm .spec file syntax to include external files, and the ability to loop over all defined patches with %patches, we can now remove the overhead of remembering to add patches to the source repo, define the patch in the .spec file, and later apply the patch in the .spec file, thus removing one of the most common mistakes made by our maintainers.

Because the patch management is done outside the package repo, our patches can be added/removed/altered with impunity and we can maintain as small of a patch set as possible, instead of having an ever increasing number of patches as earlier patches are either modified or removed. If they were all done in the same repo, we'd have to add a patch to modify an earlier patch, and even add a patch to remove an earlier patch, which means we'd only ever increase the patch count, rather than reduce it.

This patch management would be optional rather than forced, but we would strongly encourage any patches with modifications to use this new system.

Wiki Pages

Maintain a list of wiki pages here that will need modification when the conversion is made.

Discussion Points

Splitting our .spec file into multiple files doesn't sit well with everybody, this part of patch management will have to be discussed further. However it is not a requirement to completing the conversion, particularly because it is an optional utility.

Comments?

To leave a comment, use the Talk page for this proposal.

Work in progress

Dist_Git_Project is tracking the work in progress of this proposal.