Garbage Collection in Koji
By garbage collection, we mean removing builds from the system that are no longer needed. While it would be nice to keep everything forever, we have a finite amount of space and the rpms pile up quickly. While the current implementation focuses on removing builds, we may in the future perform other sorts of cleaning.
The intent of garbage collection is to be as careful as possible and only remove builds that truly are no longer needed. The process happens in stages and a notification email is sent to the build owner when a build is marked for deletion.
Stages
The three stages of garbage collection are:
- pruning
- obsoleted builds are untagged according to policies
- trashing
- untagged/unreferenced builds are placed in the trashcan
- deleting
- builds in the trashcan are deleted
You might envision a build's "life-cycle" as follows:
tagged -> untagged -> in_trashcan -> deleted
Pruning
In the pruning step, unneeded builds are automatically removed from certain tags according to a set of policies. These policies are robust and allow rules based on tag, package, age (within tag), order (within tag), and signatures. More details on this are presented below.
Because of the policies, the pruning step can be made to behave in different ways. However, the overall intent is to remove old builds from tags. Generally policies will be along the lines of "keep the latest three builds of each package in such-and-such tag."
Please note that pruning does not delete builds, it simply untags them (though this may eventually lead to their deletion via the other stages). Since a build can be multiply tagged, it may be untagged from one tag but remain in another.
Trashing
Trashing a build simply means tagging it with the special 'trashcan' tag. We say that such a build is 'marked for deletion' or 'has been placed in the trashcan.' The point of this step is to provide a safety net and give the build a chance to be salvaged if need be. The garbage collector will only place builds in the trashcan if it satisfies these basic requirements:
- the build is untagged, and has been untagged for at least 5 days (actual delay configurable)
- the build is not signed with a protected key
- the build has not been used in the buildroot of another completed build
- the build has not been used in any buildroot for 5 days (same configurable delay as above)
If a build satisfies these conditions, then it will be placed in the special trashcan tag and an email notification is sent to the build owner.
Deleting
In the deleting step, each build in the trashcan is examined. If it is still eligible for deletion and has been in the trashcan longer than the grace period (4 weeks, configurable), it is deleted. If it is not eligible (e.g. it has been tagged elsewhere or somehow acquired a protected signature), then it is removed from trashcan tag (this process is called salvage).
When a build is actually deleted, the files are removed from disc and some (but not all) of the data about the build is removed from the db. The residual data is quite small, though. In particular, the build entry and rpm entries are still present. This prevents reuse of the nvr or nvras.
Restoring
If a build is marked for deletion, because pushing the build to Bodhi was forgotten for example, it can be restored.
Use koji CLI to tag the build with the required tag. For example, for a build for a current Fedora release that was not pushed in Bodhi:
koji tag-build f37-candidate golang-uber-ratelimit-0.2.0-1.fc37
How to protect a build
If a build is marked for deletion, the odds are that this is the correct thing to do. However, it is possible that a build was mistakenly untagged. If you believe this is the case, the fix is to make sure the build is properly tagged. If you are not sure which tag is the correct one, ask a release engineering ( file an issue at: https://pagure.io/releng/ ).
More about pruning policies
The pruning policy is a series of rules. During pruning, the garbage collector goes through each tag in the system and considers its contents. For each build within the tag, it goes through the pruning rules until it finds one that matches. It it does, it takes that action for it.
In the policy configuration, each line is a rule, and the first matching rule wins. The format is:
test <args> [ && test <args> && ...] :: action
The available tests are:
- tag
- the name of the tag must match one of the patterns
- package
- the name of the package must match one of the patterns
- age
- a comparison against the length of time since the build was tagged. This is not the same as the age of the build.
- sig
- true if any of the build's component rpms are signed with a matching key
- order
- a comparison against the order number of the build within a given tag. The order number is the number of more ; recently tagged builds for the same package within the tag. For example, the latest build of glibc in dist-f8 has order number 0, the next latest has order number 1, and so on. Note that the 'skip' action modifies this -- the build is kept, but is not counted for ordering.
Note that the tests are not being applied to just a build, but to a build within a tag. If a build is multiply tagged, it will be checked against these policies for each tag and may be kept in some but untagged in others.
The available actions are:
- keep
- do not untag the build from this tag.
- untag
- untag the build from this tag
- skip
- like keep, but do not count the build for ordering
Note that, regardless of any policies, locked tags are left alone.
At present, the pruning policy is:
tag *-updates :: keep age < 1 day :: skip sig fedora-gold :: skip sig fedora-test && age < 12 weeks :: keep tag *-testing *-candidate && order >= 2 :: untag tag *-testing *-candidate && order > 0 && age > 6 weeks :: untag tag *-candidate && age > 60 weeks :: untag order > 2 :: untag
If you think the policy needs to be adjusted, please notify a koji admin.