In the interest of keeping Fedora as a self-hosted distribution (meaning you can use Fedora version Z to build Fedora version Z from source RPMs), Matt Domsch regularly runs a full rebuild of the "rawhide" tree, building rawhide with rawhide. This catches a number of packages that no longer build, and need developer attention. The results of each run are presently mailed to each failing package's owner and cc: list (as noted in the package database), and sent to fedora-devel-list.
In the interest of tracking these failures, new bugs for each failing package will be filed in Bugzilla. These bugs will all block a blocker bug, one per OS version, such as alias "F10FTBFS". Included in these bugs will be the root.log and build.log from mock for architectures where the failures occurred. These bugs are automatically marked ASSIGNED, since they are triaged.
A recursive check will be made that there is not already a bug that's blocking FTBFS for the package in question. If there's not, a new bug will be filed against the package.
Challenges
- Avoiding false positives: It somewhat often happens that a whole class of failures are due to either build system misconfiguration, or mirrors being slightly out of sync.
- Bugs in required packages: If glibc is broken on a particular day, it can affect a large number of package builds. It's most appropriate to file a single bug against glibc in this case rather than many bugs against each package that hit the bug. Unfortunately, figuring this out requires human intervention.
- Being that this is a monthly event, I think that simple sanity checking is really all that's required here - nothing fancy. Rebuild and filing should be two separate phases, so that these issues can be caught.
Process
- File bugs, once for each package.
- Block FTBFS
- FTBFS blocks Target tracker for next release
- Attach root.log and build.log, and mock.log from each architecture that has failed.
- Fedora version = 'rawhide'
- Run monthly
Results
- F9FTBFS
- F10FTBFS
- F11 was handled by Release Engineering, no FTBFS bugs were filed. http://jkeating.fedorapeople.org/needed-f11-rebuilds.html
- F12FTBFS
What to do if you get a FTBFS bug?
- Read the logs.
- If the bug is in your package:
- In your package's devel/ branch in CVS, fix the problems uncovered.
- Commit the changes and build the package. The fixed package will land in rawhide, generally the next day.
- If the build succeeds, close the bug as CLOSED: RAWHIDE, and include the package version number in the comments.
- If the bug is in a different package:
- Find an existing bug for that package describing the problem. Set your bug to "Depends on" that other bug. Do not change the component of your bug to the other package, or you will get more FTBFS bugs created against you.
- When that other bug is closed, you'll get an email from bugzilla as usual. Rebuild your package using a koji scratch build, to verify it builds cleanly again. Close your bug as CLOSED, RAWHIDE, mentioning this.
- If the package should be retired, see Retired Packages
In all cases, if you close an FTBFS bug as a duplicate of another bug, please make the other bug to block 'FTBFS' (bug number 440169). In this way the bug left open will appear in the FTBFS reports properly.
Package Removal for Long-standing FTBFS bugs
Packages with unresolved FTBFS bugs immediately following the Alpha release will be removed from the distribution. Package owners may request that their package _not_ be removed provided they are actively working on resolving the FTBFS and have a plan to resolve the FTBFS before the Release Candidate release.