Rawhide synchronization for the GNU Binutils
This page documents the process by which the rawhide binutils package is synced with the upstream GNU Binutils project. The aim of this process is to make syncing simple and easy so that it can be performed on a regular basis. Due to the fact that the binutils is an important component in the toolchain used to build many of the packages in rawhide however, it is important to make sure that any attempt at syncing does not break anything.
This process is based upon a similar process for the GNU C Library (GlibcRawhideSync), but tweak to fit the binutils package.
TL;DR
Make sure you have authenticated and meet the pre-requisites (see notes below).
If you do this every week, here's the cut and paste snippets you need, minus the bits that change every week...
kinit
export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts
export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts
cd $GLIBC_MS; git pull --rebase
cd $TOOLS; git pull --rebase
cd $HOME/src/glibc-pristine; git pull --rebase
cd $HOME/fedsrc/glibc; git pull --rebase
fedpkg sources
rm -rf glibc-patches; $GLIBC_MS/glibc-patches-to-git.py
$GLIBC_MS/glibc-sync-upstream.py --import-git $HOME/src/glibc-pristine --verbose
git diff
cd $HOME/src/glibc-pristine; git log --oneline (fill)..(in) | sed 's/[^ ]* /- /'
cd $HOME/fedsrc/glibc; ${VISUAL:=vi} glibc.spec
fedpkg new-sources `/bin/ls -1rt glibc-*.tar.xz | tail -1`
git add glibc.spec
git commit
fedpkg srpm
fedpkg scratch-build --srpm `/bin/ls -1rt $PWD/glibc-*.src.rpm | tail -1`
rm -rf logs; mkdir logs; cd logs
$TOOLS/codonell/get-build-logs.sh https://koji.fedoraproject.org/koji/taskinfo?taskID=(fill-in)
cd $HOME/fedsrc/glibc; git push
fedpkg build
https://bodhi.fedoraproject.org/updates/?search=glibc
Synchronization Process
Fedora Rawhide must pass Fedora CI/CD, in particular
baseos-qe.koji-build.scratch-build.validation before you can consider the synchronization process complete.1. Setup/Fetch a clean upstream master repository for glibc. This can be an existing directory also which you switch to master branch and ensure it has been rebased e.g. git pull --rebase cleanly to head.
mkdir -p $HOME/src cd $HOME/src git clone git://sourceware.org/git/glibc.git glibc-pristine
2. Setup/Fetch a clean Fedora rawhide glibc repository. If this is not your first time syncing, you may need to remove the glibc-patches subdirectory that the previous sync created.
cd $HOME/fedsrc fedpkg clone glibc
3. Run fedpkg sources to downloads the .tar.xy file needed for creating the glibc-patches repository (next step).
cd $HOME/fedsrc/glibc fedpkg sources
4. Convert the Fedora rawhide glibc dist-git repository to a glibc-patches repository (git repo with Fedora patches applied as commits).
export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts cd $HOME/fedsrc/glibc $GLIBC_MS/glibc-patches-to-git.py
5. Synchronize from upstream master to Fedora rawhide and review changes. NOTE: Pick the appropriate set of instructions based on where we are in the release.
No release transitions occurring:
cd $HOME/fedsrc/glibc $GLIBC_MS/glibc-sync-upstream.py --import-git $HOME/src/glibc-pristine --verbose git diff Confirm that glibcsrcdir, glibcversion, and Release have the correct NEVRA.
Transitioning from stable release -> master:
cd $HOME/fedsrc/glibc $GLIBC_MS/glibc-sync-upstream.py --import-git $HOME/src/glibc-pristine --verbose --import-git-branch master git diff Manually change the glibcsrcdir, glibcversion, and Release to match the new NEVRA.
If a merge conflict occurs, review the section on Dealing with Merge Conflicts then return to the next step.
6. Manually document note-worthy changes in the %changelog. This step is probably the most complex. You need to look at all the changes since the last sync (the hash is recorded in the %changelog) in the upstream repo and see if there is anything note-worthy to talk about e.g. git diff HASH1^..HASH2. You will use this text in your commit message also.
* Wed Nov 07 2018 Florian Weimer <fweimer@redhat.com> - 2.28.9000-14 - Auto-sync with upstream branch master, commit 1df872fd74f730bcae3df201a229195445d2e18a: - libanl: Fix crash if first helper thread creation failed (#1646381)
A useful command to summarize commits is git log like this:
cd $HOME/src/glibc-pristine git log --oneline e0cb7b6131..0ddb7ea842 | sed 's/^[^ ]* /- /'
The two hashes to use are based on the first ten digits of the last sync's git commit and the current sync's git commit. You want the log listing to show all of the new commits that have been included in the current sync. In the above example, we're updating from e0cb7b6131 (the commit of the last sync) to 0ddb7ea842 (the commit of the current sync). Verify the boundary commits make sense.
7. Add new files (still in the Fedora rawhide directory from the last step) and commit
cd $HOME/fedsrc/glibc fedpkg new-sources glibc-*.tar.xz git add glibc.spec git commit
An appropriate commit message would be:
Auto-sync with upstream branch master
Upstream commit: 1df872fd74f730bcae3df201a229195445d2e18a
- libanl: Fix crash if first helper thread creation failed (#1646381)
Following the spec file %changelog example from above.
8. Test a scratch build, and wait for it to complete.
fedpkg srpm fedpkg scratch-build --srpm ./glibc-XXX.src.rpm
9. Verify scratch build results by downloading logs and looking for any unexpected failures. A list of known failures is kept at the end of this page, and should be updated as needed.
export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts $TOOLS/codonell/get-build-logs.sh https://koji.fedoraproject.org/koji/taskinfo?taskID=$TASKID
10. Push the commit and kick off a Rawhide build if the logs look good (you are not done yet!)
git push fedpkg build
11. Fedora CI/CD must run and you must verify that baseos-qe.koji-build.scratch-build.validation succeeded. If Fedora CI/CD does not succeed please reach out to Martin Cermak (BaseOS QE). To verify the status of CI/CD go to Bodhi after your build successfully completes, select Search..., type in glibc, hit enter and pick the newest Rawhide build errata that was automatically created and select the Automated Tests tab. If baseos-qe.koji-build.scratch-build.validation is green then the test has passed and you are complete with the sync.
Authentication
1. Kerberos init with Fedora realm.
2. Make sure your ssh agent has your Fedora ssh key for pagure.io.
Pre-requisites
1. You have a Fedora account with an SSH key, and have setup your SSH key on pagure.io by logging in.
2. Install fedpkg, python3-pygit2, python3-rpm and git-merge-changelog
3. Configure your ~/.gitconfig to have entries for name, and a merge driver for GNU Changelogs.
[user]
name = My Name
[merge "merge-changelog"]
name = GNU-style ChangeLog Merge driver
driver = /usr/bin/git-merge-changelog
[core]
attributesfile = ~/.gitattributes
4. Configure your ~/.gitattributes
ChangeLog merge=merge-changelog
5. Clone glibc-maintainer-scripts into $GLIBC_MS for the git synchronization scripts.
export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts mkdir -p $HOME/fedsrc cd $HOME/fedsrc git clone https://pagure.io/glibc-maintainer-scripts.git
6. Clone UpstreamToolchainBuildScripts from the Fedora Toolchain team for the build log fetching scripts.
export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts cd $HOME/fedsrc git clone https://pagure.io/FedoraToolchainTeam/UpstreamToolchainBuildScripts.git
Note: if you plan on editing either of the two git repos you just installed, use ssh://git@pagure.io instead of https://pagure.io
Differences for Release Syncs
For syncing a glibc release branch into rawhide, you need to use a command line this, and manually reset the Release number the first time:
$GLIBC_MS/glibc-sync-upstream.py --import-git-branch [Release branch you are using e.g. release/2.31/master] --import-git [Path to your updated checked out glibc tree e.g. $HOME/src/glibc-2.31] --verbose
For Fedora release branches, the process is basically the same, except you have to use the Bodhi system to push your build to a release.
- In the above instructions, use the release branch of glibc (typically like release/2.28/master) and the release branch of Fedora (typically like f29)
- Click the ? icon in the upper right, to search for "glibc"
- Note previous updates for wordings and severities
- Select "Create" in the upper right and create a new update
- Type "glibc" in the packages field. If there's a popup, select the appropriate entry.
- type "glibc" in the candidate builds field, select the relevent build from the popup.
- Click in the "Related bugs" field. If your BZ doesn't show up, type in the number.
- Fill in the Update Notes field and final details.
- Submit! The system will advertise the update and request karma. When the update has enough karma, it's automatically pushed out.
- This is an example of a Fedora 39 sync to release/2.38/master. It assumes you have done at least one sync previously, have completed the pre-requisites, and cloned the source files.
kinit
export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts
export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts
cd $GLIBC_MS; git pull --rebase
cd $TOOLS; git pull --rebase
cd $HOME/src/glibc-pristine; git pull --rebase
* Checkout the version you are syncing to.
git checkout release/2.38/master
cd $HOME/fedsrc/glibc; git pull --rebase
* Checkout the version of Fedora that you are syncing.
git checkout f39
fedpkg sources
rm -rf glibc-patches; $GLIBC_MS/glibc-patches-to-git.py
* Specify --import-git-branch for the correct release.
$GLIBC_MS/glibc-sync-upstream.py --import-git-branch release/2.38/master --import-git $HOME/src/glibc-pristine --verbose
git diff
cd $HOME/src/glibc-pristine; git log --oneline (fill)..(in) | sed 's/[^ ]* /- /'
cd $HOME/fedsrc/glibc; ${VISUAL:=vi} glibc.spec
fedpkg new-sources `/bin/ls -1rt glibc-*.tar.xz | tail -1`
git add glibc.spec
git commit
fedpkg srpm
fedpkg scratch-build --srpm `/bin/ls -1rt $PWD/glibc-*.src.rpm | tail -1`
rm -rf logs; mkdir logs; cd logs
$TOOLS/codonell/get-build-logs.sh https://koji.fedoraproject.org/koji/taskinfo?taskID=(fill-in)
cd $HOME/fedsrc/glibc; git push
fedpkg build
* Create the bodhi update. You can use a previous update as an example:
https://bodhi.fedoraproject.org/updates/?search=glibc
Dealing with Merge Conflicts
- Dealing with merge conflicts due to backport+rebase:
Sometimes, a fix is backported from upstream into rawhide, and subsequently a rebase is scheduled. During this rebase, the backport and its corresponding patch need to be dropped.
One way to achieve this is to manually edit the spec file prior to running glibc-patches-to-git so that the merge conflict is avoided in the first place.
Another way is to let the merge conflict occur, then run git rebase --skip in the glibc-patches directory/repository to skip the application of the current patch (i.e. the one backported, which is causing the merge conflict) and continue the rebase (this rebase is an internal detail of how glibc-sync-upstream is implemented and was started by the script before erroring out due to the conflict).
Then we continue the process by executing glibc-git-to-patches in $HOME/fedsrc/glibc, which now skips the patch when creating glibc-patches.
Finally, we run glibc-patches-to-git --branch master, to bring update dist-git with the recently refreshed glibc-patches.
- Dealing with Fedora patches that no longer apply cleanly due to changes in rawhide:
Another merge conflict scenario is when a Fedora patch does not apply cleanly. You may see a message like this:
CONFLICT (content): Merge conflict in nss/nsswitch.conf error: Failed to merge in the changes.
In this case, the patch is needed for Fedora but an upstream change has caused the patch to be out of sync. You need to fix the patch in the glibc-patches directory. Using nss/nsswitch.conf as an example:
cd $HOME/fedsrc/glibc/glibc-patches edit nss/nsswitch.conf to fix the merge conflict git add nss/nsswitch.conf git rebase --continue cd $HOME/fedsrc/glibc
If you see that there is a glibc.spec and glibc.spec.new
mv glibc.spec glibc.spec.old mv glibc.spec.new glibc.spec
Now complete the git-to-patches process, regenerating the patch that failed previously:
$GLIBC_MS/glibc-git-to-patches.py -v
You should see a message similar to this:
info: regenerating patch 'glibc-fedora-nsswitch.patch' because patch failed
git add glibc-fedora-nsswitch.patch
Occasions that require special handling
- The addition of a new locale upstream:
When a new locale is added upstream, the change corresponds to a new line in localedata/SUPPORTED and glibc.spec needs to be adjusted to tackle this. The spec file includes a lua script containing a list of supported locales that is compared at build time to the list upstream. This is below the definition of %package locale-source. Grep for ^local locales = to locate the list and add a line corresponding to the new locale there before building.
Known Acceptable Build Failures
For the purposes of "looking for any unexpected failures" (above), this is the current list of build failures that have been deemed "not to stop a sync". After you complete the scratch build and download the logs, compare the test failures in the logs with the results below. For each failure not listed here, analyze it, determine if it's safe to continue syncing, and add that failure below. For each failure listed below that no longer appears in the test results, remove it from below. I.e. most syncs will require minor updates in this section.
All Hosts
There is a known misc/tst-ttyname failure that needs waiving (https://bugzilla.redhat.com/show_bug.cgi?id=2210335) since we do not currently believe this is a glibc regression - per codonell.
aarch64
Rawhide (1 fail): FAIL: misc/tst-mount FAIL: elf/tst-debug1 =====FAIL: elf/tst-debug1.out===== Didn't expect signal from child: got `Bus error' =====FAIL: elf/tst-debug1.test-result===== FAIL: elf/tst-debug1 original exit status 1 =====FAIL: dlfcn/tst-dlinfo-phdr===== see upstream bz 23293
armv7hl
Rawhide (1 fail): FAIL: debug/tst-longjmp_chk2 =====FAIL: debug/tst-longjmp_chk2.out===== not on alternate stack in signal handler on alternate stack Didn't expect signal from child: got `Aborted' =====FAIL: debug/tst-longjmp_chk2.test-result===== FAIL: debug/tst-longjmp_chk2 original exit status 1 FAIL: misc/tst-sysvshm-linux
The tst-sysvshm-linux failure is related to: https://sourceware.org/bugzilla/show_bug.cgi?id=26862:
i686
Rawhide (1 fail): FAIL: misc/tst-mount
This seems to be transient, unable to reproduce manually:
=====FAIL: io/tst-stat-time64.out===== tst-stat-time64.c:88: numeric comparison failure left: 8 (0x8); from: stx.stx_blocks right: 16 (0x10); from: st.st_blocks error: 1 test failures
FAIL: math/test-double-erfc FAIL: math/test-double-tgamma FAIL: math/test-float32x-erfc FAIL: math/test-float32x-tgamma FAIL: math/test-float64-erfc FAIL: math/test-float64-tgamma FAIL: math/test-float-hypot FAIL: math/test-float32-hypot These relate to this bug: https://sourceware.org/bugzilla/show_bug.cgi?id=20305 FAIL: signal/tst-minsigstksz-1 FAIL: signal/tst-minsigstksz-2
ppc64le
FAIL: math/test-float-nearbyint FAIL: math/test-float-rint FAIL: math/test-float32-nearbyint FAIL: math/test-float32-rint Rawhide (2 fails) FAIL: math/test-ibm128-llround FAIL: math/test-ibm128-y1 Previous failures - no longer seeing these: FAIL: misc/tst-mount FAIL: misc/tst-sigcontext-get_pc Sporadic failure: FAIL: time/tst-cpuclock1
s390x
Rawhide (1 fail): FAIL: misc/tst-mount Sporadic failure: FAIL: rt/tst-cpuclock2
x86_64
Rawhide (1 fail): FAIL: misc/tst-mount =====FAIL: nptl/tst-stack4.out===== =====FAIL: nptl/tst-stack4.test-result===== FAIL: nptl/tst-stack4 original exit status 127
Known CI Failures
Please list known CI failures here. CI failures must be coordinated with BaseOS QE.
fedora-ci.koji-build.rpmdeplint.functional
glibc-devel-2.36.9000-21.fc38.aarch64 - testout.log glibc-devel-2.36.9000-21.fc38.aarch64 provides /usr/include/bits/sigaction.h which is also provided by glibc-headers-s390-2.36.9000-21.fc38.noarch The above is a known-failure because not all headers can be installed at the same time. The CI test is attempting to install every package we can install and that's not a valid test of anything. It is not valid to install the aarch64 headers and s390x headers at the same time. The test doesn't know this. Nobody would every need to do this. But it attempts all noarch package installs.
baseos-qe.koji-build.scratch-build.validation
===i686 kernel scratch build mock-output.log === kernel-6.2.0-0.rc3.20230111git7dd4b804e080.26.fc38.i686 built with glibc-devel-2.36.9000-21.fc38.i686 No matching package to install: 'bpftool' ... Not all dependencies satisfied Error: Some packages could not be found. known failure - not a glibc issue
=== aarch64 lua scratch build build.log ===
lua-5.4.4-7.fc38.src.rpm, aarch64
built with glibc-devel-2.36.9000-21.fc38.aarch64
<pre>
testing 'format %a %A'
>>> closing state <<<
/builddir/build/BUILDROOT/lua-5.4.4-7.fc38.aarch64//usr/bin/lua: strings.lua:342: assertion failed!
stack traceback:
[C]: in function 'assert'
strings.lua:342: in main chunk
[C]: in function 'dofile'
all.lua:165: in main chunk
[C]: in ?
.error: Bad exit status from /var/tmp/rpm-tmp.tSJRwD (%check)
Bad exit status from /var/tmp/rpm-tmp.tSJRwD (%check)
RPM build errors:
Child return code was: 1
lua test suite fails assertion - not a glibc issue