Rawhide synchronization for the GNU C Library
Make sure you have authenticated and meet the pre-requisites (see notes below).
Synchronization Process
- Setup/Fetch a clean upstream master repository for glibc. This can be an existing directory also which you switch to master branch and ensure it has been rebased e.g. git pull --rebase cleanly to head.
mkdir -p $HOME/src cd $HOME/src git clone git://sourceware.org/git/glibc.git glibc-pristine
- Setup/Fetch a clean Fedora rawhide glibc repository. If this is not your first time syncing, you may need to remove the glibc-patches subdirectory that the previous sync created.
cd $HOME/fedsrc fedpkg clone glibc
- Run fedpkg sources to downloads the .tar.xy file needed for creating the glibc-patches repository (next step).
cd $HOME/fedsrc/glibc fedpkg sources
- Convert the Fedora rawhide glibc dist-git repository to a glibc-patches repository (git repo with Fedora patches applied as commits).
export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts cd $HOME/fedsrc/glibc $GLIBC_MS/glibc-patches-to-git.py
- Synchronize from upstream master to Fedora rawhide and review changes.
NOTE: Pick the appropriate set of instructions based on where we are in the release.
No release transitions occurring:
cd $HOME/fedsrc/glibc $GLIBC_MS/glibc-sync-upstream.py --import-git $HOME/src/glibc-pristine --verbose git diff Confirm that glibcsrcdir, glibcversion, and Release have the correct NEVRA.
Transitioning from stable release -> master:
cd $HOME/fedsrc/glibc $GLIBC_MS/glibc-sync-upstream.py --import-git $HOME/src/glibc-pristine --verbose --import-git-branch master git diff Manually change the glibcsrcdir, glibcversion, and Release to match the new NEVRA.
If a merge conflict occurs, review the section on Dealing with Merge Conflicts then return to the next step.
- Manually document note-worthy changes in the %changelog. This step is probably the most complex. You need to look at all the changes since the last sync (the hash is recorded in the %changelog) in the upstream repo and see if there is anything note-worthy to talk about e.g. git diff HASH1^..HASH2. You will use this text in your commit message also.
* Wed Nov 07 2018 Florian Weimer <fweimer@redhat.com> - 2.28.9000-14 - Auto-sync with upstream branch master, commit 1df872fd74f730bcae3df201a229195445d2e18a: - libanl: Fix crash if first helper thread creation failed (#1646381)
A useful command to summarize commits is git log like this:
cd $HOME/src/glibc-pristine git log --oneline e0cb7b6131..0ddb7ea842 | sed 's/^[^ ]* /- /'
The two hashes to use are based on the last sync's git commit and the current sync's git commit. You want the log listing to show all of the new commits that have been included in the current sync. In the above example, we're updating from e0cb7b6131 (the commit of the last sync) to 0ddb7ea842 (the commit of the current sync). Verify the boundary commits make sense.
- Add new files (still in the Fedora rawhide directory from the last step) and commit
cd $HOME/fedsrc/glibc fedpkg new-sources glibc-*.tar.xz git add glibc.spec git commit
An appropriate commit message would be:
Auto-sync with upstream branch master
Upstream commit: 1df872fd74f730bcae3df201a229195445d2e18a
- libanl: Fix crash if first helper thread creation failed (#1646381)
Following the spec file %changelog example from above.
- Test a scratch build, and wait for it to complete.
fedpkg srpm fedpkg scratch-build --srpm ./glibc-XXX.src.rpm
- Verify scratch build results by downloading logs and looking for any unexpected failures. A list of known failures is kept at the end of this page, and should be updated as needed.
export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts $TOOLS/codonell/get-build-logs.sh https://koji.fedoraproject.org/koji/taskinfo?taskID=$TASKID
- Lastly, push the commit and kick off a Rawhide build if the logs look good.
git push fedpkg build
Authentication
- Kerberos init with Fedora realm.
- Make sure your ssh agent has your Fedora ssh key for pagure.io.
Pre-requisites
- You have a Fedora account with an SSH key, and have setup your SSH key on pagure.io by logging in.
- Install fedpkg, python3-pygit2, python3-rpm and git-merge-changelog
- Configure your ~/.gitconfig to have entries for name, and a merge driver for GNU Changelogs.
[user]
name = My Name
[merge "merge-changelog"]
name = GNU-style ChangeLog Merge driver
driver = /usr/bin/git-merge-changelog
[core]
attributesfile = ~/.gitattributes
- Configure your ~/.gitattributes
ChangeLog merge=merge-changelog
- Clone glibc-maintainer-scripts into $GLIBC_MS for the git synchronization scripts.
export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts mkdir -p $HOME/fedsrc cd $HOME/fedsrc git clone https://pagure.io/glibc-maintainer-scripts.git
- Clone UpstreamToolchainBuildScripts from the Fedora Toolchain team for the build log fetching scripts.
export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts cd $HOME/fedsrc git clone https://pagure.io/FedoraToolchainTeam/UpstreamToolchainBuildScripts.git
Note: if you plan on editing either of the two git repos you just installed, use ssh://git@pagure.io instead of https://pagure.io
Differences for Release Syncs
For syncing a glibc release branch into rawhide, you need to use a command line this, and manually reset the Release number the first time:
$GLIBC_MS/glibc-sync-upstream.py --import-git-branch release/2.31/master --import-git $HOME/src/glibc-2.31 --verbose
For Fedora release branches, the process is basically the same, except you have to use the Bodhi system to push your build to a release.
- In the above instructions, use the release branch of glibc (typically like release/2.28/master) and the release branch of Fedora (typically like f29)
- Click the ? icon in the upper right, to search for "glibc"
- Note previous updates for wordings and severities
- Select "Create" in the upper right and create a new update
- Type "glibc" in the packages field. If there's a popup, select the appropriate entry.
- type "glibc" in the candidate builds field, select the relevent build from the popup.
- Click in the "Related bugs" field. If your BZ doesn't show up, type in the number.
- Fill in the Update Notes field and final details.
- Submit! The system will advertise the update and request karma. When the update has enough karma, it's automatically pushed out.
Dealing with Merge Conflicts
- Dealing with merge conflicts due to backport+rebase:
Sometimes, a fix is backported from upstream into rawhide, and subsequently a rebase is scheduled. During this rebase, the backport and its corresponding patch need to be dropped.
One way to achieve this is to manually edit the spec file prior to running glibc-patches-to-git so that the merge conflict is avoided in the first place.
Another way is to let the merge conflict occur, then run git rebase --skip in the glibc-patches directory/repository to skip the application of the current patch (i.e. the one backported, which is causing the merge conflict) and continue the rebase (this rebase is an internal detail of how glibc-sync-upstream is implemented and was started by the script before erroring out due to the conflict).
Then we continue the process by executing glibc-git-to-patches in $HOME/fedsrc/glibc, which now skips the patch when creating glibc-patches.
Finally, we run glibc-patches-to-git --branch master, to bring update dist-git with the recently refreshed glibc-patches.
- Dealing with Fedora patches that no longer apply cleanly due to changes in rawhide:
Another merge conflict scenario is when a Fedora patch does not apply cleanly. You may see a message like this:
CONFLICT (content): Merge conflict in nss/nsswitch.conf error: Failed to merge in the changes.
In this case, the patch is needed for Fedora but an upstream change has caused the patch to be out of sync. You need to fix the patch in the glibc-patches directory. Using nss/nsswitch.conf as an example:
cd $HOME/fedsrc/glibc/glibc-patches edit nss/nsswitch.conf to fix the merge conflict git add nss/nsswitch.conf git rebase --continue cd $HOME/fedsrc/glibc
If you see that there is a glibc.spec and glibc.spec.new
mv glibc.spec glibc.spec.old mv glibc.spec.new glibc.spec
Now complete the git-to-patches process, regenerating the patch that failed previously:
$GLIBC_MS/glibc-git-to-patches.py -v
You should see a message similar to this:
info: regenerating patch 'glibc-fedora-nsswitch.patch' because patch failed
git add glibc-fedora-nsswitch.patch
Known Acceptable Build Failures
For the purposes of "looking for any unexpected failures" (above), this is the current list of build failures that have been deemed "not to stop a sync". After you complete the scratch build and download the logs, compare the test failures in the logs with the results below. For each failure not listed here, analyze it, determine if it's safe to continue syncing, and add that failure below. For each failure listed below that no longer appears in the test results, remove it from below. I.e. most syncs will require minor updates in this section.
All Hosts
aarch64
FAIL: math/test-float128-j0 FAIL: math/test-float64x-j0 FAIL: math/test-ldouble-j0
armv7hl
FAIL: misc/tst-sysvshm-linux - NEW
i686
Various minor ULP bumps
FAIL: math/test-float128-j0 FAIL: math/test-float64x-j0 FAIL: math/test-ldouble-j0 These relate to this bug: https://sourceware.org/bugzilla/show_bug.cgi?id=20305 FAIL: signal/tst-minsigstksz-1 FAIL: signal/tst-minsigstksz-2
ppc64le
Upstream ulps patch has been submitted by Matheus Castanho - pending push: FAIL: math/test-float128-j0 FAIL: math/test-float64x-j0 FAIL: math/test-ibm128-j0 ulp : 16.0000 max.ulp : 8.0000 FAIL: math/test-ldouble-j0 Sporadic failure: FAIL: time/tst-cpuclock1
s390x
Sporadic failure: FAIL: rt/tst-cpuclock2