GlibcRawhideSync

Rawhide synchronization for the GNU C Library

Make sure you have authenticated and meet the pre-requisites (see notes below).

TL;DR

If you do this every week, here's the cut and paste snippets you need, minus the bits that change every week...

kinit

export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts
export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts

cd $GLIBC_MS; git pull --rebase
cd $TOOLS; git pull --rebase
cd $HOME/src/glibc-pristine; git pull --rebase
cd $HOME/fedsrc/glibc; git pull --rebase
fedpkg sources
rm -rf glibc-patches; $GLIBC_MS/glibc-patches-to-git.py
$GLIBC_MS/glibc-sync-upstream.py --import-git $HOME/src/glibc-pristine --verbose
git diff
cd $HOME/src/glibc-pristine; git log --oneline (fill)..(in) | sed 's/[^ ]* /- /'
cd $HOME/fedsrc/glibc; ${VISUAL:=vi} glibc.spec

fedpkg new-sources `/bin/ls -1rt glibc-*.tar.xz | tail -1`
git add glibc.spec
git commit

fedpkg srpm
fedpkg scratch-build --srpm `/bin/ls -1rt $PWD/glibc-*.src.rpm | tail -1`

rm -rf logs; mkdir logs; cd logs
$TOOLS/codonell/get-build-logs.sh https://koji.fedoraproject.org/koji/taskinfo?taskID=(fill-in)

cd $HOME/fedsrc/glibc; git push
fedpkg build

https://bodhi.fedoraproject.org/updates/?search=glibc

Synchronization Process

Fedora CI/CD
Fedora Rawhide must pass Fedora CI/CD, in particular baseos-qe.koji-build.scratch-build.validation before you can consider the synchronization process complete.

1. Setup/Fetch a clean upstream master repository for glibc. This can be an existing directory also which you switch to master branch and ensure it has been rebased e.g. git pull --rebase cleanly to head.

mkdir -p $HOME/src
cd $HOME/src
git clone git://sourceware.org/git/glibc.git glibc-pristine

2. Setup/Fetch a clean Fedora rawhide glibc repository. If this is not your first time syncing, you may need to remove the glibc-patches subdirectory that the previous sync created.

cd $HOME/fedsrc
fedpkg clone glibc

3. Run fedpkg sources to downloads the .tar.xy file needed for creating the glibc-patches repository (next step).

cd $HOME/fedsrc/glibc
fedpkg sources

4. Convert the Fedora rawhide glibc dist-git repository to a glibc-patches repository (git repo with Fedora patches applied as commits).

export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts
cd $HOME/fedsrc/glibc
$GLIBC_MS/glibc-patches-to-git.py

5. Synchronize from upstream master to Fedora rawhide and review changes. NOTE: Pick the appropriate set of instructions based on where we are in the release.

No release transitions occurring:

cd $HOME/fedsrc/glibc
$GLIBC_MS/glibc-sync-upstream.py --import-git $HOME/src/glibc-pristine --verbose
git diff
Confirm that glibcsrcdir, glibcversion, and Release have the correct NEVRA.

Transitioning from stable release -> master:

cd $HOME/fedsrc/glibc
$GLIBC_MS/glibc-sync-upstream.py --import-git $HOME/src/glibc-pristine --verbose --import-git-branch master
git diff
Manually change the glibcsrcdir, glibcversion, and Release to match the new NEVRA.

If a merge conflict occurs, review the section on Dealing with Merge Conflicts then return to the next step.

6. Manually document note-worthy changes in the %changelog. This step is probably the most complex. You need to look at all the changes since the last sync (the hash is recorded in the %changelog) in the upstream repo and see if there is anything note-worthy to talk about e.g. git diff HASH1^..HASH2. You will use this text in your commit message also.

* Wed Nov 07 2018 Florian Weimer <fweimer@redhat.com> - 2.28.9000-14
- Auto-sync with upstream branch master,
  commit 1df872fd74f730bcae3df201a229195445d2e18a:
- libanl: Fix crash if first helper thread creation failed (#1646381)

A useful command to summarize commits is git log like this:

cd $HOME/src/glibc-pristine
git log --oneline e0cb7b6131..0ddb7ea842 | sed 's/^[^ ]* /- /'

The two hashes to use are based on the first ten digits of the last sync's git commit and the current sync's git commit. You want the log listing to show all of the new commits that have been included in the current sync. In the above example, we're updating from e0cb7b6131 (the commit of the last sync) to 0ddb7ea842 (the commit of the current sync). Verify the boundary commits make sense.

7. Add new files (still in the Fedora rawhide directory from the last step) and commit

cd $HOME/fedsrc/glibc
fedpkg new-sources glibc-*.tar.xz
git add glibc.spec
git commit

An appropriate commit message would be:

    Auto-sync with upstream branch master
    
    Upstream commit: 1df872fd74f730bcae3df201a229195445d2e18a
    
    - libanl: Fix crash if first helper thread creation failed (#1646381)

Following the spec file %changelog example from above.

8. Test a scratch build, and wait for it to complete.

fedpkg srpm
fedpkg scratch-build --srpm ./glibc-XXX.src.rpm

9. Verify scratch build results by downloading logs and looking for any unexpected failures. A list of known failures is kept at the end of this page, and should be updated as needed.

export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts
$TOOLS/codonell/get-build-logs.sh https://koji.fedoraproject.org/koji/taskinfo?taskID=$TASKID

10. Push the commit and kick off a Rawhide build if the logs look good (you are not done yet!)

git push
fedpkg build

11. Fedora CI/CD must run and you must verify that baseos-qe.koji-build.scratch-build.validation succeeded. If Fedora CI/CD does not succeed please reach out to Martin Cermak (BaseOS QE). To verify the status of CI/CD go to Bodhi after your build successfully completes, select Search..., type in glibc, hit enter and pick the newest Rawhide build errata that was automatically created and select the Automated Tests tab. If baseos-qe.koji-build.scratch-build.validation is green then the test has passed and you are complete with the sync.

Authentication

1. Kerberos init with Fedora realm.

2. Make sure your ssh agent has your Fedora ssh key for pagure.io.

Pre-requisites

1. You have a Fedora account with an SSH key, and have setup your SSH key on pagure.io by logging in.

2. Install fedpkg, python3-pygit2, python3-rpm and git-merge-changelog

3. Configure your ~/.gitconfig to have entries for name, and a merge driver for GNU Changelogs.

[user]
	name = My Name

[merge "merge-changelog"]
        name = GNU-style ChangeLog Merge driver
        driver = /usr/bin/git-merge-changelog
[core]
	attributesfile = ~/.gitattributes

4. Configure your ~/.gitattributes

ChangeLog       merge=merge-changelog

5. Clone glibc-maintainer-scripts into $GLIBC_MS for the git synchronization scripts.

export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts
mkdir -p $HOME/fedsrc
cd $HOME/fedsrc
git clone https://pagure.io/glibc-maintainer-scripts.git

6. Clone UpstreamToolchainBuildScripts from the Fedora Toolchain team for the build log fetching scripts.

export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts
cd $HOME/fedsrc
git clone https://pagure.io/FedoraToolchainTeam/UpstreamToolchainBuildScripts.git

Note: if you plan on editing either of the two git repos you just installed, use ssh://git@pagure.io instead of https://pagure.io

Differences for Release Syncs

For syncing a glibc release branch into rawhide, you need to use a command line this, and manually reset the Release number the first time:

$GLIBC_MS/glibc-sync-upstream.py --import-git-branch [Release branch you are using e.g. release/2.31/master] --import-git [Path to your updated checked out glibc tree e.g. $HOME/src/glibc-2.31] --verbose

For Fedora release branches, the process is basically the same, except you have to use the Bodhi system to push your build to a release.

• In the above instructions, use the release branch of glibc (typically like release/2.28/master) and the release branch of Fedora (typically like f29)

• Go to https://bodhi.fedoraproject.org/

• Click the ? icon in the upper right, to search for "glibc"

• Note previous updates for wordings and severities

• Select "Create" in the upper right and create a new update

• Type "glibc" in the packages field. If there's a popup, select the appropriate entry.

• type "glibc" in the candidate builds field, select the relevent build from the popup.

• Click in the "Related bugs" field. If your BZ doesn't show up, type in the number.

• Fill in the Update Notes field and final details.

• Submit! The system will advertise the update and request karma. When the update has enough karma, it's automatically pushed out.

• This is an example of a Fedora 39 sync to release/2.38/master. It assumes you have done at least one sync previously, have completed the pre-requisites, and cloned the source files.

kinit
export GLIBC_MS=$HOME/fedsrc/glibc-maintainer-scripts
export TOOLS=$HOME/fedsrc/UpstreamToolchainBuildScripts

cd $GLIBC_MS; git pull --rebase
cd $TOOLS; git pull --rebase
cd $HOME/src/glibc-pristine; git pull --rebase
* Checkout the version you are syncing to.
git checkout release/2.38/master
cd $HOME/fedsrc/glibc; git pull --rebase
* Checkout the version of Fedora that you are syncing.
git checkout f39
fedpkg sources
rm -rf glibc-patches; $GLIBC_MS/glibc-patches-to-git.py
* Specify --import-git-branch for the correct release.
$GLIBC_MS/glibc-sync-upstream.py --import-git-branch release/2.38/master --import-git $HOME/src/glibc-pristine --verbose
git diff
cd $HOME/src/glibc-pristine; git log --oneline (fill)..(in) | sed 's/[^ ]* /- /'
cd $HOME/fedsrc/glibc; ${VISUAL:=vi} glibc.spec

fedpkg new-sources `/bin/ls -1rt glibc-*.tar.xz | tail -1`
git add glibc.spec
git commit

fedpkg srpm
fedpkg scratch-build --srpm `/bin/ls -1rt $PWD/glibc-*.src.rpm | tail -1`

rm -rf logs; mkdir logs; cd logs
$TOOLS/codonell/get-build-logs.sh https://koji.fedoraproject.org/koji/taskinfo?taskID=(fill-in)

cd $HOME/fedsrc/glibc; git push
fedpkg build

* Create the bodhi update. You can use a previous update as an example:
https://bodhi.fedoraproject.org/updates/?search=glibc

Dealing with Merge Conflicts

• Dealing with merge conflicts due to backport+rebase:

Sometimes, a fix is backported from upstream into rawhide, and subsequently a rebase is scheduled. During this rebase, the backport and its corresponding patch need to be dropped.

One way to achieve this is to manually edit the spec file prior to running glibc-patches-to-git so that the merge conflict is avoided in the first place.

Another way is to let the merge conflict occur, then run git rebase --skip in the glibc-patches directory/repository to skip the application of the current patch (i.e. the one backported, which is causing the merge conflict) and continue the rebase (this rebase is an internal detail of how glibc-sync-upstream is implemented and was started by the script before erroring out due to the conflict).

Then we continue the process by executing glibc-git-to-patches in $HOME/fedsrc/glibc, which now skips the patch when creating glibc-patches.

Finally, we run glibc-patches-to-git --branch master, to bring update dist-git with the recently refreshed glibc-patches.

• Dealing with Fedora patches that no longer apply cleanly due to changes in rawhide:

Another merge conflict scenario is when a Fedora patch does not apply cleanly. You may see a message like this:

CONFLICT (content): Merge conflict in nss/nsswitch.conf error: Failed to merge in the changes.

In this case, the patch is needed for Fedora but an upstream change has caused the patch to be out of sync. You need to fix the patch in the glibc-patches directory. Using nss/nsswitch.conf as an example:

cd $HOME/fedsrc/glibc/glibc-patches
edit nss/nsswitch.conf to fix the merge conflict
git add nss/nsswitch.conf
git rebase --continue

cd $HOME/fedsrc/glibc

If you see that there is a glibc.spec and glibc.spec.new

mv glibc.spec glibc.spec.old
mv glibc.spec.new glibc.spec

Now complete the git-to-patches process, regenerating the patch that failed previously:

$GLIBC_MS/glibc-git-to-patches.py -v

You should see a message similar to this:

info: regenerating patch 'glibc-fedora-nsswitch.patch' because patch failed

git add glibc-fedora-nsswitch.patch

Occasions that require special handling

• The addition of a new locale upstream:

When a new locale is added upstream, the change corresponds to a new line in localedata/SUPPORTED and glibc.spec needs to be adjusted to tackle this. The spec file includes a lua script containing a list of supported locales that is compared at build time to the list upstream. This is below the definition of %package locale-source. Grep for ^local locales = to locate the list and add a line corresponding to the new locale there before building.

Known Acceptable Build Failures

For the purposes of "looking for any unexpected failures" (above), this is the current list of build failures that have been deemed "not to stop a sync". After you complete the scratch build and download the logs, compare the test failures in the logs with the results below. For each failure not listed here, analyze it, determine if it's safe to continue syncing, and add that failure below. For each failure listed below that no longer appears in the test results, remove it from below. I.e. most syncs will require minor updates in this section.

All Hosts

There is a known misc/tst-ttyname failure that needs waiving (https://bugzilla.redhat.com/show_bug.cgi?id=2210335) since we do not currently believe this is a glibc regression - per codonell.

aarch64

Rawhide (1 fail): 
FAIL: misc/tst-mount

FAIL: elf/tst-debug1
=====FAIL: elf/tst-debug1.out=====
Didn't expect signal from child: got `Bus error'
=====FAIL: elf/tst-debug1.test-result=====
FAIL: elf/tst-debug1
original exit status 1
=====FAIL: dlfcn/tst-dlinfo-phdr=====
see upstream bz 23293

armv7hl

Rawhide (1 fail): FAIL: debug/tst-longjmp_chk2
=====FAIL: debug/tst-longjmp_chk2.out=====
not on alternate stack
 in signal handler
 on alternate stack
Didn't expect signal from child: got `Aborted'
=====FAIL: debug/tst-longjmp_chk2.test-result=====
FAIL: debug/tst-longjmp_chk2
original exit status 1

FAIL: misc/tst-sysvshm-linux

The tst-sysvshm-linux failure is related to: https://sourceware.org/bugzilla/show_bug.cgi?id=26862:

i686

Rawhide (1 fail):
FAIL: misc/tst-mount

This seems to be transient, unable to reproduce manually:

=====FAIL: io/tst-stat-time64.out=====
tst-stat-time64.c:88: numeric comparison failure
   left: 8 (0x8); from: stx.stx_blocks
  right: 16 (0x10); from: st.st_blocks
error: 1 test failures

These are all ulp diffs:
=====FAIL: math/test-double-log2p1.out=====
testing double (without inline functions)
Failure: Test: log2p1 (-0x4.f37d38p-4)
Result:
 is:         -5.3417300298066683e-01  -0x1.117f1fb46a88cp-1
 should be:  -5.3417300298066672e-01  -0x1.117f1fb46a88bp-1
 difference:  1.1102230246251565e-16   0x1.0000000000000p-53
 ulp       :  1.0000
 max.ulp   :  0.0000

FAIL: math/test-double-log2p1
FAIL: math/test-float-log2p1
FAIL: math/test-float128-log2p1
FAIL: math/test-float32-log2p1
FAIL: math/test-float32x-log2p1
FAIL: math/test-float64-log2p1
FAIL: math/test-float64x-log2p1
FAIL: math/test-ldouble-log2p1

ppc64le

FAIL: math/test-float-nearbyint
FAIL: math/test-float-rint
FAIL: math/test-float32-nearbyint
FAIL: math/test-float32-rint

Rawhide (2 fails)
FAIL: math/test-ibm128-llround
FAIL: math/test-ibm128-y1

Previous failures - no longer seeing these:
FAIL: misc/tst-mount

FAIL: misc/tst-sigcontext-get_pc
Sporadic failure:
FAIL: time/tst-cpuclock1

s390x

Rawhide (1 fail):
FAIL: misc/tst-mount

<pre>


FAIL: math/test-double-log2p1
FAIL: math/test-float-log2p1These are all ulp diffs:
=====FAIL: math/test-double-log2p1.out=====
testing double (without inline functions)
Failure: Test: log2p1 (-0x4.f37d38p-4)
Result:
 is:         -5.3417300298066683e-01  -0x1.117f1fb46a88cp-1
 should be:  -5.3417300298066672e-01  -0x1.117f1fb46a88bp-1
 difference:  1.1102230246251565e-16   0x1.0000000000000p-53
 ulp       :  1.0000
 max.ulp   :  0.0000

FAIL: math/test-float128-log2p1
FAIL: math/test-float32-log2p1
FAIL: math/test-float32x-log2p1
FAIL: math/test-float64-log2p1
FAIL: math/test-float64x-log2p1
FAIL: math/test-ldouble-log2p1

Sporadic failure: FAIL: rt/tst-cpuclock2

x86_64

Rawhide (1 fail): 
FAIL: misc/tst-mount

=====FAIL: nptl/tst-stack4.out=====
=====FAIL: nptl/tst-stack4.test-result=====
FAIL: nptl/tst-stack4
original exit status 127

Known CI Failures

Please list known CI failures here. CI failures must be coordinated with BaseOS QE.

fedora-ci.koji-build.rpmdeplint.functional

glibc-devel-2.36.9000-21.fc38.aarch64 - testout.log

glibc-devel-2.36.9000-21.fc38.aarch64 provides /usr/include/bits/sigaction.h which is also provided by glibc-headers-s390-2.36.9000-21.fc38.noarch

The above is a known-failure because not all headers can be installed at the same time. The CI test is attempting to install every package we can install and that's not a valid test of anything.
It is not valid to install the aarch64 headers and s390x headers at the same time. The test doesn't know this. Nobody would every need to do this.  But it attempts all noarch package installs.

baseos-qe.koji-build.scratch-build.validation

===i686 kernel scratch build mock-output.log ===
kernel-6.2.0-0.rc3.20230111git7dd4b804e080.26.fc38.i686
built with glibc-devel-2.36.9000-21.fc38.i686

No matching package to install: 'bpftool'
...
Not all dependencies satisfied
Error: Some packages could not be found.

known failure - not a glibc issue

=== aarch64 lua scratch build build.log ===
lua-5.4.4-7.fc38.src.rpm, aarch64
built with glibc-devel-2.36.9000-21.fc38.aarch64

<pre>
testing 'format %a %A'
>>> closing state <<<
/builddir/build/BUILDROOT/lua-5.4.4-7.fc38.aarch64//usr/bin/lua: strings.lua:342: assertion failed!
stack traceback:
	[C]: in function 'assert'
	strings.lua:342: in main chunk
	[C]: in function 'dofile'
	all.lua:165: in main chunk
	[C]: in ?
.error: Bad exit status from /var/tmp/rpm-tmp.tSJRwD (%check)
    Bad exit status from /var/tmp/rpm-tmp.tSJRwD (%check)
RPM build errors:
Child return code was: 1

lua test suite fails assertion - not a glibc issue