Bodhi2 push monitoring
Troubleshooting
# get the errors sudo journalctl --since=yesterday -o short -u fedmsg-hub > ~/error.out awk '/E[Rr][Rr]/' ~/error.out # or with color egrep 'ERROR|Errno' ~/error.out
# reset the fedmsg-hub service sudo systemctl restart fedmsg-hub
OSError: [Errno 16] Device or resource busy: '/var/lib/mock/fedora-23-updates-x86_64/root/var/tmp/rpm-ostree.hjvMfC'
A bind mount needs to be removed. Look for TMPFS relics from rpm-ostree
findmnt -t tmpfs -o TARGET | grep rpm-ostree
Dismount ALL the relic TMPFS mounts (warning it's unwise to do this unless you know bodhi2 is completely idle)
sudo umount /var/lib/mock/*/root/var/tmp/rpm-ostree.??????
IOError: Cannot open /mnt/koji/mash/updates/dist-6E-epel-151105.1606/../dist-6E-epel.repocache/repodata/repomd.xml: File /mnt/koji/mash/updates/dist-6E-epel-151105.1606/../dist-6E-epel.repocache/repodata/repomd.xml doesn't exists or not a regular file
rm -rf /mnt/koji/mash/updates/dist-6E-epel-151105.1606/../dist-6E-epel.repocache
OSError: [Errno 39] Directory not empty: '/mnt/koji/mash/updates/dist-6E-epel-testing-151201.1956/../dist-6E-epel-testing.repocache/repodata/'
Inspect the repo cache areas (optional) to verify empty ?
/mnt/fedora_koji/koji/mash/updates/dist-5E-epel{,-testing}.repocache/repodata/
/mnt/fedora_koji/koji/mash/updates/dist-6E-epel{,-testing}.repocache/repodata/
/mnt/fedora_koji/koji/mash/updates/epel7{,-testing}.repocache/repodata
/mnt/fedora_koji/koji/mash/updates/f21-updates{,-testing}.repocache/repodata
/mnt/fedora_koji/koji/mash/updates/f22-updates{,-testing}.repocache/repodata
/mnt/fedora_koji/koji/mash/updates/f23-updates{,-testing}.repocache/repodata
ERROR: can't download kf5-kdesu-None:5.13.0-1.fc22.i686 from signed path /mnt/koji/packages/kf5-kdesu/5.13.0/1.fc22/data/signed/8e1431d5/i686/kf5-kdesu-5.13.0-1.fc22.i686.rpm
Sometimes an update has lingered so long that the signed rpm's have been garbage collected, so we simply put them back from the signature cache
NSS_HASH_ALG_SUPPORT=+MD5 ~/releng/scripts/sigulsign_unsigned.py fedora-22 -v --write-all --sigul-batch-size=25 kf5-kdesu-5.13.0-1.fc22
Signing
Start the bodhi push for signing (responding 'no' when prompted)
cd /var/cache/sigul yes 'no' | sudo -u masher -S bodhi-push --releases '23 22 21 5 6 7' --username parasense
Sign the builds
for i in 23 22 21 ; do NSS_HASH_ALG_SUPPORT=+MD5 ~/releng/scripts/sigulsign_unsigned.py fedora-$i -v --write-all --sigul-batch-size=25 $(cat /var/cache/sigul/{Stable,Testing}-F${i}) ; done
for i in 7 6 5 ; do NSS_HASH_ALG_SUPPORT=+MD5 ~/releng/scripts/sigulsign_unsigned.py epel-$i -v --write-all --sigul-batch-size=25 $(cat /var/cache/sigul/{Stable,Testing}-*EL-${i}) ; done
Another way to do this would be to use screen to tmux, start the push but dellay the y/n answer. The files in /var/cache/sigul are created, so go ahead and sign those builds, then send "y" to the bodhi-push prompt.
Test if bodhi masher is running
These checks are not scientific, but may help inform.
Check for existing masher locks of currently running push, or failed previous push
ls -l /mnt/koji/mash/updates/MASHING-*
Check for running bodhi2 push (via masher)
pgrep -af /usr/bin/mash
Also check for rsync
pgrep -af rsync
Resuming failed push
# resume the push interactivly sudo -u masher bodhi-push --resume --username parasense # resume the push responding yes to everything yes| sudo -u masher -S bodhi-push --resume --username parasense
# Follow the output of fedmsg-hub sudo journalctl -o short -u fedmsg-hub -l -f
Sign Bridge Tasks
# monitor the signing on bridge for potential stalls from bodhi-backend ssh -v -o'ControlPath=none' sign-bridge01 'tail -f /var/log/sigul_bridge.log'
# Verify the bridge is running or not pgrep -af bridge.py # Restart the bridge as necessary sudo pkill -f -9 bridge.py sudo NSS_HASH_ALG_SUPPORT=+MD5 sigul_bridge -d -v -v # Review the bridge.py output tail -f /var/log/sigul_bridge.log
Stable push requests
Sometimes an urgent request to have something pushed to stable.
Here we had two lorax builds in the testing queue, and QA engineer requested they go to stable ASAP. So you have to header over to the bodhi2 web front end, revoke the "tasting push", then choose to push the build to stable. Once that is done the bodhi-push comment will permit the stable push.
sudo -u masher bodhi-push --releases '21 22 23' --request=stable --builds 'lorax-21.34-1.fc21 lorax-22.13-1.fc22 freeipa-4.2.3-1.fc23' --username parasense
Testing push request
So there was a stable push in progress, and folks requesting testing push. The testing push uses a different lock file, so they will go in parallel with the stable push in progress.
sudo -u masher bodhi-push --releases '21 22 23' --request=testing --username parasense