Bodhi Infrastructure SOP

Bodhi is used by Fedora developers to submit potential package updates for releases. From here, bodhi handles all of the dirty work, from sending around emails, dealing with Koji, to mashing the repositories.

Bodhi project Trac: https://fedorahosted.org/bodhi/

Contact Information

Owner: Fedora Infrastructure Team

Contact: #fedora-admin

Persons: lmacken, f13

Location: Phoenix

Servers: admin.fedoraproject.org/updates

Purpose: Push package updates, and handle new submissions.

Adding a new pending release

The following commands need to be run from a bodhi checkout at the moment. The easiest way to do this is to log into releng02, and do the following.

$ sudo su - masher
$ cd bodhi
$ make shell
>>> Release(name='F13', long_name='Fedora 13', id_prefix='FEDORA', dist_tag='dist-f13', locked=True)
>>> from bodhi.jobs import refresh_metrics
>>> refresh_metrics()

0-day Release Actions

Going from pending to a proper release in bodhi requires setting the Release.locked field to false:

ssh releng02
sudo su - masher
cd bodhi
make shell
>>> Release.byName('F15').locked = False
>>> exit()

You may also need to disable any pre-release policy defined in the bodhi config in puppet.

ssh puppet01
cd puppet/modules/bodhi/templates
git pull
vi bodhi-prod.cfg.erb

Then you can safely remove or comment out the release-specific policy:

# Release status
# post-beta enforces the 'Beta to Pre Release' policy defined here:
# https://fedoraproject.org/wiki/Updates_Policy
f15.status = 'post_beta'
f15.post_beta.mandatory_days_in_testing = 3
f15.post_beta.critpath.num_admin_approvals = 1
f15.post_beta.critpath.min_karma = 2

Pushing EPEL updates

SSH into the EPEL masher, and run the following command:

* sudo bodhi --push --push-release=EL-4 --push-release=EL-5 -u $YOUR_USERNAME

This will display a list of updates that are ready to be pushed. If you press enter, it will quit the tool and write out the package lists to corresponding files (eg: Testing-EL5, Stable-EL4) which can then be used for piping through the signing scripts.

Once the packages are signed you can run the above bodhi command again and press y to begin the push. You can then keep track of the progress by running tail -f /var/log/bodhi/server.log. If a problem occurs, you can resume the push by running:

* sudo bodhi --push --push-release=EL-4 --push-release=EL-5 -u $YOUR_USERNAME --resume-push

Doing a production bodhi upgrade

for app in 01 02 03 04 5 6 07; do
  sudo func "app$app*" call command run 'yum clean metadata'
  sudo func "app$app*" call command run 'yum -y update bodhi-server'
  sudo func "app$app*" call command run '/usr/sbin/puppetd -t'
  curl -I http://app$app/updates/
  sleep 5
done

Release EOL

We currently remove releases from bodhi when they reach their End of Life.

First, we take a database snapshot:

[masher@releng02 bodhi]$ PYTHONPATH=$(pwd) python bodhi/tools/pickledb.py save

Then, we can remove all F13 related updates and comments from the database:

[masher@releng02 bodhi]$ PYTHONPATH=$(pwd) python bodhi/tools/rmrelease.py F13

Troubleshooting and Resolution

Empty <id/> tags in the updateinfo.xml.gz

Bodhi caches the most recent repodata for all releases in /mnt/koji/mash/updates/*.repodata. Sometimes, if a push fails or crashes for some reason, this can lead to corrupted metadata. So, if you see this message:

* Repodata sanity check failed!
* updateinfo.xml.gz contains empty ID tags

You can remove the cached metadata and try resuming the push:

* rm -fr /mnt/koji/mash/updates/el*.repodata
* bodhi --push --push-release=EL-4 --push-release=EL-5 -u $YOUR_USERNAME --resume-push

Creating tables at sqlite:/// when trying to use postgres!?

When using a third-party TurboGears identity provider or visit manager, this may sometimes require you to enable SQLAlchemy in your configuration. Since bodhi v0.x uses SQLObject, this can currently confuse tg-admin initialization configurations, and will require you to temporarily comment out the following line when you need to run the tg-admin sql create database initialization.

# Required by jsonfas identity provider to run, but confuses tg-admin.
#sqlalchemy.dburi="sqlite:///"

More Tips/tricks for working with a Bodhi instance can be found here:

https://fedorahosted.org/bodhi/wiki/Administration