Plan for Koji
This is the basic plan for how automatic image generation will work in Koji.
Overview
Inputs
- Kickstart files (from https://git.fedorahosted.org/cgit/cloud-kickstarts.git)
- Repo to use (for manual builds of release candidates, etc.)
- name (ditto; weekly should have deterministic name with "weekly" and date)
Outputs
- Images in all EC2 regions
- Images uploaded to https://dl.fedoraproject.org/pub/alt/cloud
- Fedmsg message announcing new uploads
- simple web service providing stateful report on current images (JSON)
Steps
- Cron job starts image build in koji (manual initiation also possible)
- As they complete, Koji sends messages on the Fedora Message Bus (fedmsg)
- uploader service consumes consumes these messages and uploads AMIs images to Amazon and qcow2 and tar xz'd raw images to http://alt.fedoraproject.org/cloud/ (see note on formats)
- uploader service sends Fedora Message Bus message when each image is successfully uploaded
- web listener service consumes these messages and stores to DB, serves out as JSON (also other possible formats, including something pretty?)
Not Pictured
- Automatic QE system will pick up fedora messages and run QE automatically
- Pretty web site showing weekly and test candidate image builds, possibly with a way for releng to mark certain ones as the official release
Notes
1. Cron Tasks
Right now, livecd nightly builds are launched by hand. A cron script should automate this instead. The current process requires Koji admin credentials; the new system should avoid that.
2. Koji Fedmsg Integration
Right now, Koji does not send notifications for scratch builds. We need to either change that or make these "real" builds.
More on Fedmsg at http://www.fedmsg.com/
3. Formats
Koji can produce either RAW or qcow2 images. My preference is for the service to produce compressed qcow2 images, because they're smallest, but the important thing is that the files that go on the FTP server should be compressed qcow2 (using qcow2's native compression) and tar.xz'd raw image files. (Tar must be used because RHEL 6 and earlier versions of tar do not understand sparseness).
4. Security
Consumers need to verify that messages are actually coming from the service that they're claiming to come from, and files should only be transferred from their expected locations rather than from where the message says to look
5. "DB of some kind"
Maybe https://launchpad.net/simplestreams is appropriate here. (Thanks gholms!)