Features/FirstClassCloudImages/KojiPlan

Plan for Koji

This is the basic plan for how automatic image generation will work in Koji.

Overview

Inputs

1. Kickstart files (from https://git.fedorahosted.org/cgit/cloud-kickstarts.git)
2. Repo to use (for manual builds of release candidates, etc.)
3. name (ditto; weekly should have deterministic name with "weekly" and date)

Outputs

1. Images in all EC2 regions
2. Images uploaded to https://dl.fedoraproject.org/pub/alt/cloud
3. Fedmsg message announcing new uploads
4. simple web service providing stateful report on current images (JSON)

Steps

1. Cron job starts image build in koji (manual initiation also possible)
2. As they complete, Koji sends messages on the Fedora Message Bus (fedmsg)
3. uploader service consumes these messages and uploads AMIs images to Amazon and qcow2 and tar xz'd raw images to http://alt.fedoraproject.org/cloud/ (see note on formats)
4. uploader service sends Fedora Message Bus message when each image is successfully uploaded
5. web listener service consumes these messages and stores to DB, serves out as JSON (also other possible formats, including something pretty?)

Not Pictured

1. Automatic QE system will pick up fedora messages and run QE automatically
2. Pretty web site showing weekly and test candidate image builds, possibly with a way for releng to mark certain ones as the official release

Notes

1. Cron Tasks

Right now, livecd nightly builds are launched by hand. A cron script should automate this instead. The current process requires Koji admin credentials; the new system should avoid that.

Rawhide images are now built by a nightly cron job. I assume we will do this for the branch, too.

2. Koji Fedmsg Integration

Right now, Koji does not send notifications for scratch builds has been setup to send notifications for scratch builds. We need to either change that or make these "real" builds This is already taken care of.

More on Fedmsg at http://www.fedmsg.com/

3. Formats

Koji can produce either RAW or qcow2 images. My preference is for the service to produce compressed qcow2 images, because they're smallest, but the important thing is that the files that go on the FTP server should be compressed qcow2 (using qcow2's native compression) and tar.xz'd raw image files. (Tar must be used because RHEL 6 and earlier versions of tar do not understand sparseness).

4. Security

Consumers need to verify that messages are actually coming from the service that they're claiming to come from, and files should only be transferred from their expected locations rather than from where the message says to look

5. "DB of some kind" / Web app

Maybe https://launchpad.net/simplestreams is appropriate here. (Thanks gholms!)

See for example our friends at Ubuntu: http://cloud-images.ubuntu.com/releases/streams/v1/

The datanommer/datagrepper db could be re-used here. It already stores all messages and can be queried to produce status reports on images over time. It, for instance, is what drives the rel-eng dashboard: https://apps.fedoraproject.org/releng-dash

6. Taskbot?

See http://tirfa.com/an-initial-idea-for-taskbot.html

In recent months, Taskbot was renamed to Taskotron - https://fedoraproject.org/wiki/Taskotron

Fedimg

Fedimg was started in summer 2014 to perform automatic cloud image uploading to multiple internal and external cloud providers, such as Amazon EC2 and our internal Openstack. The uploading process is triggered by fedmsgs emitted by createImage Koji build methods. Docs have been started on RTFD.