Koji/BuildingImages

This page documents a few additional steps to enable building LiveCDs in an existing Koji installation, how to build one, and where the code will grow from here. This is a very new feature and there hasn't been much external testing yet, so please keep that mind if you run into difficulty. For additional questions and information, ask around in #koji on FreeNode IRC and sign up to the right mailing lists as the Koji project website dictates.

Build System Preparation

This section assumes you have know-how required to install and configure a new instance of Koji, and that you have already done so. You can learn how to do so here if you need to. You should be working with version 1.3.2 or later of the software. If you upgraded to that version there are a few additional steps you'll have to complete, as detailed below. You should also have some familiarity with how livecd-creator from the livecd-tools package works.

Follow this procedure step by step to get things prepared they way they need to be. Italics indicate commands to run on the console.

Database Prep

Skip this section if you freshly installed Koji 1.3.2, or updated the schema when you upgraded.

You'll have to add a few tables and entries to the database that are used to track information about the images created in Koji. The SQL looks like this:

CREATE TABLE imageinfo (
       id SERIAL NOT NULL PRIMARY KEY,
       task_id INTEGER NOT NULL REFERENCES task(id),
       filename TEXT NOT NULL,
       filesize BIGINT NOT NULL,
       arch VARCHAR(16) NOT NULL,
       hash TEXT NOT NULL,
       mediatype TEXT NOT NULL
) WITHOUT OIDS;
CREATE INDEX imageinfo_task_id on imageinfo(task_id);
CREATE TABLE imageinfo_listing (
       image_id INTEGER NOT NULL REFERENCES imageinfo(id),
       rpm_id INTEGER NOT NULL REFERENCES rpminfo(id),
       UNIQUE (image_id, rpm_id)
) WITHOUT OIDS;
CREATE INDEX imageinfo_listing_rpms on imageinfo_listing(rpm_id);

INSERT INTO channels (name) VALUES ('livecd');
INSERT INTO permissions (name) VALUES ('livecd');

System Prep

• mkdir /mnt/koji/images: Create a directory to put the finished images. This should be done on the system running kojihub.
• chown apache:apache /mnt/koji/images: Let apache read and write to it.
• koji add-host-to-channel <your-host> livecd: add a builder to the livecd channel
• koji grant-permission livecd <user>: grant the permission to build livecds to a user. This step is optional since admins have all permissions, technically.
• koji add-group <build-tag> livecd-build: add the livecd-build group
• koji add-group-pkg <build-tag> livecd-build <pkg> ...: add packages to the livecd-build group. The packages should match what you have in the "build" group, plus: python-dbus, yum, squashfs-tools, livecd-tools, and selinux-targeted-policy as of Fedora 10.
• You will need a tag and target to build the LiveCDs from. The yum repo generated for the build tag of the target is what Koji will use to populate the LiveCDs with by default. (the alternative is to use the --repo option, more on that later)
• Version 1.3.2 introduces another PythonOption in kojiweb.conf: KojiImagesURL. This option is set to the URL where completed LiveCD images can be downloaded from. Make sure that your apache server is allowing access to that location.

Building a LiveCD

Building a Livecd in Koji is accomplished with the spin-livecd directive. Like all other Koji directives, usage information is available with --help:

Usage: koji spin-livecd [options] <target> <arch> <kickstart-file>
(Specify the --help global option for a list of other help options)

Options:
  -h, --help           show this help message and exit
  --nowait             Don't wait on livecd creation
  --noprogress         Do not display progress of the upload
  --background         Run the livecd creation task at a lower priority
  --isoname=ISONAME    Use a custom name for the iso file
  --ksurl=SCMURL       The URL to the SCM containing the kickstart file
  --ksversion=VERSION  The syntax version used in the kickstart file
  --scratch            Create a scratch LiveCD image.
  --repo=REPO          Specify a comma-separated list of repos that will
                       override the repo used to install RPMs in the LiveCD
                       image. The  repo associated with the target is the
                       default.

The way LiveCDs are created in Koji is fairly straightforward. A chroot is initialized and populated with the packages and their dependencies in the livecd-build group. Next, the kickstart file is copied into it if it was provided from local storage. If not, it is checked out into it from an SCM. It is then modified to use the repo associated with the build tag for the target specified in the command unless the --repo option was given. Both the original and the modified kickstart files are saved as part of the output for the task for later review. A livecd command is executed as such (using the mock('--chroot', ...) method): livecd-creator -f <filename> -c <kickstart-file> -d -v --logfile=/tmp/livecd.log --cache=/tmp/koji-livecd. Once that completes, the image file is copied to /mnt/koji/images/livecd/$imageID if it is not a scratch image. Note that the logfile and cache directory were arbitrarily chosen and hardcoded.

Caveats

1. Firstly, word of caution about kickstart files and the %include macro. livecd-creator is smart enough to search the current directory of the submitted kickstart file if it has %include macros. If the kickstart specified to koji is from local storage, only that kickstart file will be copied into the chroot, and this creates a problem if it has %include macros, because the other kickstart files it needs will be inaccessible. This issue is not present when the kickstart file is retrieved from a remote SCM (such as the spin-kickstarts git repo), because the entire repository is checked out. Presumably it will include any other kickstart files the specified one is including in the same directory. A workaround for the issue would be to use ksflatten (from pykickstart) on kickstart files with %include macros that are going to be submitted to koji from the user's local disk.

2. Another issue is if the kickstart file passed to livecd-creator defines repositories that Koji isn't aware of, and pulls packages from those repositories, the task will fail when storing RPM information to the database. This is because packages may be installed into the image that Koji cannot resolve the origin of, which causes the task to fail out. The way we sidestep this issue is by overriding the repos defined in the kickstart file with the repositories associated with the build target supplied on the command line. This is achieved using pykickstart. (kojid is now dependent on it with these changes) Note that if you use --repo though, you may run into this issue.

3. Package Groups in the kickstart file cause a problem if the Koji repos do not define them, which they most likely don't since Koji's comps.xml is based on the "groups" set up from the CLI. livecd-creator's behavior is to ignore package groups that are not defined in the repo it is using, so this can be troublesome when creating the image since packages could be left out. There are two possible workarounds: do not use package groups in the kickstart file and just specify a huge list of packages, or use --repo and specify a repo that does have a comps.xml that defines the groups it uses. Just make sure that repo is also one that Koji knows about. (see #2)

4. LiveCD images cannot be populated using local repos cannot use signed RPMs, because the repos Koji creates never have signed RPMs. You'll have to use external repos.

5. Debuginfo RPMs are tricky to get into LiveCDs as well, because by default the local repos generated by Koji do not have them. You can include them on a regeneration, but the next time Kojira regenerates them, they'll be gone again. The workaround is to regen the repo using koji call newRepo "debuginfo=True" just before building the LiveCD.