Koji/BuildingImages: Difference between revisions

Revision as of 18:28, 1 October 2010

This page documents how images can be built in Koji. It covers the supported use cases, how to configure your Koji instance to build them, and how to build them once it is ready. This is a very new feature and there hasn't been much external testing yet, so please keep that mind if you run into difficulty. Be certain to read the next section to be certain your use case is supported as well. For additional questions and information, ask around in #koji on FreeNode IRC and sign up to the right mailing lists as the Koji project website dictates.

Supported Images & Caveats

The way images are created in Koji is fairly straightforward. A chroot is initialized and populated with the packages and their dependencies from the livecd-build or appliance-build package group. Next, the kickstart file is copied into it if it was provided from local storage. If not, it is checked out into it from an SCM. It is then modified to use the repo associated with the build tag for the target specified in the command unless the --repo option was given. Both the original and the modified kickstart files are saved as part of the output for the task for later review. A livecd-creator or appliance-creator command is executed using the mock('--chroot', ...) method. This produces the desired image which is uploaded to /mnt/koji/images/<image>/$imageID if it is not a scratch image.

%include macros

A word of caution about kickstart files and the %include macro. livecd-creator and appliance-creator are smart enough to search the current directory of the submitted kickstart file if it has %include macros. If the kickstart specified to koji is from local storage, only that kickstart file will be copied into the chroot, and this creates a problem if it has %include macros, because the other kickstart files it needs will be inaccessible. This issue is not present when the kickstart file is retrieved from a remote SCM (such as the spin-kickstarts git repo), because the entire repository is checked out. Presumably it will include any other kickstart files the specified one is including in the same directory. A workaround for the issue would be to use ksflatten (from pykickstart) on kickstart files with %include macros that are going to be submitted to koji from the user's local disk.

Package Groups in the Kickstart File

Package Groups in the kickstart file cause a problem if the Koji repos do not define them, which they most likely don't since Koji's comps.xml is based on the "groups" set up from the CLI. {livecd,appliance}-creator's behavior is to ignore package groups that are not defined in the repo it is using, so this can be troublesome when creating the image since packages could be left out. There are a couple possible workarounds:

do not use package groups in the kickstart file and just specify a huge list of packages
use --repo and specify a repo that does have a comps.xml that defines the groups it uses

Only Include RPMs Built in Koji

The image building tasks will fail if your image tries to include a package that was not built in your build system. This is because the package does not have any origin information stored in Koji's database. The repos defined in the kickstart will automatically be overridden with the repo of the build tag for the build target, unless you use the --repo option. Since only packages you have built (or include from an external repo) should be there, you should never have this problem unless you use --repo.

No Signed or Debuginfo RPMs in Koji's Build Tags

If you need signed RPMs or debuginfo RPMs, you will run into trouble because Koji does not keep those in its build tag repos. The only work around for this is to create a repo yourself that includes these RPMs and then use --repo. This will force the image to take RPMs from that repo.

LiveCD-Specific Caveats

While livecd-tools does support building on SELinux disabled hosts, you can run into denials when booting if you create and use new files in the %post section of your kickstart file. If you do, you should either set the labels appropriately at the end of the %post section, or instigate an autorelabel at boot time.

Appliance-Specific Caveats

Koji only supports Appliances that are single disk images. If your kickstart file contains "part" commands for multiple devices, the task will fail. Patches are welcome to remove this limitation.

Build System Preparation

This section assumes you have know-how required to install and configure a new instance of Koji, and that you have already done so. You can learn how to do so here if you need to. Please ensure you are using the latest version of the software and that your database schema is updated as well. You should also have some familiarity with how livecd-creator and appliance-creator work.

Follow this procedure step by step to get things prepared they way they need to be. Italics indicate commands to run on the console. Any time you see <image>, substitute in the desired image type. This can be either "livecd" or "appliance".

mkdir /mnt/koji/images: Create a directory to put the finished images. This should be done on the system running kojihub.
chown apache:apache /mnt/koji/images: Let apache read and write to it.
Version 1.3.2 introduces another PythonOption in kojiweb.conf: KojiImagesURL. This option is set to the URL where completed LiveCD images can be downloaded from. Make sure that your apache server is allowing access to that location.
You need to set the topurl option in /etc/kojid/kojid.conf. Usually it looks something like http://your.domain.com/koji
koji add-host-to-channel <your-host> livecd: add a builder to the livecd channel
koji grant-permission <image> <user>: grant the permission to build an image type to a user. This step is optional since admins have all permissions.
You will need a tag and target to build the images from. The yum repo generated for the build tag of the target is what Koji will use to populate the LiveCDs with by default. (the alternative is to use the --repo option, more on that later)
koji add-group <build-tag> <image>-build: add the <image>-build group
koji add-group-pkg <build-tag> <image>-build <pkg> ...: add packages to the <image>-build group. These package lists vary has packages and dependencies change. As of October, 2010 for Fedora 13 the needed packages for each image type are:
- livecd: bash, coreutils, dbus-python, dmraid, livecd-tools, selinux-policy-targeted, yum
- appliance: appliance-tools, bash, coreutils, grub, parted, selinux-policy, policycoreutils

Building an Image

Building a LiveCD in Koji is accomplished with the spin-livecd directive. Like all other Koji directives, usage information is available with --help:

Usage: koji spin-livecd [options] <target> <arch> <kickstart-file>
(Specify the --help global option for a list of other help options)

Options:
  -h, --help           show this help message and exit
  --nowait             Don't wait on livecd creation
  --noprogress         Do not display progress of the upload
  --background         Run the livecd creation task at a lower priority
  --isoname=ISONAME    Use a custom name for the iso file
  --ksurl=SCMURL       The URL to the SCM containing the kickstart file
  --ksversion=VERSION  The syntax version used in the kickstart file
  --scratch            Create a scratch LiveCD image.
  --repo=REPO          Specify a comma-separated list of repos that will
                       override the repo used to install RPMs in the LiveCD
                       image. The  repo associated with the target is the
                       default.

For appliances, use the spin-appliance directive.

Usage: kojay spin-appliance [options] <target> <arch> <kickstart-file>
(Specify the --help global option for a list of other help options)

Options:
  -h, --help            show this help message and exit
  --wait                Wait on the appliance creation, even if running in the
                        background
  --nowait              Don't wait on appliance creation
  --noprogress          Do not display progress of the upload
  --background          Run the appliance creation task at a lower priority
  --ksurl=SCMURL        The URL to the SCM containing the kickstart file
  --ksversion=VERSION   The syntax version used in the kickstart file
  --scratch             Create a scratch appliance
  --tag                 Interpret <target> as a tag instead
  --repo=REPO           Specify a comma-separated list of repos that will
                        override the repo used to install RPMs in the
                        appliance. The build tag repo associated with the
                        target is the default.
  --name=NAME           Specify the appliance file name
  --version=VERSION     Set an appliance version, used in .xml metadata
  --release=RELEASE     Set an appliance release number, used in .xml metadata
  --vmem=VMEM           Set the amount of virtual memory in the appliance in
                        MB, default is 512
  --vcpu=VCPU           Set the number of virtual cpus in the appliance,
                        default is 1
  --format=DISK_FORMAT  Disk format, default is raw. Other options are qcow,
                        qcow2, and vmx.

@@ Line 1: / Line 1: @@
-This page documents a few additional steps to enable building LiveCDs in an existing Koji installation, how to build one, and where the code will grow from here. This is a very new feature and there hasn't been much external testing yet, so please keep that mind if you run into difficulty. For additional questions and information, ask around in #koji on FreeNode IRC and sign up to the right mailing lists as the [https://fedorahosted.org/koji/wiki Koji project website] dictates.
+This page documents how images can be built in Koji. It covers the supported use cases, how to configure your Koji instance to build them, and how to build them once it is ready. This is a very new feature and there hasn't been much external testing yet, so please keep that mind if you run into difficulty. Be certain to read the next section to be certain your use case is supported as well. For additional questions and information, ask around in #koji on FreeNode IRC and sign up to the right mailing lists as the [https://fedorahosted.org/koji/wiki Koji project website] dictates.
-== Build System Preparation ==
+== Supported Images & Caveats ==
-This section assumes you have know-how required to install and configure a new instance of Koji, and that you have already done so. You can learn how to do so [[koji/ServerHowTo| here]] if you need to. You should be working with version 1.3.2 or later of the software. If you upgraded to that version there may be database-related steps you'll have to complete, as detailed below. You should also have some familiarity with how livecd-creator from the livecd-tools package works.
+The way images are created in Koji is fairly straightforward. A chroot is initialized and populated with the packages and their dependencies from the livecd-build or appliance-build package group. Next, the kickstart file is copied into it if it was provided from local storage. If not, it is checked out into it from an SCM. It is then modified to use the repo associated with the build tag for the target specified in the command unless the --repo option was given. Both the original and the modified kickstart files are saved as part of the output for the task for later review. A livecd-creator or appliance-creator command is executed using the mock('--chroot', ...) method. This produces the desired image which is uploaded to /mnt/koji/images/<image>/$imageID if it is not a scratch image.
+=== %include macros ===
+A word of caution about kickstart files and the %include macro. livecd-creator and appliance-creator are smart enough to search the current directory of the submitted kickstart file if it has %include macros. If the kickstart specified to koji is from local storage, only that kickstart file will be copied into the chroot, and this creates a problem if it has %include macros, because the other kickstart files it needs will be inaccessible. This issue is not present when the kickstart file is retrieved from a remote SCM (such as the spin-kickstarts git repo), because the entire repository is checked out. Presumably it will include any other kickstart files the specified one is including in the same directory. A workaround for the issue would be to use ksflatten (from pykickstart) on kickstart files with %include macros that are going to be submitted to koji from the user's local disk.
+=== Package Groups in the Kickstart File ===
+Package Groups in the kickstart file cause a problem if the Koji repos do not define them, which they most likely don't since Koji's comps.xml is based on the "groups" set up from the CLI. {livecd,appliance}-creator's behavior is to ignore package groups that are not defined in the repo it is using, so this can be troublesome when creating the image since packages could be left out. There are a couple possible workarounds:
+* do not use package groups in the kickstart file and just specify a huge list of packages
+* use --repo and specify a repo that does have a comps.xml that defines the groups it uses
+=== Only Include RPMs Built in Koji ===
+The image building tasks will fail if your image tries to include a package that was not built in your build system. This is because the package does not have any origin information stored in Koji's database. The repos defined in the kickstart will automatically be overridden with the repo of the build tag for the build target, unless you use the --repo option. Since only packages you have built (or include from an external repo) should be there, you should never have this problem unless you use --repo.
+=== No Signed or Debuginfo RPMs in Koji's Build Tags ===
+If you need signed RPMs or debuginfo RPMs, you will run into trouble because Koji does not keep those in its build tag repos. The only work around for this is to create a repo yourself that includes these RPMs and then use --repo. This will force the image to take RPMs from that repo.
+=== LiveCD-Specific Caveats ===
-Follow this procedure step by step to get things prepared they way they need to be. Italics indicate commands to run on the console.
+While livecd-tools does support building on SELinux disabled hosts, you can run into denials when booting if you create and use new files in the %post section of your kickstart file. If you do, you should either set the labels appropriately at the end of the %post section, or instigate an autorelabel at boot time.
-=== Database Prep ===
+=== Appliance-Specific Caveats ===
-'''Skip this section if you freshly installed Koji 1.3.2 or later, or updated the schema when you upgraded.'''
+Koji only supports Appliances that are single disk images. If your kickstart file contains "part" commands for multiple devices, the task will fail. Patches are welcome to remove this limitation.
-You'll have to add a few tables and entries to the database that are used to track information about the images created in Koji. The SQL looks like this:
+== Build System Preparation ==
-<pre>
-CREATE TABLE imageinfo (
-       id SERIAL NOT NULL PRIMARY KEY,
-       task_id INTEGER NOT NULL REFERENCES task(id),
-       filename TEXT NOT NULL,
-       filesize BIGINT NOT NULL,
-       arch VARCHAR(16) NOT NULL,
-       hash TEXT NOT NULL,
-       mediatype TEXT NOT NULL
-) WITHOUT OIDS;
-CREATE INDEX imageinfo_task_id on imageinfo(task_id);
-CREATE TABLE imageinfo_listing (
-       image_id INTEGER NOT NULL REFERENCES imageinfo(id),
-       rpm_id INTEGER NOT NULL REFERENCES rpminfo(id),
-       UNIQUE (image_id, rpm_id)
-) WITHOUT OIDS;
-CREATE INDEX imageinfo_listing_rpms on imageinfo_listing(rpm_id);
-INSERT INTO channels (name) VALUES ('livecd');
+This section assumes you have know-how required to install and configure a new instance of Koji, and that you have already done so. You can learn how to do so [[koji/ServerHowTo| here]] if you need to. Please ensure you are using the latest version of the software and that your database schema is updated as well. You should also have some familiarity with how [[FedoraLiveCD|livecd-creator]] and [[ApplianceTools|appliance-creator]] work.
-INSERT INTO permissions (name) VALUES ('livecd');
-</pre>
-=== System Prep ===
+Follow this procedure step by step to get things prepared they way they need to be. Italics indicate commands to run on the console. Any time you see <image>, substitute in the desired image type. This can be either "livecd" or "appliance".
 * ''mkdir /mnt/koji/images'': Create a directory to put the finished images. This should be done on the system running kojihub.
 * ''chown apache:apache /mnt/koji/images'': Let apache read and write to it.
-* ''koji add-host-to-channel <your-host> livecd'': add a builder to the livecd channel
-* ''koji grant-permission livecd <user>'': grant the permission to build livecds to a user. This step is optional since admins have all permissions, technically.
-* ''koji add-group <build-tag> livecd-build'': add the livecd-build group
-* ''koji add-group-pkg <build-tag> livecd-build <pkg> ...'': add packages to the livecd-build group. The packages should match what you have in the "build" group, plus: python-dbus, yum, squashfs-tools, livecd-tools, and selinux-targeted-policy as of Fedora 10.
-* You will need a tag and target to build the LiveCDs from. The yum repo generated for the build tag of the target is what Koji will use to populate the LiveCDs with by default. (the alternative is to use the --repo option, more on that later)
 * Version 1.3.2 introduces another PythonOption in kojiweb.conf: KojiImagesURL. This option is set to the URL where completed LiveCD images can be downloaded from. Make sure that your apache server is allowing access to that location.
 * You need to set the topurl option in /etc/kojid/kojid.conf. Usually it looks something like <nowiki>http://your.domain.com/koji</nowiki>
+* ''koji add-host-to-channel <your-host> livecd'': add a builder to the livecd channel
+* ''koji grant-permission <image> <user>'': grant the permission to build an image type to a user. This step is optional since admins have all permissions.
+* You will need a tag and target to build the images from. The yum repo generated for the build tag of the target is what Koji will use to populate the LiveCDs with by default. (the alternative is to use the --repo option, more on that later)
+* ''koji add-group <build-tag> <image>-build'': add the <image>-build group
+* ''koji add-group-pkg <build-tag> <image>-build <pkg> ...'': add packages to the <image>-build group. These package lists vary has packages and dependencies change. As of October, 2010 for Fedora 13 the needed packages for each image type are:
+** livecd: bash, coreutils, dbus-python, dmraid, livecd-tools, selinux-policy-targeted, yum
+** appliance: appliance-tools, bash, coreutils, grub, parted, selinux-policy, policycoreutils
-== Building a LiveCD ==
+== Building an Image ==
-Building a Livecd in Koji is accomplished with the spin-livecd directive. Like all other Koji directives, usage information is available with --help:
+Building a LiveCD in Koji is accomplished with the spin-livecd directive. Like all other Koji directives, usage information is available with --help:
 <pre>
@@ Line 69: / Line 74: @@
 </pre>
-The way LiveCDs are created in Koji is fairly straightforward. A chroot is initialized and populated with the packages and their dependencies in the livecd-build group. Next, the kickstart file is copied into it if it was provided from local storage. If not, it is checked out into it from an SCM. It is then modified to use the repo associated with the build tag for the target specified in the command unless the --repo option was given. Both the original and the modified kickstart files are saved as part of the output for the task for later review. A livecd command is executed as such (using the mock('--chroot', ...) method): ''livecd-creator -f <filename> -c <kickstart-file> -d -v --logfile=/tmp/livecd.log --cache=/tmp/koji-livecd''. Once that completes, the image file is copied to /mnt/koji/images/livecd/$imageID if it is not a scratch image. Note that the logfile and cache directory were arbitrarily chosen and hardcoded.
+For appliances, use the spin-appliance directive.
-=== Caveats ===
+<pre>
+Usage: kojay spin-appliance [options] <target> <arch> <kickstart-file>
+(Specify the --help global option for a list of other help options)
-. Firstly, word of caution about kickstart files and the %include macro. livecd-creator is smart enough to search the current directory of the submitted kickstart file if it has %include macros. If the kickstart specified to koji is from local storage, only that kickstart file will be copied into the chroot, and this creates a problem if it has %include macros, because the other kickstart files it needs will be inaccessible. This issue is not present when the kickstart file is retrieved from a remote SCM (such as the spin-kickstarts git repo), because the entire repository is checked out. Presumably it will include any other kickstart files the specified one is including in the same directory. A workaround for the issue would be to use ksflatten (from pykickstart) on kickstart files with %include macros that are going to be submitted to koji from the user's local disk.
+Options:
+  -h, --help            show this help message and exit
-. Another issue is if the kickstart file passed to livecd-creator defines repositories that Koji isn't aware of, and pulls packages from those repositories, the task will fail when storing RPM information to the database. This is because packages may be installed into the image that Koji cannot resolve the origin of, which causes the task to fail out. The way we sidestep this issue is by overriding the repos defined in the kickstart file with the repositories associated with the build target supplied on the command line. This is achieved using pykickstart. (kojid is now dependent on it with these changes) Note that if you use --repo though, you may run into this issue.
+  --wait                Wait on the appliance creation, even if running in the
+                        background
-. Package Groups in the kickstart file cause a problem if the Koji repos do not define them, which they most likely don't since Koji's comps.xml is based on the "groups" set up from the CLI. livecd-creator's behavior is to ignore package groups that are not defined in the repo it is using, so this can be troublesome when creating the image since packages could be left out. There are two possible workarounds: do not use package groups in the kickstart file and just specify a huge list of packages, or use --repo and specify a repo that does have a comps.xml that defines the groups it uses. Just make sure that repo is also one that Koji knows about. (see #2)
+  --nowait              Don't wait on appliance creation
+  --noprogress          Do not display progress of the upload
-. LiveCD images cannot be populated using local repos cannot use signed RPMs, because the repos Koji creates never have signed RPMs. You'll have to use external repos.
+  --background          Run the appliance creation task at a lower priority
+  --ksurl=SCMURL        The URL to the SCM containing the kickstart file
-. Debuginfo RPMs are tricky to get into LiveCDs as well, because by default the local repos generated by Koji do not have them. You can include them on a regeneration, but the next time Kojira regenerates them, they'll be gone again. The workaround is to regen the repo using ''koji call newRepo "debuginfo=True"'' just before building the LiveCD.
+  --ksversion=VERSION   The syntax version used in the kickstart file
+  --scratch             Create a scratch appliance
+  --tag                 Interpret <target> as a tag instead
+  --repo=REPO           Specify a comma-separated list of repos that will
+                        override the repo used to install RPMs in the
+                        appliance. The build tag repo associated with the
+                        target is the default.
+  --name=NAME           Specify the appliance file name
+  --version=VERSION     Set an appliance version, used in .xml metadata
+  --release=RELEASE     Set an appliance release number, used in .xml metadata
+  --vmem=VMEM           Set the amount of virtual memory in the appliance in
+                        MB, default is 512
+  --vcpu=VCPU           Set the number of virtual cpus in the appliance,
+                        default is 1
+  --format=DISK_FORMAT  Disk format, default is raw. Other options are qcow,
+                        qcow2, and vmx.
+</pre>

Search