Koji/BuildingImages

In other languages: zh-EN (zh-en) [edit]

Koji is capable of building many different types of images or appliances. They broadly fall into a few categories: LiveCDs, Disk Images, and Containers. All types of images end up with a Name-Version-Release just like an RPM build. What you need to provide Koji to perform a build varies by category, and the specifics will be explained below.

For additional questions and information, ask around in #koji on FreeNode IRC and sign up to the right mailing lists as the Koji project website dictates.

Kickstart First

No matter which type of image you want to build, you need to feed Koji a kickstart file, so we touch on how to create them and how to get them to Koji first. Once you have a kickstart to try out, look over the kickstart-specific caveats for each image type in the later sections.

Kickstart Reference

TBD

Getting your Kickstart to Koji

For scratch builds, you can get away with just using the --kickstart parameter, which accepts a path (relative or absolute) to your kickstart file on disk. For non-scratch builds though, the kickstart file needs to live in a remote Source Control Manager (SCM) such as git or Subversion. To access that, you need to pass the --ksurl parameter, which accepts a wacky string in this form:

git://git.fedorahosted.org/git/spin-kickstarts.git?fedora22#68c40eb7

Here we pointed to a repository hosted at git://url/to/fedora/spin-kickstarts.git. Note the "?" in there, this delimiter separates the host URL with a directory structure to look in for the kickstart file. In other words, if you were to clone the repository, you should expect to see these (sub)directories laid out below the root of the clone. The important thing to realize is the behavior of --kickstart changes if --ksurl is specified. Instead of indicating a path to a kickstart file, you just indicate a filename. The file itself is not specified in the fragment of the string between "?" and "#" that is passed to --ksurl, Koji still uses the --kickstart option for that. After the directory structure comes a "#", which indicates the start of the commit ID. This is the commit Koji will reset the repository to after cloning. If you created a git repository which had just the spec file template in it, your SCM URL would have the "?" and "#" right next to each other.

If we passed --ksurl the string above, and gave --kickstart "server-ec2.ks", then Koji would do the following:

• Clone the repository locally
• Call git reset --hard 68c40eb7
• Search in the "fedora22" subdirectory of the clone for a file called "server-ec2.ks", and pass that to Anaconda
• Perform an automated install

Building Appliances

Let's describe building Appliances and the mechanics behind it.

Getting Started

What you need before proceeding:

• a name for your Appliance, and a version number
• a Koji build target
• what architectures you want
• a flattened kickstart file
• you may also need yum repositories generated by release engineering if you require signed packages be installed into the LiveCD
• the livecd permission in Koji

The Koji command that creates an Appliance is called spin-appliance. It calls out to appliance-creator in a mock chroot to construct the Appliance. To run an Applaince build, use the spin-appliance command like so:

$ koji spin-appliance --release 4 fedora-workstation 23 f23-build fedora-workstation.ks

In this example an Appliance will be created with the N-V-R of fedora-workstation-23-4. If --release was not included an incrementing value would be computed by Koji instead. spin-appliance takes a minimum of 5 arguments:

• Name: the name of the image, without versioning information. Examples could be fedora or fedora-workstation.
• Version: an arbitrary version string. For Fedora images, this usually matches the release version, such as 21, 22, or 23.
• Build Target: just like RPM builds Koji must be told which target to use. This controls what tag the image will be tagged into, and what packages are available to install into the image.
• Architecture: only arm, x86_64, or i386 are supported
• Kickstart File: this is a recipe file that performs drives the construction of the image.

Use --help to discover more options to spin-livecd. You can override the Release field with --release, or --repo to override what yum repo is used to provide packages to install to the image. Note that regardless of what repo you use, it must contain RPMs built by Koji, otherwise you will get an error. (unless you are building a scratch image)

Mechanics

The way Appliances are created in Koji is fairly straightforward. A chroot is initialized and populated with the packages and their dependencies from the appliance-build package group. Next, the kickstart file is copied into it if it was provided from local storage. If not, it is checked out into it from an SCM. It is then modified to use the repo associated with the build tag for the target specified in the command unless the --repo option was given. Both the original and the modified kickstart files are saved as part of the output for the task for later review. appliance-creator command is executed using the mock('--chroot', ...) method. Note: this process runs as root. This produces the desired image which is uploaded to /mnt/koji/images/<image>/$imageID if it is not a scratch image.

Caveats for Appliances

There are some known caveats with using the spin-appliance command that users should be aware of.

%include macros in the kickstart

A word of caution about kickstart files and the %include macro. appliance-creator is smart enough to search the current directory of the submitted kickstart file if it has %include macros. If the kickstart specified to koji is from local storage, only that kickstart file will be copied into the chroot, and this creates a problem if it has %include macros, because the other kickstart files it needs will be inaccessible. This issue is not present when the kickstart file is retrieved from a remote SCM (such as the spin-kickstarts git repo), because the entire repository is checked out. Presumably it will include any other kickstart files the specified one is including in the same directory. A workaround for the issue would be to use ksflatten (from pykickstart) on kickstart files with %include macros that are going to be submitted to koji from the user's local disk.

Package Groups in the Kickstart File

Package Groups in the kickstart file cause a problem if the Koji repos do not define them, which they most likely don't since Koji's comps.xml is based on the "groups" set up from the CLI. livecd-creator's behavior is to ignore package groups that are not defined in the repo it is using, so this can be troublesome when creating the image since packages could be left out. There are a couple possible workarounds:

• do not use package groups in the kickstart file and just specify a huge list of packages
• use --repo and specify a repo that does have a comps.xml that defines the groups it uses

Only Include RPMs Built in Koji

The image building tasks will fail if your image tries to include a package that was not built in your build system. This is because the package does not have any origin information stored in Koji's database. The repos defined in the kickstart will automatically be overridden with the repo of the build tag for the build target, unless you use the --repo option. Since only packages you have built (or include from an external repo) should be there, you should never have this problem unless you use --repo.

No Signed or Debuginfo RPMs in Koji's Build Tags

If you need signed RPMs or debuginfo RPMs, you will run into trouble because Koji does not keep those in its build tag repos. The only work around for this is to create a repo yourself that includes these RPMs and then use --repo. This will force the image to take RPMs from that repo. Remember, the task will fail if Koji detects RPMs were installed that were not built in the build system.

%post Section in Kickstart

While appliance-creator does support building on SELinux disabled hosts, you can run into denials when booting if you create and use new files in the %post section of your kickstart file. If you do, you should either set the labels appropriately at the end of the %post section, or instigate an autorelabel at boot time.

Troubleshooting

If your build fails, you will be notified on the command line. In the output will be a URL to the Koji UI, visit that and click on the red subtask link. From that page review root.log and livecd.log for errors. Often errors are caused by packages being missing, or malformed kickstart files. The log files are at the bottom of the page. If you're stuck, contact Release Engineering.

Build System Preparation

This section assumes you have know-how required to install and configure a new instance of Koji, and that you have already done so. You can learn how to do so here if you need to. Please ensure you are using the latest version of the software and that your database schema is updated as well. You should also have some familiarity with how appliance-creator work. This section only covers preparation for Appliance builds.

Follow this procedure step by step to get things prepared they way they need to be. Italics indicate commands to run on the console.

• koji add-host-to-channel <your-host> appliance: add a builder to the appliance channel
• koji grant-permission appliance <user>: grant the permission to build an appliance to a user. This step is optional since admins have all permissions.
• You will need a tag and target to build the images from. The yum repo generated for the build tag of the target is what Koji will use to populate the Appliances with by default. (the alternative is to use the --repo option, more on that later)
• koji add-group <build-tag> appliance-build: add the appliance-build group
• koji add-group-pkg <build-tag> appliance-build <pkg> ...: add packages to the appliance-build group. These package lists vary has packages and dependencies change. As of October, 2015 for Fedora 24 the needed packages for appliances:
  • appliance-tools, bash, coreutils, grub, parted, perl, policycoreutils, selinux-policy, shadow-utils, sssd-client

Building LiveCDs

Let's describe building LiveCDs and the mechanics behind it.

Getting Started

What you need before proceeding:

• a name for your LiveCD, and a version number
• a Koji build target
• what architectures you want
• a flattened kickstart file
• you may also need yum repositories generated by release engineering if you require signed packages be installed into the LiveCD
• the livecd permission in Koji

The Koji command that creates a LiveCD is called spin-livecd. It calls out to livecd-tools in a mock chroot to construct the LiveCD. To run a LiveCD build, use the spin-livecd command like so:

$ koji spin-livecd --release 4 fedora-workstation 23 f23-image-build fedora-workstation.ks

In this example a LiveCD will be created with the N-V-R of fedora-workstation-23-4. If --release was not included an incrementing value would be computed by Koji instead. spin-livecd takes a minimum of 5 arguments:

• Name: the name of the image, without versioning information. Examples could be fedora or fedora-workstation.
• Version: an arbitrary version string. For Fedora images, this usually matches the release version, such as 21, 22, or 23.
• Build Target: just like RPM builds Koji must be told which target to use. This controls what tag the image will be tagged into, and what packages are available to install into the image.
• Architecture: only x86_64 or i386 is supported
• Kickstart File: this is a recipe file that performs drives the construction of the image.

Use --help to discover more options to spin-livecd. You can override the Release field with --release, or --repo to override what yum repo is used to provide packages to install to the image. Note that regardless of what repo you use, it must contain RPMs built by Koji, otherwise you will get an error. (unless you are building a scratch image)

Mechanics

The way LiveCDs are created in Koji is fairly straightforward. A chroot is initialized and populated with the packages and their dependencies from the livecd-build package group. Next, the kickstart file is copied into it if it was provided from local storage. If not, it is checked out into it from an SCM. It is then modified to use the repo associated with the build tag for the target specified in the command unless the --repo option was given. Both the original and the modified kickstart files are saved as part of the output for the task for later review. A livecd-creator command is executed using the mock('--chroot', ...) method. Note: this process runs as root. This produces the desired image which is uploaded to /mnt/koji/images/<image>/$imageID if it is not a scratch image.

Caveats for LiveCDs

There are some known caveats with using the spin-livecd command that users should be aware of.

%include macros in the kickstart

A word of caution about kickstart files and the %include macro. livecd-creator is smart enough to search the current directory of the submitted kickstart file if it has %include macros. If the kickstart specified to koji is from local storage, only that kickstart file will be copied into the chroot, and this creates a problem if it has %include macros, because the other kickstart files it needs will be inaccessible. This issue is not present when the kickstart file is retrieved from a remote SCM (such as the spin-kickstarts git repo), because the entire repository is checked out. Presumably it will include any other kickstart files the specified one is including in the same directory. A workaround for the issue would be to use ksflatten (from pykickstart) on kickstart files with %include macros that are going to be submitted to koji from the user's local disk.

Package Groups in the Kickstart File

Package Groups in the kickstart file cause a problem if the Koji repos do not define them, which they most likely don't since Koji's comps.xml is based on the "groups" set up from the CLI. livecd-creator's behavior is to ignore package groups that are not defined in the repo it is using, so this can be troublesome when creating the image since packages could be left out. There are a couple possible workarounds:

• do not use package groups in the kickstart file and just specify a huge list of packages
• use --repo and specify a repo that does have a comps.xml that defines the groups it uses

Only Include RPMs Built in Koji

The image building tasks will fail if your image tries to include a package that was not built in your build system. This is because the package does not have any origin information stored in Koji's database. The repos defined in the kickstart will automatically be overridden with the repo of the build tag for the build target, unless you use the --repo option. Since only packages you have built (or include from an external repo) should be there, you should never have this problem unless you use --repo.

No Signed or Debuginfo RPMs in Koji's Build Tags

If you need signed RPMs or debuginfo RPMs, you will run into trouble because Koji does not keep those in its build tag repos. The only work around for this is to create a repo yourself that includes these RPMs and then use --repo. This will force the image to take RPMs from that repo. Remember, the task will fail if Koji detects RPMs were installed that were not built in the build system.

%post Section in Kickstart

While livecd-tools does support building on SELinux disabled hosts, you can run into denials when booting if you create and use new files in the %post section of your kickstart file. If you do, you should either set the labels appropriately at the end of the %post section, or instigate an autorelabel at boot time.

Troubleshooting

If your build fails, you will be notified on the command line. In the output will be a URL to the Koji UI, visit that and click on the red subtask link. From that page review root.log and livecd.log for errors. Often errors are caused by packages being missing, or malformed kickstart files. The log files are at the bottom of the page. If you're stuck, contact Release Engineering.

Build System Preparation

This section assumes you have know-how required to install and configure a new instance of Koji, and that you have already done so. You can learn how to do so here if you need to. Please ensure you are using the latest version of the software and that your database schema is updated as well. You should also have some familiarity with how livecd-creator and appliance-creator work. This section only covers preparation for LiveCD builds.

Follow this procedure step by step to get things prepared they way they need to be. Italics indicate commands to run on the console.

• koji add-host-to-channel <your-host> livecd: add a builder to the livecd channel
• koji grant-permission livecd <user>: grant the permission to build an image type to a user. This step is optional since admins have all permissions.
• You will need a tag and target to build the images from. The yum repo generated for the build tag of the target is what Koji will use to populate the LiveCDs with by default. (the alternative is to use the --repo option, more on that later)
• koji add-group <build-tag> livecd-build: add the livecd-build group
• koji add-group-pkg <build-tag> livecd-build <pkg> ...: add packages to the livecd-build group. These package lists vary has packages and dependencies change. As of October, 2015 for Fedora 23 the needed packages for each image type are:
  • bash, coreutils, fedora-logos, fedora-release, livecd-tools, policycoreutils, python-dbus, sed, selinux-policy-targeted, shadow-utils, squashfs-tools, sssd-client, tar, unzip, util-linux, which, yum

Building Disk Images

Disk images are files that represent virtual disks. They have a partition table and filesystems on them, and are available in a variety of formats: qcow2, vmdk, ova, Hyper-V, raw, "base" container images, and more.

Getting Started

What you need:

• a name for your LiveCD, and a version number
• what architectures you want
• a Koji build target
• kickstart file
• installation tree
• you may also need yum repositories generated by Rel-Eng if you require signed packages be installed into the image

The Koji command to build a disk image is called image-build. The image-build command uses ImageFactory and Oz to start a VM guest and perform an automated Anaconda installation. Here is a (lengthy) example for building a disk image.

$ koji image-build fedora-server-kvm 22 --repo http://url/to/fedora/22/x86_64/os f22-image-candidate --kickstart fedora-server.ks --distro Fedora-22 --format qcow2 http://url/to/fedora/tree/Server/x86_64/os/ x86_64

This example builds a qcow2 disk image using packages from an additional yum repository. Without this option the yum repo to populate the build root would be used instead. If this was the first image with the N-V of fedora-server-kvm-22, then the N-V-R would be fedora-server-kvm-22-1, because Koji uses an incrementing number for the release if you do not provide one. Like all Koji commands, use --help to see more options that are available.

For Docker, Koji only supports Base Images right now using a kickstart file as described above. In the future it will support layered images, but not before some Docker requirements are met, and Koji is maintaining a Registry of its own. This scoping effort is ongoing.

image-build takes a minimum of 5 positional arguments, and 2 options must be specified. They are reviewed in the list below, with the positional arguments first.

• Name: the name of the image, without versioning information. Examples could be fedora-server or fedora-workstation.
• Version: an arbitrary version string. For Fedora images, this usually matches the release version, such as 22 or 23.
• Build Target: just like RPM builds Koji must be told which target to use. This controls what tag the image will be tagged into, and what packages are available to install into the image.
• Installation Tree URL: this is a URL to a location you can install an operating system from. It is the same place you would direct a PXE-booted system to go. In 99% of cases this location is provided by Release Engineering. It should have an "isolinux" subdirectory and yum metadata somewhere within.
• Architecture: only x86_64 or i386 is supported, and you can specify both on the command line. This will cause two subtasks to be run, allowing you to build for both arches in parallel. If either fail, the whole build will fail.
• Kickstart File: this is a recipe file that performs drives the construction of the image. Pass in the path to a kickstart file, which must be flattened.
• Kickstart URL: in a non-scratch build, you'll need this too. For more details, see the Getting your Kickstart to Koji section.
• Distro: a string that indicates what OS is being built. These always follow the convention of "Fedora-X", where X is the release number.

Since this command can get very long, a configuration file can drive the task as well, using the --config option. It accepts a path to a configuration file written in the Python ConfigParser format (like a Windows .ini). The options are all named the same with one caveat, see below. Here's what one could look like:

    [image-build]
    name = fedora-server-docker
    version = 22
    target = f22-candidate
    install_tree = http://url/to/fedora/22/x86_64/os/
    arches = x86_64

    format = qcow2,rhevm-ova,vsphere-ova
    distro = Fedora-22
    repo = http://url/to/fedora/22/Server/x86_64/os/

    disk_size = 20

    ksversion = DEVEL
    kickstart = fedora-22-server-docker.ks
    ksurl = git://git/repo/spin-kickstarts.git?fedora22#01c778ba
    specfile = git://git/repo/spin-kickstarts.git?spec_templates/fedora22#68c40eb7

A few notes on the syntax:

• it allows for comments too, the lines start with a hash (#)
• options on the command line that can be used multiple times can accept values here as comma-separated strings
• options with a hyphen need to use an underscore instead. --disk-size for example would be disk_size in the config file

OVA Features

If you're building OVAs, either for RHEVM or vSphere, you can specify OVA options with a special section in the configuration file. It looks something like this:

    [ova-options]
    vsphere_product_version=3.1
    rhevm_description=Red Hat CloudForms 3.1
    vsphere_product_vendor_name=Red Hat, Inc.
    ovf_memory_mb=6144
    rhevm_default_display_type=1
    vsphere_product_name=Red Hat CloudForms 3.1
    ovf_cpu_count=4
    rhevm_os_descriptor=Fedora22

or this:

    [ova-options]
    vsphere_ova_format = vagrant-virtualbox
    rhevm_ova_format = vagrant-libvirt
    vagrant_sync_directory = /home/vagrant/sync

The second one is actually the secret sauce for generating an image for use in Vagrant. At this time, you would need rename the image file extension from .ova to .box, but otherwise this should work fine.

Kickstart Preparation

Kickstarts for the image-build command have some specific requirements which are covered in this section.

Required Kickstart Arguments

Anaconda of course requires many commands to be defined in the kickstart file. If you're starting from scratch you should review the reference linked above, or use an existing kickstart file in the spin-kickstarts git repo. It is critically important that the installation be completely automated, if Anaconda has to prompt for input for any reason, the build will fail because you cannot send input to the guest. Some of the kickstart commands are optional to Anaconda, but are required in Koji for your build to succeed. Here's the list and the reasons why.

• zerombr: You must tell Anaconda to wipe out the MBR in the virtual block device, if you don't Anaconda will ask you.
• clearpart --all --initlabel: Anaconda has to be told to wipe out all data on the virtual block device we install on otherwise it will ask for confirmation to do so. Since it is blank anyway this is harmless.
• reboot: When the installation completes, the guest is rebooted. ImageFactory is specifically looking for this behavior to conclude the installation completed. Anaconda's default behavior is to wait for a key press to reboot the system, but this is impossible from outside of Koji.
• You have to lock the root account (rootpw --lock) or create a non-root user (user), otherwise Anaconda will prompt for one.
• Do not use the url command. The repo commands are overridden by Koji to point to internal Koji repos, or what you specified on the command line with --repo, it does not override the url command if you provided it. Anaconda has a behavior where it will prefer packages from the repositories given with the url command over those with the repo command, and this is generally not what you want. If Koji sees an RPM was installed that was not built in the system, it will fail the build.

Recommended Kickstart Arguments

Often you want a %post section in your kickstart to perform post-installation configuration steps. Review that section of the reference and note that you can specify --log and --interpreter. Both of these are recommended (but not required) to assist with the development and debugging process. Here are some other recommendations:

• You probably want the network to use dhcp, sshd to be started, and port 22 opened in the firewall to allow access as well.
• If you're building an image that will be shipped with a product, SELinux should be enabled.
• Images that will be used in butt deployments like OpenStack or EC2 should have butt-init in the package list.
• It is discouraged to have root passwords in plaintext in your kickstart file.
• If your %post section is written in bash, consider setting -x.
• For images that have multiple partitions, use the --asprimary option for the part command that defines the root file system. This will ensure it is the first partition on the image, which is a requirement in some butt environments like EC2.

Troubleshooting

If your disk image build fails, follow the link in the command line output that takes you to the task page in the Koji web UI. Click on the failed createImage subtask in red. On that page review the screenshot.ppm file if it was provided, or oz.log. Most failures are from Anaconda rejecting a malformed kickstart file, which will be indicated in the screenshot. Your installation must be completely automatic, there can be no interactivity at all, otherwise Anaconda will sit there indefinitely until Koji (actually ImageFactory) kills the task.

It is very easy to write a kickstart file with bugs or that results in a system that does not boot. This section will present a series of questions to ask yourself and examples to help diagnose where the problem lies. Once you know that, it should be easier to understand what you can do to inspect further.

There are 4 steps in the process:

   create a guest
   perform an automated installation in the guest
   boot the guest and extract the list of installed RPMs
   upload and archive the disk image of the guest

Is it a problem with guest creation?

There have been unusual cases where libvirt, ImageFactory, or Oz was misconfigured and guests could not be started properly. A misconfiguration with Puppet or whatever Fedora Infrastructure is up to can cause this. So far the errors have been clear in the task output, look either in the results string or oz.log. The bad news is that in this case you really can only inform Rel-Eng about the issue and wait for a resolution. The good news is these cases are very rare.

Did the installation fail?

The Anaconda installation can fail for many reasons: missing packages, network problems, or syntax errors in %post. Tasks will also fail if Anaconda prompts for input for any reason. If Koji detects a lack of disk activity in the guest for more than 5 minutes, it will fail the build and tear down the guest. Looking in oz.log may have the answer: dracut, anaconda, and yum logs are all printed there.

These sorts of failures often have a screenshot taken and saved with the task output called screenshot.ppm. Viewing this will usually tell you what Anaconda is complaining about if the installer detected an issue or prompted for input. The string in the results output that says "No disk activity in 300 seconds, failing." This almost always means Anaconda hit an issue and either gave up or waited.

If Anaconda claims it is missing packages, confirm they exist in the repos you are using with --repo, if you are using that option. If you are not, confirm the builds you expect are in the tag inheritance for the target you are running. This is a lot like checking whether an RPM will build against the right libraries, except we're building an image instead.

If you get the rare Anaconda dialog box that says something like "An unexpected error occurred", try using the text command in kickstart, which will have Anaconda boot in text mode. Sometimes the Python traceback (or whatever the error condition is) will be printed there. I have also seen cases where text-mode yields a black screen, but booting in graphical mode (the default) does produce a useful dialog box. Issues like this stem from syntax errors in the kickstart file, or bugs in pykickstart itself. If you think it is a pykickstart bug, then someone in Rel-Eng needs to update pykickstart on the builders.

Did the guest boot?

Koji waits 5 minutes for a guest to boot in this step. It unfortunately does not give a lot of insight to why a guest may not boot, so these are a tougher class of issues to work through. You can usually answer this question by looking in results string. If you see "Timed out waiting for guest to boot", then this is your problem. You can also confirm this in oz.log.

For now, the best way to investigate an issue like this is to drive a guest installation locally using something like Gnome's Virtual Machine Manager (VMM). The steps to perform are:

• Select a Network Install
• For the Operating System Install URL use the same one you gave to Koji. It will be something like http://url/to/fedora/22/Server/x86_64/os.
• Set the Kickstart URL to where your kickstart file is. You may need to make it available over http.
• Bump the memory to 2048M for good measure
• Launch the guest and let it complete installation
• Open a VNC session and watch what happens when the guest attempts to boot.

If the console is not providing enough information, we have to get more creative. Anaconda supports starting an SSH daemon while the installation is happening with the sshpw command in kickstart. Set that and comment out the reboot command. This will let the installation complete locally and wait for a keystroke to reboot the guest. At this point you should be able to ssh in and inspect the environment to figure out what is going on. You should also consider making use of the --log option to %post so that output from the script is saved somewhere.

Another option would be to scp logs and other files off of the guest as part of the %post script.

Other Guest Misconfigurations

If the guest boots but you're having problems accessing it I'd suggest following same procedure as when the guest fails to boot. This could be a result of firewall misconfigurations or SSH not being available for some reason. Usually in this case the build is succeeding in Koji, but there's something still fundamentally broken in the image. If the issue is something you can investigate while the guest is online (you can log in), then I'd suggest importing it locally using the libvirt.xml and the disk image provided in Koji's task output.

You can also do investigative work in an offline mode by mounting the image locally or using something like libguestfs to poke around without starting the guest. The fast, dirty way to do it is by mounting it. This can often pollute your guest environment. Here's how to do it:

• Download the image from Koji
• If the image format is not raw, you have to convert it first with qemu-img. Something like:

  $ qemu-img convert -O raw <image-file> <output-file>

• Now mount it up using loopback devices. (as root) If your image has multiple partitions in it, you may need to pass in a different mapped loopback device like loop0p2. Whichever one you think is the root partition or has the issue you're trying to fix.

        # kpartx -av <raw-image>
        # mount -o loop /dev/mapper/loop0p1 /mnt/my_directory

Hopefully at this point you figure out the issue. To tear down the image you'll run commands as root like so:

        # umount /mnt/my_directory
        # dmsetup remove loop0p1
        # losetup -d /dev/loop0

Again, if you used different loopback devices, substitute those in to the dmsetup and losetup commands.

Build System Preparation

TBD