LiveOS image

Introduction

Fedora has developed Live CD USB DVD images for their GNU/Linux operating system. Since the image file systems are stored in the /LiveOS folder of the image, this is the name we'll use to reference the product.

This page shares some critical information about the LiveOS design that helps users take better advantage of their more-limited-than-usual filesystem storage resources.

Take note:
It is very easy to get lost and confused while working with the software supporting LiveOS installation and booting. It helps to recognize that the product was developed first using the Device Mapper snapshot style of overlay for write support on a read-only base filesystem. Later new kernel technologies and application options were grafted into the code, like Device Mapper thin provisioning and yet later OverlayFS union mounts. This evolution has produced a classic example of Spaghetti code, where independent technologies and sometimes overlapping configuration options all share a monolithic code base by means of a tangled web of conditional statements and code blocks.

Furthermore, all the potential data storage locations, types, & styles multiply the complexity of the subject and code base.

One must be very careful to separate one's thinking about options and features to only those that apply to a particular technology.

References

• livecd-iso-to-disk Usage instructions are on the first pages (or on the manual page).
• Booting live images - from the dracut initramfs command line manual.
• FedoraLiveCD

Multi Live Image boot installations

With large capacity USB/SD disc drives readily available, it is convenient to install multiple images on one device.

The livecd-iso-to-disk --multi option allows one to install more than one LiveOS image on a single device. Version 24.2 or greater of livecd-tools will offer to configure the device bootloader to give a Multi Live Image Boot Menu for the device if the target device contains an existing LiveOS image.

The first live image is installed by default in the device's /LiveOS directory. Subsequent images are installed in sister directories with randomly assigned, 4 character names, such as /sYBm. The target directory may be optionally specified by the [--livedir <dir>] option of livecd-iso-to-disk.

The livecd-iso-to-disk installation script allows one to install from and to a booted LiveOS device and to select any of the installed images as the source for a new installation by using its directory path as the source parameter, for example, /run/initramfs/live/sYBm. The booted LiveOS image can be sourced by the source shortcut 'live' and the booted LiveOS device can be targeted by the target shortcut 'live'.

Storage

When a Live CD or Live DVD (a LiveOS image on read-only disc media) is booted, temporary storage is prepared for the system in RAM on each boot by /sbin/dmsquash-live-root in initrd.img, the initial ramdisk filesystem. By default, a 32 GiB, in-memory, copy-on-write, system overlay in a sparse file is prepared (see File Systems below). The rd.live.overlay.size kernel command line option may be used to set a different, temporary, overlay size. Since the temporary overlay is a sparse file in a tmpfs, a large size, even larger than available memory, may be specified and only what is needed will be allocated as needed.

When the rd.live.overlay kernel command line option is provided on boot, such as with a Live USB device, the dmsquash-live-root.sh script will search for a persistent overlay file to use for storage of root filesystem changes. See Booting live images in the dracut package.

LiveOS device users should take note of their overlay status. See #Overlay allocation status, below, for a way to monitor overlay consumption to anticipate and avoid overlay Overflow or Invalidation.

With Fedora 24 (kernel 4.3+), if the overlay storage space is filled, the overlay will enter a 'Overflow' state and the root file system will continue to operate in a read-only mode. There will not be an explicit warning or signal when this happens, but applications may begin reporting errors due to the restriction. See the instructions at LiveOS image/overlay#Overlay recovery for how one may then enlarge and attempt to repair and merge the overlay.

Note: In systems built before Fedora 24, should the overlay storage space, whether temporary or persistent, be totally consumed, the filesystem will be flagged 'Invalid' and the system will crash with Input/output or Bus errors. Achieving a reboot will require a hard reset since attempting a software initiated reboot on the Invalid filesystem will fail with more Input/output or Bus errors. If such a crash occurs while using temporary storage space for the overlay, a simple reboot will rectify the situation. With persistent storage, the situation is more dire. One can attempt to enlarge and repair the overlay using instructions at LiveOS image/overlay#Overlay recovery. Or one can vacate all changes to the root file system by appending the option rd.live.overlay.reset (formerly, reset_overlay) to the kernel command line on boot-up. A persistent home filesystem, if used, will be unaffected by an overlay reset.

Structure of LiveOS file systems

Live Operating System, LiveOS, file systems are found within disk image .img files.

Live.iso

If one mounts a Fedora-Workstation-Live.iso file or Live CD, the file system will list folders such as the following:

/EFI
/images
/isolinux
/LiveOS
    |- squashfs.img
    
  (Systems before Fedora 24 also had these files
    |- livecd-iso-to-disk  -  a Bash script for installing the LiveOS image onto a USB device
    |- osmin.img  -  a minimized OS image formerly used to aid installation to a hard disk

squashfs.img

The squashfs.img is a SquashFS compressed, read-only, file system holding the Fedora operating system root file system inside another /LiveOS folder containing a rootfs.img file. Mounting the squashfs.img file and listing its file system will show this hierarchy:

/LiveOS
    |- rootfs.img    (This contains a filesystem of type ext4. Before F-24 it was named ext3fs.img.)

Mounting the rootfs.img file will finally reveal the Fedora operating system root file system.

Live.iso + squashfs.img

Here is a summary of the whole structure:

Fedora-Workstation-Live.iso
  !(mount)
  /EFI
  /image
  /isolinux
  /LiveOS
      |- squashfs.img
           !(mount)
           /LiveOS
               |- rootfs.img
                    !(mount)
                    /bin -> usr/bin
                    /boot
                    /dev
                    /etc
                    /home
                    /lib -> usr/lib
                    /lib64 -> usr/lib64
                    /lost+found
                    /media
                    /mnt
                    /opt
                    /proc
                    /root
                    /run
                    /sbin -> usr/sbin
                    /srv
                    /sys
                    /tmp
                    /usr
                    /var

Installed LiveUSB/SD device

When the LiveOS image is loaded onto a USB or SD disk, the /isolinux folder is copied into a /syslinux folder.

If persistence is requested with the --overlay-size-mb NNN option, a Device-mapper overlay file for the root file system is created in /LiveOS.

If a separate home file system is requested with the --home-size-mb NNN option, an ext4 formatted home.img file system is created.

The structure is then as follows:

Fedora-Workstation-Live USB/SD device
  !(mounted on /run/initramfs/live) (Before Fedora 17 the mount point is /mnt/live in the root file system.)
  /syslinux
  /LiveOS
      |- home.img
           !(mounted on /home)      (This occurs during boot up by the /etc/rc.d/init.d/livesys script.)
           /liveuser
               /Desktop
               /Documents
               /Downloads
               ...
           /lost+found
      |- livecd-iso-to-disk
      |- overlay-NAME-XXXX-XXXX     (Where NAME is the device partition name and XXXX are hex numerals.)
      |- squashfs.img
           !(mount)                 (At boot up by the /usr/sbin/dmsquash-live-root.sh script in the initramfs.)
           /LiveOS
               |- rootfs.img
                    !(mounted on '/')    (This occurs during boot up by the dmsquash-live-root.sh script.)
                    /bin -> usr/bin
                    /boot
                    /dev
                    /etc
                    /home                (If there is a home.img, then this is its mount point directory.)

Uncompressed LiveUSB/SD installation

If the --skipcompress option is used during loading with livecd-iso-to-disk, the squashfs.img compressed image will be expanded. The structure is then as follows:

Fedora-Workstation-Live USB/SD device
  !(mounted on /run/initramfs/live)
  /syslinux
  /LiveOS
      |- home.img
           !(mounted on /home)
           /liveuser
               /...
      |- livecd-iso-to-disk
      |- overlay-NAME-XXXX-XXXX
      |- rootfs.img
           !(mounted on '/')          
           /bin -> usr/bin
           /boot
           /dev
           /etc
           /home
           ...

Flattened squashfs.img

For OverlayFS mount overlays, the squashfs.img structure may also be a direct compression of the root filesystem: (Booting this structure is supported in dracut version 049 available in Fedora 30.)

squashfs.img          |  SquashFS from LiveCD .iso
   !(mount)
   /bin               |  Live filesystem
   /boot              |
   /dev               |
   ...                |

Persistent storage

The Fedora LiveOS system allows for persistent storage in 3 locations:

1. • By default, an all-purpose, persistent, Device-mapper snapshot overlay-based file space that saves all updates and changes to the root LiveOS filesystem. This storage space is limited by its allocate-once, fixed-sized design, and deserves some caution (see File Systems below).
   • Or an OverlayFS based special mount of the root filesystem invoked by the --overlayfs option to livecd-iso-to-disk (see Booting Live Images and OverlayFS based overlay below).
2. Persistent home filesystem - a re-writable, re-sizable (with difficulty), uncompressed, optionally-encryptable, file space for anything that goes in the users /home/ folder. A persistent home filesystem is an option that may be selected at the time of installation of the LiveOS image (although with some effort, one could manually create a home.img filesystem in /LiveOS/ and move the /home/ folder to it). This installation option is only available through the script, livecd-iso-to-disk. The Windows and Fedora Fedora Media Writer/Live USB Creator installers do not provide this option at present.
3. Host device file space - this is the underlying USB stick or SD card file system that is outside of the LiveOS file tree, but which is accessible through the /run/initramfs/live or /mnt/live (before Fedora 17) mount point of a booted LiveOS installation. There, one finds the boot configuration files and anything else one had on the device before loading the LiveOS image. One may save files here without consuming the other, limited file spaces. (This file space is limited only by the device capacity).

/run/initramfs/live
                  /LiveOS
                      |- home.img
                      |- livecd-iso-to-disk
                      |- overlay-NAME-XXXX-XXXX
                      |- squashfs.img
                  /syslinux

The all-purpose, persistent overlay is needed for keeping operating system changes and updates between boots.

The file systems are registered on each boot by the /usr/libexec/livesys/livesys-main script.

Home filesystem

One may find many advantages to installing the LiveOS, with a persistent, home filesystem (using the --home-size-mb NNN option), which will hold all the user files and documents one wishes and, perhaps later, throw away—all without consuming storage space in the overlay, which can be depleted very quickly.

Device filesystem

Additionally, keeping some storage space on the device disc outside of the LiveOS system will let you copy, carry, and delete large resource files, such as image.iso files, or anything you might want to use or share.

Installation options

Fedora 24 Live Workstation may be installed on a 2 GB USB device using the following options with livecd-iso-to-disk (on a single, terminal or console command line):

./livecd-iso-to-disk --reset-mbr --overlay-size-mb 300 --home-size-mb 200 --unencrypted-home /path/to/source/iso/or/device /dev/sd?1

where '?' in the final parameter represents the target bootable device node, such as sdb1 or sdc1, etc.

The above configuration would allow space for the home folder, the operating system, and a minimal amount on the device root.

But with a larger capacity device, one may allocate the resources to suit the anticipated use, as described above.

File Systems

Take note:
It is important to realize that because the LiveOS root filesystem normally resides in a compressed SquashFS image, the standard estimates of free disc space, such as those reported by df and graphical file managers, will report "apparent" file space (as if the filesystem were not compressed), thus, such estimates are usually vast overestimates of the actual free space physically available to the root filesystem. The embedded rootfs.img filesystem partition is often sized to just fit the distributed operating system at the time of its creation.

For example, the Fedora-Workstation-Live-x86_64-28-1.1.iso contains an apparent 6.4 GiB ext4 root filesystem that has been sparsed and compressed into under 1.6 GiB of actual disc space in the .iso file (or on a LiveOS installed device file system). df -h on an untouched mounting of the root filesystem will report that 4.9G is used and 1.5G is available space out of a total size of 6.4G.

The actually available and allocated physical disc space for the root filesystem is determined by the size of the device-mapper overlay file and can only be queried by the dmsetup status command (see below).

For example, one may add, and then delete, a large file in the root filesystem and see (the apparent virtual) used space go up, and then down, when checking with df, but the dmsetup status live-rw allocation report will only ever show an increase—even though allocated overlay sectors are available for reuse if an added file has been deleted (see discussion below).

Fedora LiveOS uses the Device-mapper service of the Linux kernel to manage the file stores on the device. This is the same service that is used by Logical Volume Manager to provide disc partition services.

Overlay allocation status

One critical limitation, mentioned above, is that the LiveOS persistent overlay is an allocate-once, fixed size, file space. This is related to its use of device mapper snapshots to combine a read-only file system image (copied from the compressed SquashFS.img on the read-only LiveCD or installation .iso file) with a Copy-on-write service that tracks only changed blocks of data in the snapshot (overlay) file and then re-referencing file pointers to the updated blocks. Only changed sectors in files are stored in the overlay. "Deletions" of any original files in the base filesystem are recorded by changes in the index metadata so that the original file blocks are hidden. With this mechanism, no physical storage space of the read-only, base filesystem can be reused. The apparent free space in the virtual root filesystem as reported by the df command will increase, but this additional space is not physically available; however, when a new or changed file is deleted, the new or changed sectors only (in the overlay) are available for reuse, but, unfortunately, the count of allocated sectors does not change, so one cannot be certain of the availability of free physical space after arbitrary changes to the root filesystem.

With Fedora 24 (kernel 4.3+), if the overlay storage space is filled, the overlay will enter an 'Overflow' state and the root file system will continue to operate in a read-only mode. See the instructions at LiveOS image/overlay for how one may then enlarge and attempt to repair and merge the overlay.

The status of the space allocated for persistent storage in the snapshot overlay file may be tracked with the device mapper dmsetup status report.

For example, dmsetup status live-rw may return

live-rw: 0 8388608 snapshot 42296/1048576 176

where the fraction after snapshot is the number of 512-byte sectors allocated in the overlay over the total number available in the overlay. (The final number is the metadata sectors allocated, and the number before snapshot is the apparent size of the virtual filesystem.)

(For systems built before Fedora 24 that exhaust the overlay to an 'Invalid' state, last-ditch recovery of a persistent overlay for the failed root filesystem may be attempted from a separate boot of a working system.)

OverlayFS based overlay

When installation of large software packages is expected or long-term usage of a LiveOS image is anticipated, an alternate overlay strategy is advised. If storage or memory resources are available, one may take advantage of the --overlayfs option to livecd-iso-to-disk. This option applies a specialized OverlayFS mount to the root filesystem, which allows one to escape from the limitation of the partition size of the embedded rootfs.img (see Booting Live Images).

Another option for non-persistent storage is to use the rd.writable.fsimg (see also Booting Live Images).

Flash-Friendly File System (F2FS)

An update to livecd-iso-to-disk has enabled the formatting and booting of a LiveUSB device with the Flash-Friendly File System. The package f2fs-tools must be installed in order to format and check F2FS filesystems for errors. It is not needed to read or write data from or to existing F2FS filesystems.

There are currently a few booting issues to appreciate when choosing this format:

1. SYSLINUX-EXTLINUX does not support F2FS booting.
2. Stock Fedora images have not included an F2FS module in their GNU GRUB EFI binaries until Fedora 35 Rawhide in June 2021.
3. GRUB's F2FS module has not been updated to read systems formatted with the extra_attr, compression, and perhaps other newer features. See this bug report.

The second issue can be addressed by building a GRUB EFI boot loader binary that includes the F2FS module. The subpackage grub2-efi-x64-modules to grub2 can be installed and grub2-mkimage can be called with the appropriate options for the x86_64 architecture. Other architectures would require additional building.

The third issue can be addressed by bypassing GRUB and using dracut with its --UEFI option to build an EFI Boot Stub, which serves as a boot loader. objcopy, in the dependency package binutils, is required on the host for this solution.

An EFI Boot Stub boot loader is somewhat less convenient than GRUB because altering or entering kernel command line options requires rebuilding the EFI Boot Stub or employing efibootmgr with its -@ | --append-binary-args option on each system rather than within a configuration file in the image for use by any system booting the LiveUSB device.

If the extra_attr, compression, or another newer feature of F2FS is significant for a particular use, then an EFI Boot Stub may be a reasonable compromise in convenience for the benefit of utility.

The livecd-iso-to-disk script will test for the availability of the x86_64 architecture dependencies and guide the user to formatting and configuring a Fedora image to boot from an F2FS-formatted LiveUSB device.

Kernel updates for images with a persistent overlay

If you have sufficient overlay space to accommodate a kernel update on a Live USB installation, the kernel and initramfs will be installed to the /boot directory. To put these into service they must be moved to the /syslinux directory of the installation partition. This is accessible from the running Live USB filesystem at the /run/initramfs/live mount point. The new initramfs (such as initramfs-4.17.7-200.fc28.x86_64.img) and kernel (such as vmlinuz-4.17.7-200.fc28.x86_64) should be moved to replace the /run/initramfs/live/syslinux/initrd.img and /run/initramfs/live/syslinux/vmlinuz files, respectively.

Note: The initial RAM filesystem builder, dracut, no longer includes the dmsquash-live module by default. Starting with Fedora 19, dracut defaults to the hostonly="yes" option, which itself precludes the dmsquash-live module. The following command will show what arguments were used in building a given initialramfs:

lsinitrd <path to initrd.img file> usr/lib/dracut/build-parameter.txt

For Fedora-Workstation-Live-x86_64-39-1.5.iso the arguments are

--nomdadmconf --nolvmconf --xz --add 'livenet dmsquash-live dmsquash-live-ntfs convertfs pollcdrom qemu qemu-net' --no-hostonly --debug --no-early-microcode --force

These are mostly set in the builder at https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L51

So, one can add a dracut config file, as root, before updating the kernel (or rebuild the initialramfs by calling dracut):

echo 'mdadmconf="no"
lvmconf="no"
squash_compress="xz"
add_dracutmodules+=" livenet dmsquash-live dmsquash-live-ntfs convertfs pollcdrom qemu qemu-net "
hostonly="no"
early_microcode="no"
' > /etc/dracut.conf.d/01-liveos.conf

Refresh kernel and initial RAM filesystem with editliveos

This update to editliveos available in version 30+ of the livecd-tools package will detect when a new kernel has been installed, build a new initrd.img, and copy these to the appropriate directories of a refreshable Live USB installation and the new .iso image distribution file. Save the above /etc/dracut.conf.d/01-liveos.conf to your host computer before running editliveos. (A future version of editliveos will automatically mimic the configuration in the existing initrd.img.)

References

See this dm-devel thread.

• LiveOS image/overlay