LiveOS image/overlay

To maximize the endurance of a LiveOS image, several elements of the LiveOS design deserve consideration.

Avoid persistent overlay consumption

• Use a persistent home.img file system to mount on the /home folder (discussed on main page, LiveOS image#Home filesystem).

  This is a very effective method to avoid consumption and will also make the user files more available for sharing and backup purposes.

• Mount more folders onto temporary, in-memory filesystems (like /var/tmp, & /tmp are now).
  • /var/log/audit (holds the SELinux audit.log that records a great many file accesses.)
  • /var/spool/abrt (holds often large, error reports and core dumps). Users could be advised to act on any abrt reports in the current session or copy the reports to other permanent storage, such as in /home/ or external storage.
  • there may be other good candidates...

Overlay recovery

Test and practice the following
This procedure is not thoroughly validated, may not work, and may destroy your data.

• On Fedora 24+, if the overlay storage space on a LiveOS device is completely filled, Device-mapper will report 'Overflow' status:

# dmsetup status
live-base: 0 13635584 linear
live-rw: 0 13635584 snapshot Overflow

While the system continues to function, the root filesystem has been remounted read-only. You will probably see errors due to this restriction.

Once the system is shut down, possibly by a hard reset, you may attempt to recover the filesystem with the procedure below, though success is not certain.

First, determine the size of the overlay file:

# losetup /dev/loop0 /run/media/user/devicename/LiveOS/overlay-devicename-discUUID -r
# blockdev -q --getsz /dev/loop0
# losetup -d /dev/loop0

Next, enlarge the overlay:

# dd if=/dev/zero of=/path/to/overlay-file seek=overlay_size count=size_increase conv=fsync

where overlay_size and size_increase are given in 512 byte units.

Then, determine the size of the root filesystem:

# mount /run/media/user/devicename/LiveOS/squashfs.img /mnt/some_mountpoint -r
# losetup /dev/loop1 /mnt/some_mountpoint/LiveOS/rootfs.img -r
# blockdev -q --getsz /dev/loop1

And register the LiveOS image snapshot with Device-mapper:

# losetup /dev/loop2 /run/media/user/devicename/LiveOS/overlay-devicename-discUUID
# dmsetup create devicename --table "0 13635584 snapshot 7:1 7:2 PO 8"

• devicename in the last line may be any string.
• loop1 and loop2 (and the corresponding 7:1 7:2) may be any free loop devices; just substitute the appropriate node numbers.

Check that the the virtual filesystem has been configured:

# dmsetup status

Finally, try to repair any corruption in the filesystem with the following command:

# e2fsck -f -y /dev/mapper/devicename

where devicename is the string you used in the dmsetup create command.

Run a second check,

# e2fsck -p /dev/mapper/devicename

to verify if the filesystem could be repaired.

Once you have a working filesystem, you may want to merge your overlay into the original filesystem as described below to recover unused storage space.

• On builds before Fedora 24, if one exhausts the limited storage capacity of a LiveOS overlay, Device-mapper will mark the filesystem as 'Invalid', as shown by the dmsetup status command executed in Terminal or a console (if you haven't crashed):

# dmsetup status
live-osimg-min: 0 8388608 snapshot 2464/2464 24
live-rw: 0 8388608 snapshot Invalid

The invalid flag is 00 at byte 5 of the overlay.

Switch the invalid flag to 01 with the following command line:

# echo $'\x01' | dd of=/path/to/overlay-file bs=1 count=1 seek=4 conv=notrunc

Then, proceed with the recovery instructions, above, as given for the overflow condition.

Merge overlay into new image

The editliveos Python script in the livecd-tools package will merge overlays into a new base image. The BASH script below may help one to understand what steps are involved.

A new root filesystem image can be constructed by using some Device-mapper tools.

If one has sufficient free disk space available, typically on an attached hard drive, one can copy (and uncompress) the current filesystem to a working folder. (This may require, for example, over 4.0 GiB of free space for Fedora 24 Workstation Live.)

Device-mapper's mirror target allows one to create a 'mirror' image of the LiveOS root filesystem (the snapshot based on the SquashFS and overlay device). The mirror image can be then be recompressed, and if one has an additional 1.5 GiB or more of free space (more depending on how much software or other root filesystem files have been added) on your underlying device filesystem (the USB/SD card filesystem), one can delete the old SquashFS image and replace it with the recompressed version.

The snapshot overlay can then be reset before rebooting the LiveOS device.

The following Bash script demonstrates the process, where the $1 parameter is a mount point directory for a disk partition with sufficient storage capacity (a LiveOS folder will be created there for holding the mirrored filesystem), and $2 is the LiveOS device node name (such as, /run/initramfs/livedev, for the currently booted LiveOS image, or /dev/sdc1, for example, for an attached LiveOS device). Options xtrace and verbose have been set to aid debugging.

Test and practice the following script!
This procedure will delete the LiveOS filesystem before replacing it with a recompressed merged version. Be sure to verify that you will have sufficient space for the replacement before running the full script. (You could comment out the deletion, and squash into a directory on a large, working drive to determine the new filesystem size.)

#!/bin/bash -xv
#
# Merge a LiveOS snapshot overlay into a new root filesystem, recompress it
# into a SquashFS image (if the source was so packaged), replace the source
# with the refresh, and reset the overlay.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Library General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

oldLC=$LC_NUMERIC
LC_NUMERIC="en_US.UTF-8"
export PATH=/usr/sbin:$PATH
export \
   PS4='+(${SHLVL}:${LINENO}:${BASH_SOURCE}:${EUID}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
set -o errexit
set -o pipefail

t0=$(date +%s)

cleanup () {
    umount -l $SRCMNT
    rmdir $SRCMNT
    dt=$(($(date +%s)-t0))
    h=$((dt/3600))
    m=$((dt%3600/60))
    s=$((dt%60))
    printf '\nTotal time elapsed: %02d:%02d:%02d hh:mm:ss\n' $h $m $s
    [[ $1 == 0 ]] && exit 0 || exit 1
}

if [[ ! -f /usr/sbin/mksquashfs ]]; then
    echo "Please install squashfs-tools (yum install squashfs-tools)";
    exit
fi

TMPDIR=$1
[[ ! -d $TMPDIR/LiveOS ]] && mkdir $TMPDIR/LiveOS 2>/dev/null

# Mount the source device and SquashFS.
mkdir -p /run/media #Doesn't exist in fedora 20 livecd
SRCMNT=$(mktemp -d /run/media/XXXXXX)
mount $2 $SRCMNT

if [[ -e $SRCMNT/LiveOS/squashfs.img ]]; then
    SQUASHMNT=$(mktemp -d /run/media/XXXXXX)
    mount $SRCMNT/LiveOS/squashfs.img $SQUASHMNT --read-only
    ORIGFS=$SQUASHMNT/LiveOS/ext3fs.img
    [[ ! -f $ORIGFS ]] && ORIGFS=$SQUASHMNT/LiveOS/rootfs.img
else
    ORIGFS=$SRCMNT/LiveOS/ext3fs.img
    [[ ! -f $ORIGFS ]] && ORIGFS=$SRCMNT/LiveOS/rootfs.img
fi

OVERLAY=/LiveOS/$(basename $SRCMNT/LiveOS/overlay-*)
if [[ -f $SRCMNT$OVERLAY ]]; then
    OVDEV=$(losetup -f --show $SRCMNT$OVERLAY --read-only)
else
    printf '\n  No persistent overlay for the filesystem was found.
    Exiting...\n'
    cleanup 1
fi

# >= Fedora 17
LiveOSMNT=/run/initramfs/live
# < Fedora 17
[[ -d /mnt/live/LiveOS ]] && LiveOSMNT=/mnt/live

# If we are processing a booted LiveOS device's root filesystem, then
if [[ $(mountpoint -x $2) == $(mountpoint -dq $LiveOSMNT) ]]; then
    # Remove live-osimg-min; free its loop devices.
    if [[ -e $SRCMNT/LiveOS/osmin.img ]]; then
        dmsetup remove live-osimg-min || :
        losetup -d /dev/loop1 || :
        losetup -d /dev/loop0 || :
    fi
else
    LiveOSMNT=''
fi
# Remove expiring osmin.img
rm $SRCMNT/LiveOS/osmin.img || :

# Prepare temporary devices for the Device-mapper mirror target.
FSTYPE=$(blkid -s TYPE -o value $ORIGFS || :)
FSLABEL=$(blkid -s LABEL -o value $ORIGFS || :)
stinfo=($(stat -c '%B %s %o' $ORIGFS || :))
BLOCKSZ=${stinfo[0]}
BLOCKS=$((${stinfo[1]}/$BLOCKSZ))
IOBLKSZ=${stinfo[2]}

NEWFS=$TMPDIR/LiveOS/rootfs.img
dd if=/dev/null of=$NEWFS count=1 bs=$BLOCKSZ seek=$BLOCKS
mkfs.$FSTYPE -F -L $FSLABEL -m 1 -b $IOBLKSZ $NEWFS
tune2fs -c0 -i0 -Odir_index -ouser_xattr,acl $NEWFS
NEWFSDEV=$(losetup -f --show $NEWFS)

roORIGFSDEV=$(losetup -f --show $ORIGFS --read-only)

table="0 $BLOCKS snapshot $roORIGFSDEV $OVDEV P 8"
dmsetup create ori --readonly --table "$table"

# Invoke mirror target device.
table="0 $BLOCKS mirror core 2 32 sync 2 /dev/mapper/ori 0 $NEWFSDEV 0"
dmsetup create mir --table "$table"

set +xv
# Wait for mirror completion.
while state=$(dmsetup status mir)
state=${state#*mirror 2 * * }
alloc=${state%/*}
size=${state#*/}
size=${size%% *}
[[ $alloc != $size ]]; do
    percent=$(dc <<< "8k 10 0.05 100 $alloc $size /*+*p")
    percent=$(dc <<< "1k $percent 10 /p")
    printf '\r  Mirroring %5.1f %% complete.  ' $percent
    printf '\b|'
    sleep 0.5
    printf '\b/'
    sleep 0.5
    printf '\b-'
    sleep 0.5
    printf '\b\'
    sleep 0.5
done
printf '\r  Mirroring 100 %% complete.    \n'
set -xv

# Check the new filesystem.
e2fsck -f -y $NEWFS || e2fsck -f -y $NEWFS

# Clean up.
dmsetup remove mir ori
sync

dt=$(($(date +%s)-t0))
h=$((dt/3600))
m=$((dt%3600/60))
s=$((dt%60))
printf '\nTime elapsed: %02d:%02d:%02d hh:mm:ss\n' $h $m $s

# Replace SquashFS or uncompressed root filesystem image.
if [[ -e $SRCMNT/LiveOS/squashfs.img ]]; then
    umount -l $SQUASHMNT
    rm $SRCMNT/LiveOS/squashfs.img
    mksquashfs $TMPDIR/LiveOS $SRCMNT/LiveOS/squashfs.img -comp xz -keep-as-directory
else
    if [[ -x /usr/bin/rsync ]]; then
        rsync --inplace --progress $NEWFS $SRCMNT/LiveOS
    else
        cp $NEWFS $SRCMNT/LiveOS
    fi
fi

# Clean up.
losetup -d $OVDEV $NEWFSDEV $roORIGFSDEV
OVDEV=$(losetup -f --show $SRCMNT$OVERLAY)

# Reset overlay.
dd if=/dev/zero of=$OVDEV bs=64k count=1 conv=notrunc,fsync

# If we are processing a booted LiveOS device's root filesystem, then
if [[ -n $LiveOSMNT ]]; then
    # Try to unmount the TMPDIR.
    mountpoint $TMPDIR && umount -l $TMPDIR || :
    shutdown -r +1 'The system will reboot in 1 minute.'
else
    sleep 1
    losetup -d $OVDEV
fi
cleanup 0

Note: If the above script is invoked on a currently booted LiveOS root filesystem, the system will be scheduled for a reboot one minute after the script completes.

The Device-mapper snapshot-merge target also allows one to merge changes in a persistent snapshot into the original filesystem (See notes from Andrew Gilmore).

Merge overlay with snapshot-merge

#!/bin/bash t0=$(date +%s) # Prepare working directory. TMPDIR=$1 mkdir $TMPDIR/LiveOS 2>/dev/null # Mount source device, SquashFS, if needed. SRCMNT=$(mktemp -d /media/XXXXXX) mount $2 $SRCMNT if [[ -e $SRCMNT/LiveOS/squashfs.img ]]; then SQUASHMNT=$(mktemp -d /media/XXXXXX) mount $SRCMNT/LiveOS/squashfs.img $SQUASHMNT --read-only ORIGFS=$SQUASHMNT/LiveOS/ext3fs.img else ORIGFS=$SRCMNT/LiveOS/ext3fs.img fi NEWFS=$TMPDIR/LiveOS/ext3fs.img printf '\nCopying original root filesystem.\n' if [[ -x /usr/bin/rsync ]]; then rsync --inplace --progress $ORIGFS $NEWFS else cp $ORIGFS $NEWFS fi # Prepare temporary devices for Device-mapper snapshot-merge. BLOCKS=$(stat -c '%b' $ORIGFS || :) NEWFSDEV=$(losetup -f --show $NEWFS) OVDEV=$(losetup -f --show $SRCMNT/LiveOS/overlay-*) # Invoke snapshot-merge target device. dmsetup create merge <<< "0 $BLOCKS snapshot-merge $NEWFSDEV $OVDEV P 8" # Wait for merge completion. while state=$(dmsetup status merge) state=${state#*snapshot-merge } meta=${state#* } alloc=${state%/*} [[ $alloc != $meta ]]; do printf '/' sleep 0.5 printf '\b-' sleep 0.5 printf '\b\' sleep 0.5 printf '\b|' sleep 0.5 done printf '\n' # Clean up from merge. dmsetup remove merge sync dt=$(($(date +%s)-t0)) h=$((dt/3600)) m=$((dt%3600/60)) s=$((dt%60)) printf '\nTime elapsed: %02d:%02d:%02d hh:mm:ss\n' $h $m $s # Replace original image if [[ -e $SRCMNT/LiveOS/squashfs.img ]]; then umount $SQUASHMNT rmdir $SQUASHMNT rm $SRCMNT/LiveOS/squashfs.img mksquashfs $TMPDIR/LiveOS $SRCMNT/LiveOS/squashfs.img -comp xz -keep-as-directory else if [[ -x /usr/bin/rsync ]]; then rsync --inplace --progress $NEWFS $SRCMNT/LiveOS else cp $NEWFS $SRCMNT/LiveOS fi fi # Reset overlay. dd if=/dev/zero of=$OVDEV bs=64k count=1 conv=notrunc,fsync # Clean up. sleep 2 losetup -d $OVDEV $NEWFSDEV sleep 2 umount $SRCMNT rmdir $SRCMNT LC_NUMERIC=$oldLC dt=$(($(date +%s)-t0)) h=$((dt/3600)) m=$((dt%3600/60)) s=$((dt%60)) printf '\nTotal time elapsed: %02d:%02d:%02d hh:mm:ss\n' $h $m $s

The mirror method copies and merges in one pass, while the snapshot-merge method copies the original filesystem first and then merges in the changes. I've found the mirror method to be about 15% faster.

editliveos is a Python script that performs overlay merging and, optionally, other Live OS editing. It is available in Version 25 of the livecd-tools package.

Credits

• The method of mirroring the LiveOS was adapted from Douglas McClendon's ZyX-LiveInstaller.