From Fedora Project Wiki
(add overlay avoidance section)
m (→‎Merge overlay into new image: refer to editliveos up front)
 
(12 intermediate revisions by 2 users not shown)
Line 1: Line 1:
<noinclude>[[Category:LiveMedia]]{{TOCright}}
</noinclude>
To maximize the endurance of a LiveOS image, several elements of the LiveOS design deserve consideration.
==Avoid persistent overlay consumption==
==Avoid persistent overlay consumption==


* Use a persistent {{Code|home.img}} file system to mount on the {{Code|/home}} folder (discussed on main page, [[LiveOS image#Home filesystem]]).
* Use a persistent {{Code|home.img}} file system to mount on the {{Code|/home}} folder (discussed on main page, [[LiveOS image#Home filesystem]]).
*: This is a very effective method to avoid consumption and will also make the user files more available for sharing and backup purposes.
*: This is a very effective method to avoid consumption and will also make the user files more available for sharing and backup purposes.
* Mount more folders onto temporary, in-memory filesystems (like /var/cache/yum, /var/tmp, & /tmp are now).
* Mount more folders onto temporary, in-memory filesystems (like /var/tmp, & /tmp are now).
** /var/lib/NetworkManager (holds a timestamps file that is deleted and refreshed quite often, every 5 minutes)
** /var/log/audit (holds the SELinux audit.log that records a great many file accesses.)
** /var/log/audit (holds the SELinux audit.log that records a great many file accesses.)
** /var/spool/abrt (holds often large, error reports and core dumps).  Users could be advised to act on any abrt reports in the current session or copy the reports to other permanent storage, such as in /home/ or external storage.
** /var/spool/abrt (holds often large, error reports and core dumps).  Users could be advised to act on any abrt reports in the current session or copy the reports to other permanent storage, such as in /home/ or external storage.
** there may be other good candidates...
** there may be other good candidates...
==Overlay recovery==
{{admon/warning|Test and practice the following|This procedure is not thoroughly validated, may not work, and may destroy your data.}}
* On Fedora 24+, if the overlay storage space on a LiveOS device is completely filled, Device-mapper will report 'Overflow' status:
# dmsetup status
live-base: 0 13635584 linear
live-rw: 0 13635584 snapshot Overflow
: While the system continues to function, the root filesystem has been remounted read-only.  You will probably see errors due to this restriction.
: Once the system is shut down, possibly by a hard reset, you may attempt to recover the filesystem with the procedure below, though success is not certain.
First, determine the size of the overlay file:
# losetup /dev/loop0 /run/media/user/devicename/LiveOS/overlay-devicename-discUUID -r
# blockdev -q --getsz /dev/loop0
# losetup -d /dev/loop0
Next, enlarge the overlay:
# dd if=/dev/zero of=/path/to/overlay-file seek=overlay_size count=size_increase conv=fsync
: where overlay_size and size_increase are given in 512 byte units.
Then, determine the size of the root filesystem:
# mount /run/media/user/devicename/LiveOS/squashfs.img /mnt/some_mountpoint -r
# losetup /dev/loop1 /mnt/some_mountpoint/LiveOS/rootfs.img -r
# blockdev -q --getsz /dev/loop1
And register the LiveOS image snapshot with Device-mapper:
# losetup /dev/loop2 /run/media/user/devicename/LiveOS/overlay-devicename-discUUID
# dmsetup create devicename --table "0 13635584 snapshot 7:1 7:2 PO 8"
:* devicename in the last line may be any string.
:* loop1 and loop2 (and the corresponding 7:1 7:2) may be any free loop devices; just substitute the appropriate node numbers.
Check that the the virtual filesystem has been configured:
# dmsetup status
Finally, try to repair any corruption in the filesystem with the following command:
# e2fsck -f -y /dev/mapper/devicename
: where devicename is the string you used in the dmsetup create command.
Run a second check,
# e2fsck -p /dev/mapper/devicename
: to verify if the filesystem could be repaired.
Once you have a working filesystem, you may want to merge your overlay into the original filesystem as described [[#Merge overlay into new image|below]] to recover unused storage space.
* On builds before Fedora 24, if one '''exhausts''' the limited storage capacity of a LiveOS overlay, Device-mapper will mark the filesystem as 'Invalid', as shown by the {{Code|dmsetup status}} command executed in Terminal or a console (if you haven't crashed):
# dmsetup status
live-osimg-min: 0 8388608 snapshot 2464/2464 24
live-rw: 0 8388608 snapshot Invalid
: The invalid flag is 00 at byte 5 of the overlay.
Switch the invalid flag to 01 with the following command line:
# echo $'\x01' | dd of=/path/to/overlay-file bs=1 count=1 seek=4 conv=notrunc
Then, proceed with the recovery instructions, above, as given for the overflow condition.


==Merge overlay into new image==
==Merge overlay into new image==
The [https://github.com/livecd-tools/livecd-tools/blob/master/tools/editliveos editliveos] Python script in the livecd-tools package will merge overlays into a new base image.  The BASH script below may help one to understand what steps are involved.


A new root filesystem image can be constructed by using some Device-mapper tools.
A new root filesystem image can be constructed by using some Device-mapper tools.


If one has sufficient free disk space available, typically on an attached hard drive, one can copy (and uncompress) the current filesystem to a working folder. (This may require, for example, over 4.0 GiB of free space for Fedora 24 Live Workstation.)
If one has sufficient free disk space available, typically on an attached hard drive, one can copy (and uncompress) the current filesystem to a working folder. (This may require, for example, over 4.0 GiB of free space for Fedora 24 Workstation Live.)


Device-mapper's mirror target allows one to create a 'mirror' image of the LiveOS root filesystem (the snapshot based on the SquashFS and overlay device).  The mirror image can be then be recompressed, and if one has an additional 1.5 GiB or more of free space (depending on how much software or other root filesystem files have been added) on your LiveOS device filesystem (the USB/SD card filesystem), one can delete the old SquashFS image and replace it with the recompressed version.
Device-mapper's mirror target allows one to create a 'mirror' image of the LiveOS root filesystem (the snapshot based on the SquashFS and overlay device).  The mirror image can be then be recompressed, and if one has an additional 1.5 GiB or more of free space (more depending on how much software or other root filesystem files have been added) on your underlying device filesystem (the USB/SD card filesystem), one can delete the old SquashFS image and replace it with the recompressed version.


The snapshot overlay can then be reset before rebooting the LiveOS device.
The snapshot overlay can then be reset before rebooting the LiveOS device.


The following Bash script demonstrates the process, where the {{Code|$1}} parameter is a mount point directory for a disk partition with sufficient storage capacity (a LiveOS folder will be created there for holding the mirrored filesystem), and {{Code|$2}} is the LiveOS device node name (such as, {{Code|/run/initramfs/livedev}}, for the currently booted LiveOS image, or {{Code|/dev/sdc1}}, for example, for an attached LiveOS device). Options xtrace and verbose have been set to aid debugging.
The following Bash script demonstrates the process, where the {{Code|$1}} parameter is a mount point directory for a disk partition with sufficient storage capacity (a LiveOS folder will be created there for holding the mirrored filesystem), and {{Code|$2}} is the LiveOS device node name (such as, {{Code|/run/initramfs/livedev}}, for the currently booted LiveOS image, or {{Code|/dev/sdc1}}, for example, for an attached LiveOS device). Options xtrace and verbose have been set to aid debugging.
{{admon/warning|Test and practice the following script!|This procedure will delete the LiveOS filesystem before replacing it with a recompressed merged version. Be sure to verify that you will have sufficient space for the replacement before running the full script.  (You could comment out the deletion, and squash into a directory on a large working drive to determine the new filesystem size.)}}
{{admon/warning|Test and practice the following script!|This procedure will delete the LiveOS filesystem before replacing it with a recompressed merged version. Be sure to verify that you will have sufficient space for the replacement before running the full script.  (You could comment out the deletion, and squash into a directory on a large, working drive to determine the new filesystem size.)}}
<pre>
<pre>
#!/bin/bash -xv
#!/bin/bash -xv
Line 304: Line 358:
The mirror method copies and merges in one pass, while the snapshot-merge method copies the original filesystem first and then merges in the changes. I've found the mirror method to be about 15% faster.
The mirror method copies and merges in one pass, while the snapshot-merge method copies the original filesystem first and then merges in the changes. I've found the mirror method to be about 15% faster.


[https://github.com/FGrose/livecd-tools/blob/editliveos/imgcreate/editliveos.py editliveos.py] is a Python script that performs overlay merging and, optionally, other Live OS editing.  It should be run from a directory containing [https://github.com/FGrose/livecd-tools/tree/editliveos/imgcreate these supporting files].
[https://github.com/livecd-tools/livecd-tools/blob/master/tools/editliveos editliveos] is a Python script that performs overlay merging and, optionally, other Live OS editing.  It is available in Version 25 of the {{package|livecd-tools}} package.
 
===Overlay recovery===
{{admon/warning|Test and practice the following|This procedure is not thoroughly validated, and may destroy your data.}}
If one 'exhausts' the limited storage capacity of a LiveOS overlay, Device-mapper will mark the filesystem as 'Invalid', as shown by the {{Code|dmsetup status}} command executed in Terminal or a console (if you haven't crashed):
# dmsetup status
live-osimg-min: 0 8388608 snapshot 2464/2464 24
live-rw: 0 8388608 snapshot Invalid
The invalid bit is 00 at byte 5 of the overlay.  You might try to recover the overlay by switching it to 01 with the following command line:
# echo $'\x01' | dd of=/path/to/overlay-file bs=1 count=1 seek=4 conv=notrunc
where /path/to/overlay may be /mnt/live/LiveOS/overlay-devicename-discUUID or<br>/media/devicename/LiveOS/overlay-devicename-discUUID for an attached device.
 
Follow this by registering the LiveOS image with Device-mapper:
# mount /media/devicename/LiveOS/squashfs.img /mnt/some_mountpoint
# losetup /dev/loop1 /mnt/some_mountpoint/LiveOS/ext3fs.img -r
# losetup /dev/loop2 /media/devicename/LiveOS/overlay-devicename-discUUID
# dmsetup create devicename --table "0 8388608 snapshot 7:1 7:2 P 8"
* devicename in the last line may be any string.
* loop1 and loop2 (and the corresponding 7:1 7:2) may be any free loop devices; just substitute the appropriate numbers.
* 8388608 is the size of the ext3fs.img file in 512 byte units.  This can be read by the following command:
# blockdev -q --getsz /dev/loop1
Execute
# dmsetup status
to check the the virtual filesystem has been configured.
 
Then try to repair any damage with the following command:
# e2fsck -f -y /dev/mapper/devicename
where devicename is the string you used in the dmsetup create command.
 
Run a second check,
# e2fsck -p /dev/mapper/devicename
to verify if the filesystem could be repaired.
At this point you will want to enlarge the overlay, or backup or rebuild the image.
 
Remove the virtual filesystem registration,
# dmsetup remove devicename
Determine the size of the overlay file:
# blockdev -q --getsz /dev/loop2
# losetup -d /dev/loop2
# losetup -d /dev/loop1
# dd if=/dev/zero of=/path/to/overlay-file seek=overlay_size count=size_increase
where overlay_size and size_increase are now 512 byte units.


Once you have a working filesystem, you should proceed to merge your overlay into the original filesystem as described [[#Merge overlay into new image|above]].
==Credits==
* The method of [[LiveOS_image/overlay#Merge_overlay_into_new_image|mirroring the LiveOS]] was adapted from Douglas McClendon's [http://cloudsession.com/dawg/projects/zyx-liveinstaller/ ZyX-LiveInstaller].

Latest revision as of 04:00, 26 September 2021

To maximize the endurance of a LiveOS image, several elements of the LiveOS design deserve consideration.

Avoid persistent overlay consumption

  • Use a persistent home.img file system to mount on the /home folder (discussed on main page, LiveOS image#Home filesystem).
    This is a very effective method to avoid consumption and will also make the user files more available for sharing and backup purposes.
  • Mount more folders onto temporary, in-memory filesystems (like /var/tmp, & /tmp are now).
    • /var/log/audit (holds the SELinux audit.log that records a great many file accesses.)
    • /var/spool/abrt (holds often large, error reports and core dumps). Users could be advised to act on any abrt reports in the current session or copy the reports to other permanent storage, such as in /home/ or external storage.
    • there may be other good candidates...

Overlay recovery

Test and practice the following
This procedure is not thoroughly validated, may not work, and may destroy your data.
  • On Fedora 24+, if the overlay storage space on a LiveOS device is completely filled, Device-mapper will report 'Overflow' status:
# dmsetup status
live-base: 0 13635584 linear
live-rw: 0 13635584 snapshot Overflow
While the system continues to function, the root filesystem has been remounted read-only. You will probably see errors due to this restriction.
Once the system is shut down, possibly by a hard reset, you may attempt to recover the filesystem with the procedure below, though success is not certain.

First, determine the size of the overlay file:

# losetup /dev/loop0 /run/media/user/devicename/LiveOS/overlay-devicename-discUUID -r
# blockdev -q --getsz /dev/loop0
# losetup -d /dev/loop0

Next, enlarge the overlay:

# dd if=/dev/zero of=/path/to/overlay-file seek=overlay_size count=size_increase conv=fsync
where overlay_size and size_increase are given in 512 byte units.

Then, determine the size of the root filesystem:

# mount /run/media/user/devicename/LiveOS/squashfs.img /mnt/some_mountpoint -r
# losetup /dev/loop1 /mnt/some_mountpoint/LiveOS/rootfs.img -r
# blockdev -q --getsz /dev/loop1

And register the LiveOS image snapshot with Device-mapper:

# losetup /dev/loop2 /run/media/user/devicename/LiveOS/overlay-devicename-discUUID
# dmsetup create devicename --table "0 13635584 snapshot 7:1 7:2 PO 8"
  • devicename in the last line may be any string.
  • loop1 and loop2 (and the corresponding 7:1 7:2) may be any free loop devices; just substitute the appropriate node numbers.

Check that the the virtual filesystem has been configured:

# dmsetup status

Finally, try to repair any corruption in the filesystem with the following command:

# e2fsck -f -y /dev/mapper/devicename
where devicename is the string you used in the dmsetup create command.

Run a second check,

# e2fsck -p /dev/mapper/devicename
to verify if the filesystem could be repaired.

Once you have a working filesystem, you may want to merge your overlay into the original filesystem as described below to recover unused storage space.


  • On builds before Fedora 24, if one exhausts the limited storage capacity of a LiveOS overlay, Device-mapper will mark the filesystem as 'Invalid', as shown by the dmsetup status command executed in Terminal or a console (if you haven't crashed):
# dmsetup status
live-osimg-min: 0 8388608 snapshot 2464/2464 24
live-rw: 0 8388608 snapshot Invalid
The invalid flag is 00 at byte 5 of the overlay.

Switch the invalid flag to 01 with the following command line:

# echo $'\x01' | dd of=/path/to/overlay-file bs=1 count=1 seek=4 conv=notrunc

Then, proceed with the recovery instructions, above, as given for the overflow condition.


Merge overlay into new image

The editliveos Python script in the livecd-tools package will merge overlays into a new base image. The BASH script below may help one to understand what steps are involved.

A new root filesystem image can be constructed by using some Device-mapper tools.

If one has sufficient free disk space available, typically on an attached hard drive, one can copy (and uncompress) the current filesystem to a working folder. (This may require, for example, over 4.0 GiB of free space for Fedora 24 Workstation Live.)

Device-mapper's mirror target allows one to create a 'mirror' image of the LiveOS root filesystem (the snapshot based on the SquashFS and overlay device). The mirror image can be then be recompressed, and if one has an additional 1.5 GiB or more of free space (more depending on how much software or other root filesystem files have been added) on your underlying device filesystem (the USB/SD card filesystem), one can delete the old SquashFS image and replace it with the recompressed version.

The snapshot overlay can then be reset before rebooting the LiveOS device.

The following Bash script demonstrates the process, where the $1 parameter is a mount point directory for a disk partition with sufficient storage capacity (a LiveOS folder will be created there for holding the mirrored filesystem), and $2 is the LiveOS device node name (such as, /run/initramfs/livedev, for the currently booted LiveOS image, or /dev/sdc1, for example, for an attached LiveOS device). Options xtrace and verbose have been set to aid debugging.

Test and practice the following script!
This procedure will delete the LiveOS filesystem before replacing it with a recompressed merged version. Be sure to verify that you will have sufficient space for the replacement before running the full script. (You could comment out the deletion, and squash into a directory on a large, working drive to determine the new filesystem size.)
#!/bin/bash -xv
#
# Merge a LiveOS snapshot overlay into a new root filesystem, recompress it
# into a SquashFS image (if the source was so packaged), replace the source
# with the refresh, and reset the overlay.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU Library General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.

oldLC=$LC_NUMERIC
LC_NUMERIC="en_US.UTF-8"
export PATH=/usr/sbin:$PATH
export \
   PS4='+(${SHLVL}:${LINENO}:${BASH_SOURCE}:${EUID}): ${FUNCNAME[0]:+${FUNCNAME[0]}(): }'
set -o errexit
set -o pipefail

t0=$(date +%s)

cleanup () {
    umount -l $SRCMNT
    rmdir $SRCMNT
    dt=$(($(date +%s)-t0))
    h=$((dt/3600))
    m=$((dt%3600/60))
    s=$((dt%60))
    printf '\nTotal time elapsed: %02d:%02d:%02d hh:mm:ss\n' $h $m $s
    [[ $1 == 0 ]] && exit 0 || exit 1
}

if [[ ! -f /usr/sbin/mksquashfs ]]; then
    echo "Please install squashfs-tools (yum install squashfs-tools)";
    exit
fi

TMPDIR=$1
[[ ! -d $TMPDIR/LiveOS ]] && mkdir $TMPDIR/LiveOS 2>/dev/null

# Mount the source device and SquashFS.
mkdir -p /run/media #Doesn't exist in fedora 20 livecd
SRCMNT=$(mktemp -d /run/media/XXXXXX)
mount $2 $SRCMNT

if [[ -e $SRCMNT/LiveOS/squashfs.img ]]; then
    SQUASHMNT=$(mktemp -d /run/media/XXXXXX)
    mount $SRCMNT/LiveOS/squashfs.img $SQUASHMNT --read-only
    ORIGFS=$SQUASHMNT/LiveOS/ext3fs.img
    [[ ! -f $ORIGFS ]] && ORIGFS=$SQUASHMNT/LiveOS/rootfs.img
else
    ORIGFS=$SRCMNT/LiveOS/ext3fs.img
    [[ ! -f $ORIGFS ]] && ORIGFS=$SRCMNT/LiveOS/rootfs.img
fi

OVERLAY=/LiveOS/$(basename $SRCMNT/LiveOS/overlay-*)
if [[ -f $SRCMNT$OVERLAY ]]; then
    OVDEV=$(losetup -f --show $SRCMNT$OVERLAY --read-only)
else
    printf '\n  No persistent overlay for the filesystem was found.
    Exiting...\n'
    cleanup 1
fi

# >= Fedora 17
LiveOSMNT=/run/initramfs/live
# < Fedora 17
[[ -d /mnt/live/LiveOS ]] && LiveOSMNT=/mnt/live

# If we are processing a booted LiveOS device's root filesystem, then
if [[ $(mountpoint -x $2) == $(mountpoint -dq $LiveOSMNT) ]]; then
    # Remove live-osimg-min; free its loop devices.
    if [[ -e $SRCMNT/LiveOS/osmin.img ]]; then
        dmsetup remove live-osimg-min || :
        losetup -d /dev/loop1 || :
        losetup -d /dev/loop0 || :
    fi
else
    LiveOSMNT=''
fi
# Remove expiring osmin.img
rm $SRCMNT/LiveOS/osmin.img || :

# Prepare temporary devices for the Device-mapper mirror target.
FSTYPE=$(blkid -s TYPE -o value $ORIGFS || :)
FSLABEL=$(blkid -s LABEL -o value $ORIGFS || :)
stinfo=($(stat -c '%B %s %o' $ORIGFS || :))
BLOCKSZ=${stinfo[0]}
BLOCKS=$((${stinfo[1]}/$BLOCKSZ))
IOBLKSZ=${stinfo[2]}

NEWFS=$TMPDIR/LiveOS/rootfs.img
dd if=/dev/null of=$NEWFS count=1 bs=$BLOCKSZ seek=$BLOCKS
mkfs.$FSTYPE -F -L $FSLABEL -m 1 -b $IOBLKSZ $NEWFS
tune2fs -c0 -i0 -Odir_index -ouser_xattr,acl $NEWFS
NEWFSDEV=$(losetup -f --show $NEWFS)

roORIGFSDEV=$(losetup -f --show $ORIGFS --read-only)

table="0 $BLOCKS snapshot $roORIGFSDEV $OVDEV P 8"
dmsetup create ori --readonly --table "$table"

# Invoke mirror target device.
table="0 $BLOCKS mirror core 2 32 sync 2 /dev/mapper/ori 0 $NEWFSDEV 0"
dmsetup create mir --table "$table"

set +xv
# Wait for mirror completion.
while state=$(dmsetup status mir)
state=${state#*mirror 2 * * }
alloc=${state%/*}
size=${state#*/}
size=${size%% *}
[[ $alloc != $size ]]; do
    percent=$(dc <<< "8k 10 0.05 100 $alloc $size /*+*p")
    percent=$(dc <<< "1k $percent 10 /p")
    printf '\r  Mirroring %5.1f %% complete.  ' $percent
    printf '\b|'
    sleep 0.5
    printf '\b/'
    sleep 0.5
    printf '\b-'
    sleep 0.5
    printf '\b\'
    sleep 0.5
done
printf '\r  Mirroring 100 %% complete.    \n'
set -xv

# Check the new filesystem.
e2fsck -f -y $NEWFS || e2fsck -f -y $NEWFS

# Clean up.
dmsetup remove mir ori
sync

dt=$(($(date +%s)-t0))
h=$((dt/3600))
m=$((dt%3600/60))
s=$((dt%60))
printf '\nTime elapsed: %02d:%02d:%02d hh:mm:ss\n' $h $m $s

# Replace SquashFS or uncompressed root filesystem image.
if [[ -e $SRCMNT/LiveOS/squashfs.img ]]; then
    umount -l $SQUASHMNT
    rm $SRCMNT/LiveOS/squashfs.img
    mksquashfs $TMPDIR/LiveOS $SRCMNT/LiveOS/squashfs.img -comp xz -keep-as-directory
else
    if [[ -x /usr/bin/rsync ]]; then
        rsync --inplace --progress $NEWFS $SRCMNT/LiveOS
    else
        cp $NEWFS $SRCMNT/LiveOS
    fi
fi

# Clean up.
losetup -d $OVDEV $NEWFSDEV $roORIGFSDEV
OVDEV=$(losetup -f --show $SRCMNT$OVERLAY)

# Reset overlay.
dd if=/dev/zero of=$OVDEV bs=64k count=1 conv=notrunc,fsync

# If we are processing a booted LiveOS device's root filesystem, then
if [[ -n $LiveOSMNT ]]; then
    # Try to unmount the TMPDIR.
    mountpoint $TMPDIR && umount -l $TMPDIR || :
    shutdown -r +1 'The system will reboot in 1 minute.'
else
    sleep 1
    losetup -d $OVDEV
fi
cleanup 0

Note: If the above script is invoked on a currently booted LiveOS root filesystem, the system will be scheduled for a reboot one minute after the script completes.

The Device-mapper snapshot-merge target also allows one to merge changes in a persistent snapshot into the original filesystem (See notes from Andrew Gilmore).

The mirror method copies and merges in one pass, while the snapshot-merge method copies the original filesystem first and then merges in the changes. I've found the mirror method to be about 15% faster.

editliveos is a Python script that performs overlay merging and, optionally, other Live OS editing. It is available in Version 25 of the livecd-tools package.

Credits