Features/LZMA for Live Images

LZMA for Live Images

Summary

Better compression technology (LZMA) allows us to ship more software on our live images.

Owner

• Name: Bruno Wolff III

• Email: <bruno@wolff.to>

Current status

• Targeted release: Fedora 14
• Last updated: 2010-06-11
• Percentage of completion: 55%

Detailed Description

LZMA compression typically provides better compression than zlib while still providing relatively fast uncompression. Once squashfs file systems are supported by the kernel we can use them for building live images. A test with the games spin resulted in a 10% reduction in size of the resulting live image.

Benefit to Fedora

By allowing more packages to be put on a Live Image, we can make a few of our space constrained images better. For other images, having them be smaller will make them faster to download.

Scope

• kernel
  • Lougher needs to do a new set of patches since the ones for 2.6.34 were only partially accepted.
    • Patches for 2.6.35 were not supplied during the merge window. This makes it unlikely that they will be accepted for 2.6.35 and that 2.6.36 will be the next chance. If the patches do get accepted for 2.6.36, we can probably have the squashfs support in place for the release (maybe backporting patches, since F14 will probably release with 2.6.35). I'll need to have squashfs and livecd-creator updates ready to go. If I need to wait until the 2.6.36 merge window to know if the feature will be doable or not, there will be little time after that to get it tested. So we need to be able to start testing promptly.
    • Statements from Lougher:
      • https://bugzilla.redhat.com/show_bug.cgi?id=504425#c19
      • http://marc.info/?l=linux-kernel&m=127190194229951&w=2
    • It looks like some patches were being worked on, but weren't ready for 2.6.35 as there was no pull request for them.
  • Fallback plan for the kernel is to do nothing.
• squashfs-tools
  • Lougher's 4.1 has been built for F14.
    • It uses multiple processors for both zlib and lzma compression.
    • A test games spin had a 10% reduction (from almost 4.4 GB to just under 4 GB) in size using lzma compression, saving about 400 MB. (This was a nonfunctional image because of lack of kernel support for lzma squashfs.)
    • A live image was built using this version of squashfs (using the default zlib compression) which functioned properly. No changes were made to livecd-creator to make the image.
  • The fallback plan if squashfs-tools works (or at least doesn't have regressions for zlib compression) but some other part of feature can't be completed is to continue using the dev version.
  • The fallback plan if the dev version of squashfs-tools is found to have regressions affecting zlib compression, is to revert by untagging the build if this apparent before the alpha release. After the alpha release we'd want to bump the epoch.
• livecd-creator
  • livecd-creator needs to be modified to have mksquashfs use lzma compression. (Probably as the default, with other compression methods specifiable as command line options.)
  • If possible this should be usable significantly ahead of feature freeze so that spin owners can be pretty safe relying on it for their spins.
  • Proposed changes to livecd-creator:
    • Add a --compression_type= parameter that takes a string that will normally be passed to mksquashfs.
    • Add a default compression type config. This will initially be 'zlib', but will be changed to 'lzma', once we believe that is a better default for most people. (After there aren't regressions and performance is know to be acceptible in most cases.)
    • Add a default compression for mksquashfs type, so that if the compression type requested (or defaulted) matches this type, no compression option is specified for mksquashfs. This will allow the use of older versions of squashfs-tools as long as the mksquashfs default ('zlib') compression type is used.
    • Document the 'zlib' and 'lzma' types. Warn that 'lzma' needs kernel support for mounting and live images.
  • The simple fallback plan for livecd-creator is to document that only 'zlib' compression is available currently. If the default was changed to 'lzma', change it back to 'zlib. The more intrusive fallback is to revert all of the compression type changes.
  • A patch is undergoing testing now.

How To Test

• We need to make sure live images work.
• We need to make sure live images are smaller using LZMA compression.
• We need to make sure performance is acceptable.
  • Using a live image needs to not be noticeably slower (hopefully it will be faster with typcial hardware) with LZMA than with zlib compression.
  • Making a compressed file system using LZMA can be somewhat slower than with zlib, because commonly time isn't as critical when making the image (compared to using it).
  • We need to test unsquashing squashed images work using unsquashfs (both LZMA and zlib).
• Specific test cases:
  • Squashfs 4.1
    • No --compression-type
      • Should build using zlib and image should work.
    • --compression-type=zlib
      • Should build using zlib and image should work.
    • --compression-type=lzma
      • Should build using lzma and image should work with a kernel that supports lzma squashfs.
      • The image should be smaller than with zlib compression.
      • The image should run nearly as fast or faster than with zlib compression on common hardware.
      • The time needed to build the image should not be inordinately longer than with zlib on common hardware.
    • --compression-type=garbage
      • A reasonable error message should be displayed
  • Squashfs 4.0
    • No --compression-type
      • Should build using zlib and image should work.
    • --compression-type=zlib
      • Should build using zlib and image should work.
    • --compression-type=lzma
      • A reasonable error message should be displayed
    • --compression-type=garbage
      • A reasonable error message should be displayed

User Experience

Live images will be smaller or have more included packages.

Dependencies

• squashfs-tools
• kernel (may need to ask for backport)
• livecd-tools

Contingency Plan

• Since the default in the new squashfs is to use zlib compression, we shouldn't have to revert squashfs-tools.
• livecd-creator would need to have at least some of the changes reverted.

Documentation

• Most likely livecd-creator would be getting a new option and would need its help updated along with the code changes.

Release Notes

• Mention the squashfs version change, that it supports LZMA compression, point to its documentation and note that it is supported in the kernel as well.
• Mention that livecd-creator uses LZMA by default, how to get zlib compression if desired for use with older kernels or other reasons.

Comments and Discussion

• See Talk:Features/LZMA_for_Live_Images