Features/LZMA for Live Images

LZMA for Live Images

Summary

Enabling LZMA for compression to allow us ship more software on our live images.

Owner

• Name: Bruno Wolff III

• Email: <bruno@wolff.to>

Current status

• Targeted release: Fedora 15
• Last updated: 2011-01-21
• Percentage of completion: 25%

Detailed Description

LZMA compression typically provides better compression than gzip while still providing relatively fast uncompression. Once squashfs file systems are supported by the kernel we can use them for building live images. A test with the games spin resulted in a 8.7% reduction (from 4242538496 bytes to 3873439744 bytes) in size of the resulting live image.

Benefit to Fedora

By allowing more packages to be put on a Live Image, we can make a few of our space constrained images better. For other images, having them be smaller will make them faster to download.

Scope

• kernel
  • Lougher's patches have been pulled for 2.6.38 which is expected to be the release kernel for F15.
  • We still need for Fedora to start using the 2.6.38 for rawhide before doing some of the testing. This should happen in a few weeks.
    • Kyle has started do scratch builds for 2.6.38 in koji and I have done a successful test of mounting an xz squashfs image with one of those kernels.
  • Fallback plan for the kernel is to do nothing.
• livecd-creator
  • Changes to livecd-creator:
    • Change the default compression type to xz.
    • Update the documentation to reflect the changed default.
    • These changes won't be made until after the kernel feature is available, so no fallback is needed.
• squashfs-tools
  • squashfs-tools is back on the table as XZ (LZMA2) will be separate from LZMA.
  • The development version needs to get built and tested.
    • A 4.2 prerelease build is now in rawhide.
    • Things seem to have stablized, but there is a pending patch for an endian issue that I expect to get applied before upstream does the 4.2 release.
  • The squashfs-tools 4.2 release is tentatively planned for mid-February.
  • A test case has been created to make it easier to get critical path testing.

How To Test

• We need to make sure live images work.
  • I have successfully tested a live image using xz compression and a scratch build 2.6.38 kernel on a USB drive.
• We need to make sure live images are smaller using LZMA2 (XZ) compression.
  • A test games spin from 2011-01-21 rawhide + a 2.6.38 scratch build went from 4499222528 bytes to 4111771648 bytes (8.6% decrease).
• We need to make sure performance is acceptable.
  • Using a live image needs to not be noticeably slower (hopefully it will be faster with typcial hardware) with LZMA than with GZIP compression.
    • The speed seemed comparable when run off a USB drive.
  • It is OK if making a compressed file system using xz is somewhat slower than with gzip, because commonly time isn't as critical when making the image (compared to using it).
  • squashfs-tools should be tested to make sure all of the compression methods work.
• Specific test cases: (Note in early testing, the default will be gzip rather than xz.)
  • Squashfs 4.2
    • No --compression-type
      • Should build using xz and image should work.
    • --compression-type=gzip
      • Should build using gzip and image should work.
    • --compression-type=xz
      • Should build using xz and image should work with a kernel that supports xz squashfs.
      • The image should be smaller than with gzip compression.
      • The image should run nearly as fast or faster than when using gzip compression on common hardware.
      • The time needed to build the image should not be inordinately longer than when using gzip on common hardware.
    • --compression-type=garbage
      • A reasonable error message should be displayed
  • man livecd-creator
    • Should have the --compression-type option documented with the correct default. (gzip in eary testing, xz in later testing.)
  • livecd-creator --help
    • Should have the --compression-type option documented with the correct default. (gzip in eary testing, xz in later testing.)

User Experience

Live images will be smaller or have more included packages.

Dependencies

• kernel (may need to ask for backport)
  • This is unlikely. If the patches don't make it into 2.6.38, then due to kernel release timing it will be better to slip to F16.
• livecd-tools
• Spin creation process
  • I have started a dialogue with releng about this.

Contingency Plan

• Since there isn't really anything further to do until the kernel feature is available, the fallback is pretty much not to do anything.

Documentation

• Most likely livecd-creator would be getting a new option and would need its help updated along with the code changes.

Release Notes

• Mention the squashfs version change, that it supports XZ (LZMA2) compression, point to its documentation and note that it is supported in the kernel as well.
• Mention that livecd-creator uses XZ by default, how to get GZIP compression if desired for use with older kernels or other reasons.

Thanks

I would like to thank Phillip Lougher for his work on squashfs. The key work needed for this feature was done by Phillip upstream.

Comments and Discussion

• See Talk:Features/LZMA_for_Live_Images