First-Class Cloud Images
Summary
This feature expands Fedora's current cloud image deliverables beyond just EC2, and overhauls how they are produced. A goal is to produce cloud images for EC2 and other cloud deployments for the Alpha, Beta, and Final compose process and distribute them on the mirror network. There will also be nightly or weekly image builds for Rawhide to assist with early development. All images should be constructed using a newer generation of tools.
Owner
- Name: Matthew Miller
- Email: mattdm at fedoraproject dot org
Current status
- Targeted release: Fedora 19
- Last updated: January 20, 2013
- Percentage of completion: 0%
Detailed Description
- New images that can be used in other cloud deployments (such as OpenStack, CloudStack, or Eucalyptus) will be produced. They will be in a qcow2 format and lack the EC2-specific customization. Images for this feature would ideally work for all cloud deployments and there will be i686 and x86_64 versions of both image types. In total and "image drop" will have 4 images: 2 arches for 2 different types (EC2, not-EC2).
- An image drop will be produced for Alpha, Beta, and Final composes for Fedora 19 and forward.
- Scratch build image drops will be produced on a weekly basis for Fedora 19.
- Scratch build image drops will be produced on a weekly basis for Rawhide as well to enable early testing.
- The Fedora Koji instance needs to be updated to a future release that will integrate with ImageFactory and Oz from the Aeolus Project. This future release is not implemented yet.
- The EC2 images will be automatically uploaded and registered in EC2. The Final AMIs for Fedora 19 will be available in the Amazon marketplace.
Benefit to Fedora
- Cloud images more easily available to users
- Cloud images available for better testing
- Continuous building of images provides new opportunities for early platform testing
- appliance tools, which is the workhorse for image building today, does not have an upstream. ImageFactory does.
- appliance-tools uses chroots which suffer from build-time complications like kernel mismatches. Moving off of this tool will unburden Rel-Eng with that work.
Scope
ImageFactory/Oz Integration with Koji
Creating LiveCDs with the same system will also require some changes to ImageFactory/Oz. These will use existing technology in livemedia-creator (not to be confused with livecd-creator). If possible, the image building process will be executed within a chroot to reduce the support burden on Rel-Eng and follow the design spirit of Koji. If this is not possible in the time alotted, ImageFactory/Oz will need to be installed on the build hosts, and kojid will make use of them when it takes an image building task. (this has an increased support burden on Rel-Eng.
ImageFactory/Oz must be capable of consuming raw kickstarts for this feature. Use of the XML templates will be reviewed at a later date.
Build System Update Deployment
This feature requires a significant change to Koji that will need to be deployed to the production build system. ImageFactory/Oz builds the images inside a small VM, and because a nested virt scenario is not possible on RHEL 6 (which is what the builders are) it will require bare metal builders to be available.
There are 2 bare-metal builders available today to accommodate this requirement. (thank you Dennis Gilmore)
Process and Infrastructure Updates
Release Engineering will produce image drops on a weekly basis and for milestone updates. These will need to be easily discoverable so that announcements and communication about their release is easily consumed. (from the Cloud SIG web page, for example). Procedures for producing, testing, and blessing the images should be documented and communicated as well.
Milestone image drops will be released using the current Fedora mirroring system, alongside the install images.
Milestones
These dates are approximate pending the official Fedora 19 schedule is released.
- Prototype implementation of Koji-Imagefactory/Oz: February 28th, 2013
- Code landed in upstream Koji project: March 22nd, 2013
- Fedora Koji builders updated: March 29th, 2013
- Scripting for automatic (Rawhide/F19) weeklies: April 12th, 2013
- Weekly image drops made available on http://alt.fedoraproject.org/pub/alt/cloud/: April 26th, 2013
- Image drops for Alpha, Beta, and Final milestones will be on the mirrors on the same date as those milestones.
How To Test
Since images are composed of packages it would be redundant to test all package functionality in each image in each cloud environment. It should be sufficient to verify that the image boots and is capable of getting yum updates if the cloud environment is configured to provide them or the image has network access to the internet.
- Do the EC2-specific images exist in EC2?
- Are they bootable?
- Does a yum update successfully retrieve updates?
- Are the non-EC2 images available?
- expected formats: qcow2, raw.tar.xz
- Do the downloadable images boot in OpenStack?
- Do the downloadable images boot in Eucalyptus?
- Do they appear similar to one installed by Anaconda?
- Are they On the mirrors for Alpha, Beta, and Final?
User Experience
Milestone ("Official") cloud images must be downloadable from the mirror system. They should also be well announced and discoverable, perhaps on a wiki somewhere. The EC2 images must be registered as AMIs and browsable in the AWS console in all regions. It is desirable to see them in the Quickstart guide too, or the Amazon Marketplace at no additional charge beyond the usual infrastructure costs.
Weekly Rawhide images should follow the same criteria above minus the Quickstart and Marketplace items.
Dependencies
The ImageFactory/Oz changes are pending upstream approval.
The Koji integration with ImageFactory/Oz is pending upstream approval.
Contingency Plan
Since this feature involves a few moving parts, they each have their own contingency plan.
Koji Integration
If the Koji integration work is not completed before Fedora 19 is branched, consider updating Fedora Koji to the 1.7.1 release instead, which tracks images using the same data model as RPMs. This will at least improve the manageability if the images produced and enable some level of automation to track the latest.
Livemedia-creator could be used instead of Oz, but since the integration work is about the same, it would be pointless to begin that task late in the Fedora 19 release cycle.
Continuous Image Building
Building images requires a fair amount of disk space. If it proves to be too aggressive, we could throttle their creation to a bi-weekly task, or make the lifetime of the scratch images shorter.
Documentation
See Features/FirstClassCloudImages/Whiteboard
Possibly a small readme file should go alongside the images. Primary documentation on Cloud SIG web page.
Release Notes
TBD