Changes/Two Week Atomic

Two Week Atomic

Summary

Fedora Atomic Host is an implementation of the Project Atomic pattern for a specialized operating system for the deployment of containerized applications. For the past two Fedora releases, we've included an Atomic Host cloud image as a non-blocking deliverable. However, upstream Atomic is moving very fast — by the end of the alpha, beta, final stabilization cycle Fedora uses, the released artifact is basically obsolete. Additionally, the Project Atomic team at Red Hat would like to do their ongoing development work in the Fedora upstream, and the six-month release cycle does not lend itself to that.

This change moves Atomic away from the main Fedora 6-month distribution release, and instead to separate releases every two weeks on a new web site, http://atomic.fedoraproject.org/

Owners

• Names: Matthew Miller, Colin Walters, Joe Brockmeier, Kushal Das, Adam Miller, Dennis Gilmore
• Email: use the Atomic Devel mailing list

• Release notes owner:

Current status

• Targeted release: Fedora 23 (but actually ideally will start with F22, long before F23 release)
• Last updated: 2015-06-17
• Tracker bug: <will be assigned by the Wrangler>

Detailed Description

Delivery

1. Move Atomic away from the main Fedora 6-month distribution release, including:

• Possible removal from http://getfedora.org/ — decision to be made in collaboration with Cloud WG, Design Team, and Websites.
• F23 atomic images won't be on the regular mirrors

2. Updated Fedora Atomic Host images produced every two weeks, and presented at:

• New website at http://atomic.fedoraproject.org/ (may just point to https://getfedora.org/en/cloud/download/atomic.html or to http://labs.fedoraproject.org/en/atomic-host/)
• Images at http://alt.fedoraproject.org/pub/alt/fedora-atomic/
• Under http://projectatomic.org/releases/ (WIP)

Image Production

1. Images will be produced nightly (See Release Engineering Ticket #6196.)

• qcow2 / raw.xz
• vagrant variant
• (Installer ISO and PXE-to-Live?)

2. These images will be built using Anaconda's ostree-target mode from a nightly tree built from the current Fedora release (i.e., currently Fedora 22) plus updates — this may include updates-testing.

• will use current-release anaconda for this
• may need a mechanism to include an updates.img

3. When the next Fedora release (e.g., F23) branches, those images will also start production (but may not be the target for release; see the Release section below).

Testing

• Testing will be almost entirely automated and will not require extra resources from the QA team.

1. Whenever a new image is produced, a listener will receive the associated message on the Fedora message bus
2. a battery of tests will be automatically executed (initially using tunir; the goal is to migrate to Tasktron when that is ready)
3. test results will be fed back to fedmsg
4. and available on a dashboard somewhere

• If no image is successfully produced, a tracker bug should be automatically filed
  • if an open tracker bug already exists, just add a comment rather than filing a new one
  • if such a tracker bug is open, a successful build should auto-close it
• Of course, QA team expertise and help is always welcome
• There should also be a mechanism for manually marking an image as failed even if it passes the automatic tests

Release

As with testing, the goal is for this process to be entirely automatic (after initial development, of course).

1. Every two weeks, a process will scan the fedmsg history for image builds which have passed the tests, and

• copy to https://dl.fedoraproject.org/pub/alt/fedora-atomic/
• upload to EC2 and other providers via Fedimg
• publish links to download or launch to http://atomic.fedoraproject.org/
  • move older links to archive
• send an automated announcement email

2. If no image passed the tests since the previous image was posted:

• leave link to previous image
• include warning text that this is stale
• link to most recent failure-tracking bug (see above)
• decision! either:
  • a. rerun nightly until a successful image is found
  • b. just skip this cycle, to keep releases predicable

3. Bonus feature: ability to revoke a published image

• connected to the ability to manually fail an image
• on that trigger, rescan back for previous-good image
• update website linking to that, warning that the previous one was bad

The releases will be based on a current, non-EOL Fedora release, but the decision to switch to a newer base will be decoupled from release dates for the main distribution. For example, Fedora Atomic Host may be initially built on Fedora 22, and when Fedora 23 is released, the image featured on the website may remain based on F22 for another month. Or, if newer features are needed more quickly, we might switch to the F23 beta as base before the final release. (The website will bear an appropriate disclaimer, similar to that for prerelease Fedora alpha/beta downloads.)

Docs and Website

The http://atomic.fedoraproject.org/ website will need initial design and creation, of course, but after that it is intended to be automated; no one should have to update links automatically. It should be a very simple design, with pointers to Fedora docs (hopefully, soon, new short-easy-docs website) and to http://projectatomic.io/

In addition to basic "What is this all about?" information, the website will need text properly setting expectations for development status, lack of non-automated testing, support level ("you can help us fix it!"), and so on.

Naming and Trademark

We would like to use the name "Fedora Atomic Host", and phrasing like "Fedora Atomic Host, based on Fedora 22". This highlights the fact that while this project is under the Fedora umbrella, it isn't part of the traditional distribution release cycle. (The various images will be distinguished by date rather than by number.)

Fedora Atomic Host was previously presented as part of the general Fedora release under Fedora Cloud, but this new, separate approach should get council approval under the Trademark Guidelines for new combinations of unmodified Fedora software.

Marketing

The Cloud WG had previously considered making Fedora Atomic Host its main offering, however at present Atomic is not ready to fill that role, as it is changing too fast and not flexible enough for all use cases the Cloud WG targets. The Cloud WG / Cloud SIG will need to find other selling points. It may be that after several cycles, Atomic will mature enough to be the primary Fedora Cloud offering. See this related Cloud WG proposal for more background.

The two-week cycle means no big semi-annual PR blitz, but interesting new features and functionality should come rather frequently. This may provide an opportunity for showcasing Fedora in the middle of our development cycle, when Fedora usually doesn't get much press.

Benefit to Fedora

• Fedora Atomic Host puts Fedora in the spotlight as a leader in modern operating system innovation
• Fedora becomes the proper upstream for Atomic Host development
• Early-adopters interested in collaborating on Atomic development have a public place to do so
• Fedora users interested in Atomic get a better experience
• Provides a test case for delivering parts of Fedora at different rates
• Provides prototype for (closer to) continuous delivery of Fedora software, which will provide useful lessons we can apply elsewhere

Scope

An overview of scope is laid out here. See Two Week Atomic RACI Matrix for an individual breakdown of responsibilities. This also includes a basic, task-level accounting of status, which we will keep updated.

Proposal owners

• Update koji for nightly image builds

• Create automatic test system

1. listener for successful builds
2. automatic test execution
3. results to fedmsg
4. results dashboard
5. mechanism to mark a build as bad even if automatic tests pass
6. Automatic filing of ticket or bug if there are no successful builds

• Create automatic release system

1. every two weeks, scan for images which pass all tests
2. integration with fedimg
3. upload to alt.fpo
4. update website
5. email announcement
6. fallback mechanism for no builds in two weeks

• Create new website if websites team unavailable for this effort

• Coordination, cheerleading

Other developers

• Websites:
  • Drop Atomic from http://getfedora.org/ for F23 (and possibly before)
  • Perhaps a sidebar / banner linking to new http://atomic.fedoraproject.org/
  • New http://atomic.fedoraproject.org/ needs creation — we'll need help from the websites team or need to find someone else interested (n.b. this may actually be a redirect to somewhere on http://labs.fedoraproject.org/ rather than a new, stand-alone site).
• Design:
  • Help with new website
  • Possibly new logo

• Cloud SIG:
  • Help with tunir and the automatic testing component
  • Fedimg integration with the auto-release tool

• Quality Assurance:
  • Help with tests would be awesome

• Package Maintainers:
  • If updates to packages which are included in Atomic happen to break the image, package maintainers will be relied on to fix the update. That's the case for any update regardless of this change, but if the problem happens to be only triggered on Atomic, the change owners will help coordinate fixes and testing.

Release engineering

The primary release engineering effort is the nightly production of images. The current release engineering tools have all the parts needed to do this, but have never been used in that way before, so some development effort is required to enable the production of current + updates nightlies. (This work, however, many also benefit other areas of the project, like updated Cloud Base images or even Workstation live CDs.)

Additionally, as described above, the tool to automatically ship auto-accepted images every two weeks logically falls under the rel-eng banner.

Adam Miller, on the release engineering team, is specifically prioritizing this work.

Other

• Policies and guidelines: none

• Trademark approval: ticket to be filed with the Council

Upgrade/compatibility impact

It will be possible to do an Atomic upgrade of a system installed with the F22 Atomic to the tree matching the images produced in the new model. Users are recommended to use the latest image for new systems.

How To Test

This comes down to:

A. Is there a web page at http://atomic.fedoraproject.org?

B. Can I download or launch images from it?

C. Is the web page updated with fresh images every other week?

From the perspective of this change itself, testing the actual functionality of the images is a separate problem. (That is, this change involves the image production and running those images through a test framework. The actual contents of the image and the nature of the tests are out of scope.)

User Experience

1. Shiny new Fedora Atomic web site
2. Fresh images always available with latest updates to packages relevant to Atomic
3. Developers hacking on Fedora Atomic Host can jump right in, and have their changes shipped to users quickly
4. Fedora Cloud download page less of a maze

Dependencies

In addition to the development listed above, this change relies on current-release Anaconda continuing to be able to install the current release + updates. If some change in the updates prevents this, a patch to anaconda may need to be included in the build system via updates.img.

This also assumes that tunir or another test framework is in a complete enough state to execute the required tests.

Contingency Plan

• Contingency mechanism: Three possibilities. The first is to revert to six-month cycle releases as current. The second is to do as much of the above plan as possible, but as a Fedora Remix. This would be unfortunate, as we'd really like Fedora Atomic Host to be a genuine member of the Fedora family, and it is indeed all built from official Fedora software, so nothing in the contents should necessitate relegation to remix status. But, it is there as a possibility. And the third is to temporarily drop any Fedora Atomic release — just as if the F22 non-blocking deliverable had a last-minute problem.
• Contingency deadline: Should decide by Alpha; beta freeze is the final decision point.
• Blocks release? No.
• Blocks product? No.

Documentation

The http://atomic.fedoraproject.org/ will contain brief introductory documentation, and link to http://www.projectatomic.io/ and whatever Fedora docs are relevant. It would be nice to start building up short howto and recipe-style documents on Fedora Atomic Host within Fedora Docs.

Release Notes

Fedora Atomic Host is an implementation of the Project Atomic pattern for a specialized operating system for the deployment of containerized applications. For the past two Fedora releases, we've included an Atomic Host cloud image as a non-blocking deliverable. However, upstream Atomic is moving very fast — by the end of the alpha, beta, final stabilization cycle Fedora uses, the released artifact is basically obsolete. Additionally, the Project Atomic team at Red Hat would like to do their ongoing development work in the Fedora upstream, and the six-month release cycle does not lend itself to that.

After Fedora 23, we have moved Atomic away from the main Fedora 6-month distribution release, and instead to separate releases every two weeks on a new web site, http://atomic.fedoraproject.org/