Changes/Two Week Atomic

Two Week Atomic

Summary

Fedora Atomic Host is an implementation of the Project Atomic pattern for a specialized operating system for the deployment of containerized applications. For the past two Fedora releases, we've included an Atomic Host cloud image as a non-blocking deliverable. However, upstream Atomic is moving very fast — by the end of the alpha, beta, final stabilization cycle Fedora uses, the released artifact is basically obsolete. Additionally, the Project Atomic team at Red Hat would like to do their ongoing development work in the Fedora upstream, and the six-month release cycle does not lend itself to that.

This change moves Atomic away from the main Fedora 6-month distribution release, and instead to separate releases every two weeks.

Owners

• Names: Matthew Miller, Colin Walters, Joe Brockmeier, Kushal Das, Adam Miller, Dennis Gilmore
• Email: use the Atomic Devel mailing list

• Release notes owner:

Current status

• Targeted release: Fedora 23 (but actually ideally will start with F22, long before F23 release)
• Last updated: 2015-10-22
• Tracker bug: bz #1238411
• Ongoing Work Tracking (Taiga Kanban board): http://taiga.cloud.fedoraproject.org/project/acarter-fedora-docker-atomic-tooling/kanban

October video demo of tools ready for post-Fedora-23 release: https://www.youtube.com/watch?v=r1jrqClho3U

Detailed Description

Overview

1. Move Atomic away from the main Fedora 6-month distribution release, including:

• Possible removal from http://getfedora.org/ — decision to be made in collaboration with Cloud WG, Design Team, and Websites.
• F23 atomic images won't be on the regular mirrors

2. Updated Fedora Atomic Host images produced every two weeks, and presented at https://getfedora.org/en/cloud/download/

• Images at http://alt.fedoraproject.org/pub/alt/fedora-atomic/
• Under http://projectatomic.org/releases/ (WIP)

Schematic

Image Production

1. Images will be produced nightly (See Release Engineering Ticket #6196.)

• qcow2 / raw.xz
• vagrant variant
• Installer ISO and PXE-to-Live? (eventually wanted; initially lower priority if they present a problem)

2. These images will be built using Anaconda's ostree-target mode from a nightly tree built from the current Fedora release (i.e., currently Fedora 22) plus updates — this may include updates-testing.

• will use current-release anaconda for this
• may need a mechanism to include an updates.img

3. When the next Fedora release (e.g., F23) branches, those images will also start production (but may not be the target for release; see the Release section below).

Note that Fedora Atomic Host is currently x86_64 only. Other architectures may be added in the future.

Testing

• Testing will be almost entirely automated and will not require extra resources from the QA team.

1. Whenever a new image is produced, a listener will receive the associated message on the Fedora message bus
2. a battery of tests will be automatically executed (may initial use tunir if Taskotron is not ready; the goal is to migrate to Tasktron when possible)
3. test results will be fed back to fedmsg
4. and available on a dashboard somewhere

• If no image is successfully produced, a tracker bug should be automatically filed
  • if an open tracker bug already exists, just add a comment rather than filing a new one
  • if such a tracker bug is open, a successful build should auto-close it
• Initially, failure of any test will fail the entire image; "advisory" tests which can fail but still allow the image through might be nice to have.
• Of course, QA team expertise and help is always welcome
• There should also be a mechanism for manually marking an image as failed even if it passes the automatic tests

Release

As with testing, the goal is for this process to be entirely automatic (after initial development, of course).

1. Every two weeks, a process will scan the fedmsg history for image builds which have passed the tests, and

• copy to https://dl.fedoraproject.org/pub/alt/fedora-atomic/
• upload to EC2 and other providers via Fedimg
• publish links to download or launch to https://getfedora.org/en/cloud/download/
  • this can be done by publishing ami ids and URLs for images to fedmsg; website is built on a schedule by cron and that task can query datagrepper for latest release
  • move older links to archive
• send an automated announcement email

2. If no image passed the tests since the previous image was posted:

• leave link to previous image
• include warning text that this is stale
• link to most recent failure-tracking bug (see above)
• decision! either:
  • a. rerun nightly until a successful image is found
  • b. just skip this cycle, to keep releases predicable

3. Bonus feature: ability to revoke a published image

• connected to the ability to manually fail an image
• on that trigger, rescan back for previous-good image
• update website linking to that, warning that the previous one was bad

The releases will be based on a current, non-EOL Fedora release, but the decision to switch to a newer base will be decoupled from release dates for the main distribution. For example, Fedora Atomic Host may be initially built on Fedora 22, and when Fedora 23 is released, the image featured on the website may remain based on F22 for another month. Or, if newer features are needed more quickly, we might switch to the F23 beta as base before the final release. (The website will bear an appropriate disclaimer, similar to that for prerelease Fedora alpha/beta downloads.)

Docs and Website

The new https://getfedora.org/en/cloud/download/ website will need design work and coding, of course, but after that it is intended to be automated; no one should have to update links automatically. It can follow the current design, with the addition of automated links and EC2 ids

It should also have pointers to Fedora docs and to http://projectatomic.io/.

In addition to basic "What is this all about?" information, the website will need text properly setting expectations for development status, lack of non-automated testing, support level ("you can help us fix it!"), and so on.

More detail on changes at Changes/Two_Week_Atomic/Website.

Naming and Trademark

We will to use the name "Fedora Atomic Host", and phrasing like "Fedora Atomic Host, based on Fedora 22". This highlights the fact that while this project is under the Fedora umbrella, it isn't part of the traditional distribution release cycle. (The various images will be distinguished by date rather than by number.)

This has been granted trademark approval by the Fedora Council, under the Trademark Guidelines for new combinations of unmodified Fedora software.

Marketing

The Cloud WG had previously considered making Fedora Atomic Host its main offering, however at present Atomic is not ready to fill that role, as it is changing too fast and not flexible enough for all use cases the Cloud WG targets. The Cloud WG / Cloud SIG will need to find other selling points. It may be that after several cycles, Atomic will mature enough to be the primary Fedora Cloud offering. See this related Cloud WG proposal for more background.

Update! Cloud WG has made Atomic the primary offering. Need marketing material updates around that.

The two-week cycle means no big semi-annual PR blitz, but interesting new features and functionality should come rather frequently. This may provide an opportunity for showcasing Fedora in the middle of our development cycle, when Fedora usually doesn't get much press.

Benefit to Fedora

• Fedora Atomic Host puts Fedora in the spotlight as a leader in modern operating system innovation
• Fedora becomes the proper upstream for Atomic Host development
• Early-adopters interested in collaborating on Atomic development have a public place to do so
• Fedora users interested in Atomic get a better experience
• Provides a test case for delivering parts of Fedora at different rates
• Provides prototype for (closer to) continuous delivery of Fedora software, which will provide useful lessons we can apply elsewhere
  • In fact, same mechanism can be used to produce updated Fedora Cloud Base and Fedora Docker Base images, which the Cloud WG has been interested in for a long time

Scope

Proposal owners

• Update koji for nightly image builds

• Create automatic test system

• Create automatic release system

• Create new website if websites team unavailable for this effort

• Coordination, cheerleading

Task matrix

See Two Week Atomic RACI Matrix for an individual breakdown of tasks and responsibilities. This also includes a basic, task-level accounting of status, which we will keep updated.

Other developers

• Websites:
  • Drop older Atomic images from http://getfedora.org/ for F23 (and possibly before)
  • New version of page needs creation — we'll need help from the websites team or need to find someone else interested
• Design:
  • Help with website update
  • Possibly new logo

• Cloud SIG:
  • Help with tunir and the automatic testing component
  • Fedimg integration with the auto-release tool
  • Could use same or similar mechanism for refreshed Fedora Cloud Base images.

• Quality Assurance:
  • Help with tests would be awesome

• Package Maintainers:
  • If updates to packages which are included in Atomic happen to break the image, package maintainers will be relied on to fix the update. That's the case for any update regardless of this change, but if the problem happens to be only triggered on Atomic, the change owners will help coordinate fixes and testing.

Release engineering

The primary release engineering effort is the nightly production of images. The current release engineering tools have all the parts needed to do this, but have never been used in that way before, so some development effort is required to enable the production of current + updates nightlies. (This work, however, many also benefit other areas of the project, like updated Cloud Base images or even Workstation live CDs.)

Additionally, as described above, the tool to automatically ship auto-accepted images every two weeks logically falls under the rel-eng banner.

Adam Miller, on the release engineering team, is specifically prioritizing this work.

Other

• Policies and guidelines: none

• Trademark approval: Granted — see Council ticket #33

Upgrade/compatibility impact

It will be possible to do an Atomic upgrade of a system installed with the F22 Atomic to the tree matching the images produced in the new model. Users are recommended to use the latest image for new systems.

How To Test

This comes down to:

A. Is there a web page at https://getfedora.org/en/cloud/download/?

B. Can I download or launch images from it?

C. Is the web page updated with fresh images every other week?

From the perspective of this change itself, testing the actual functionality of the images is a separate problem. (That is, this change involves the image production and running those images through a test framework. The actual contents of the image and the nature of the tests are out of scope.)

User Experience

1. Shiny new Fedora Atomic web site
2. Fresh images always available with latest updates to packages relevant to Atomic
3. Developers hacking on Fedora Atomic Host can jump right in, and have their changes shipped to users quickly
4. Fedora Cloud download page less of a maze

Dependencies

In addition to the development listed above, this change relies on current-release Anaconda continuing to be able to install the current release + updates. If some change in the updates prevents this, a patch to anaconda may need to be included in the build system via updates.img.

This also assumes that tunir or another test framework is in a complete enough state to execute the required tests.

Contingency Plan

• Contingency mechanism: Three possibilities. The first is to revert to six-month cycle releases as current. The second is to do as much of the above plan as possible, but as a Fedora Remix. This would be unfortunate, as we'd really like Fedora Atomic Host to be a genuine member of the Fedora family, and it is indeed all built from official Fedora software, so nothing in the contents should necessitate relegation to remix status. But, it is there as a possibility. And the third is to temporarily drop any Fedora Atomic release — just as if the F22 non-blocking deliverable had a last-minute problem.
• Contingency deadline: Should decide by Alpha; beta freeze is the final decision point.
• Blocks release? No.
• Blocks product? No.

Documentation

The https://getfedora.org/en/cloud/ "brochure" page will contain brief introductory documentation, and link to http://www.projectatomic.io/ and whatever Fedora docs are relevant. It would be nice to start building up short howto and recipe-style documents on Fedora Atomic Host within Fedora Docs.

Release Notes

Fedora Atomic Host is an implementation of the Project Atomic pattern for a specialized operating system for the deployment of containerized applications. For the past two Fedora releases, we've included an Atomic Host cloud image as a non-blocking deliverable. However, upstream Atomic is moving very fast — by the end of the alpha, beta, final stabilization cycle Fedora uses, the released artifact is basically obsolete. Additionally, the Project Atomic team at Red Hat would like to do their ongoing development work in the Fedora upstream, and the six-month release cycle does not lend itself to that.

After Fedora 23, we have moved Atomic away from the main Fedora 6-month distribution release, and instead to separate releases every two weeks.