FedoraAtomicCI/pipeline

CI-Pipeline Architecture and Design

In this section we will describe the different steps that the pipeline aims to introduce and support.

Stage 0. The packager opts into the pipeline for one or more packages and agrees the tests of these packages will be gating their push to the Fedora repositories.

• The tests are contained in dist-git repos and can be invoke via Ansible following the Changes/InvokingTestsAnsible proposal.
• The packager should be able to opt into which tests and test suites gate their package.

• Goal: eventually all packages involved in Fedora Atomic Host will have opted into the pipeline.

Stage 1. Trigger: The pipeline starts after a commit is pushed to dist-git repo

• Once one or more commits are pushed to Fedora dist-git, a git hook triggers a fedmsg message. (fedmsg and Jenkins integration is via the Jenkins JMS Plugin)
• The change is what we is tested in the pipeline against the known-good (already merged) set of changes.
• Only Fedora Atomic Host packages will be monitored for changes.

• Requirement: This dist-git commit should not yet affect other packagers.

Stage 2. The package is built with the new change

• The build occurs in Jenkins using Mock.
• In order to meet meet the requirements for CI/CD, interacting with Koji in some form is required.
• Once the pipeline is triggered as part of the build process if a %check section exists in the spec file to run unit-tests they will be executed.
• The end result is package will be produced to then be used for further testing. Success or failure will result with a fedmsg back to the Fedora reviewer/maintainer.
  • Build result on the fedmsg bus will be plumbed through to bodhi via resultsDB

Stage 3: Functional Tests on Package

• Functional tests will be executed on the produced package from the previous stage of the pipeline if they exist. This will help identify issues isolated to the package themselves.
• Success or failure will result with a fedmsg back to the Fedora reviewer/maintainer.
  • Functional tests results on the fedmsg bus will be plumbed through to bodhi via resultsDB

Stage 4. The package built is pulled into a compose

• If functional tests are successful in the previous stage of the pipeline then an OStree compose is generated.
• The compose occurs in Jenkins
  • Compose result on the fedmsg bus will be plumbed through to bodhi via resultsDB

Stage 5: Execute Integration Tests on OStree

• Integration tests are run on the OStree compose.
• Success or failure will result with a fedmsg back to the Fedora reviewer/maintainer.
  • Integration tests result on the fedmsg bus will be plumbed through to bodhi via resultsDB

Stage 6: e2e Conformance Tests on Openshift Clusters

• If integration tests of the images are successful an Openshift cluster will be configured using the Atomic Openshift Installer with the new Fedora Atomic Host image as the base system.
• Once the cluster is configured Kubernetes conformance tests will run.
• Success or failure will result with a fedmsg back to the Fedora reviewer/maintainer.
  • e2e Conformance Integration tests result on the fedmsg bus will be plumbed through to bodhi via resultsDB

Stage 7: Image Generated From Successful Integration Tests On OStree

• An image will be initially generated at a certain interval when there has been successful integration test execution on an OStree compose.
• Success or failure will result with a fedmsg back to the Fedora reviewer/maintainer.
  • Image build result on the fedmsg bus will be plumbed through to bodhi via resultsDB

Stage 8: Image Smoke Test Validation

• The validation left is to make sure the image can boot and more smoke tests may follow if required.
• Success or failure will result with a fedmsg back to the Fedora reviewer/maintainer.
  • Image smoke tests result on the fedmsg bus will be plumbed through to bodhi via resultsDB

Gating

• Builds are gated in Bodhi.
• If a test passes CI then that change is allowed through Bodhi. At this point the change starts to affect further development. This results in future changes going through CI being layered on top of this change.
• If any tests fail, then the change will not be allowed through Bodhi (packager will be notified of failures), and no later runs of CI will include the change. A different change would then be run through the pipeline.

• Requirement: A package that has failed testing in the pipeline is gated. It does not affect other developers or further runs of the pipeline.

This is how the gating we envision the gating being done today:

Notifications

1. The CI system will 1 fedmsg message for each commit that it will not test (because the packager did not opt-in).
2. For each commit tested, the CI system will send 1 fedmsg message per test run, so there will be a total of 3 fedmsg messages sent:
   • 1 testing the new RPM
   • 1 testing the compose containing the new RPM
   • 1 testing the image built from the new compose
   • Each message will contain information about the namespace, git repository and hash of the commit that triggered the test run
3. A system will be recording each message from the CI system and index them with the namespace, git repository and hash of the commit that they were testing (included ignored commits)
4. When a bodhi update is created, a system will query the database described above to check if there are already known test results, if there are, the update will be flag accordingly
5. When a bodhi update is created, a system will add to the database described above the identifier of the bodhi update and link it to the namespace, git repository and hash of the commit of the builds the update is composed of
6. The system updating bodhi will check for each new results published if there is a bodhi update related to it and if so flag the update accordingly

Flags

We will be adding four different states to bodhi updates:

• TEST_SKIPPED: when all commits of the builds composing the update have all been ignored by the CI system
• TEST_QUEUED: when at least one commit of the builds composing the update have been queued by the CI system for tests
• TEST_RUNNING: when at least one commit of the builds composing the update are being tested in the CI system
• TEST_FAILED: when at least one' commit of the builds composing the update has failed at least one of its tests in the CI system
• TEST_PASSED: when all commits of the builds composing the update have passed their tests in the CI system

These flags will be reflected in the UI, so packagers will be able to see in which stage an update is.

• Requirements: Only updates flagged as TEST_SKIPPED or TEST_PASSED will be allowed to be pushed to updates-testing