From Fedora Project Wiki
(New consolidated CI portal)
Line 1: Line 1:
'''Continuous integration''' aims to ensure broken changes do not affect other developers, packagers, maintainers or users. Continuous delivery aims to ensure broken changes do not get delivered or released.
= Why =


= The Manifesto =
== Vision ==


Because there are several Continuous Integration efforts, we need to set the basic rules to make sure we’re all playing the same game. When we call a game “football” we need to agree on what that means.
There are hundreds of packages which make up the Operating System.
Making sure that they all work together as a whole is not an easy
task. This becomes even harder as the number of packages and their
inter-dependencies grows. An extensive testing is required before
a new version of the operating system is released to ensure it is
stable enough. That is the past.


Here is a [[CI/Manifesto|Continuous Integration manifesto]] that helps describe the goals, terminology, and rules.
Imagine an '''Always Ready Operating System''' which consists of
packages which are constantly kept in a good shape.  Integrated
and stable thanks to an extensive test coverage which is
continuously executed upon changes in individual packages, in this
way allowing to prepare a new release in much shorter time, or
even in no time.


= Tests locations =
Imagine an operating system distribution which you could release
at any moment. This is where we are heading. Here comes the CI,
Continuous Integration, as an invaluable tool to ensure everything
is working together as expected in every point of time.


The tests to be executed are stored in the dist-git repositories. The tests are stored or wrapped along-side the spec files that describe the software sources. The tests are branched, maintained along with the package or module they pertain to.
== Manifesto ==


Package or RPM specific tests are stored in that package's [https://src.fedoraproject.org/cgit/rpms/ dist-git repository]. Such package repositories will have a <code>rpms/</code> prefix in their name. Other tests refer to larger sets of functionality, [[Modularity|called modules]] and are [https://src.fedoraproject.org/cgit/modules/ stored in a dist-git repository] with <code>modules/</code> prefix in their name.
Continuous integration aims to ensure broken changes are revealed
as soon as possible and do not affect other developers, packagers,
maintainers or users. The feedback that continuous integration
provides is vital for fast paced agile delivery of software. Late
testing, long after a change occurs, does not scale to the pace of
Fedora. Learn the goals, terminology and rules for a working CI
in the manifesto.


* To checkout a dist-git repo, use [[Package_maintenance_guide|fedpkg tooling]].
* [[CI/Manifesto|Continuous Integration Manifesto]]
* To contribute tests to a dist-git repository get a [https://admin.fedoraproject.org/accounts/ Fedora Account].
* Use Pagure.io to contribute to a dist-git repository using a GitHub style interface.
** '''Done'''
** Contact: [[User:Pingou|Pierre Yves-Chibon]]
* To move or wrap downstream tests to Fedora, use https://upstreamfirst.fedorainfracloud.org/
** Contact: [[User:Tflink|Tim Flink]]


= Test format =
= How =


Tests may be written all sorts of different ways, but have to be exposed and invoked in a standard way. They are stored or wrapped in the <code>tests/</code> directory in a dist-git repository. There is a detailed description of how this is done:
There are three main pieces of the puzzle to get this nicely
working: A process which clearly defines how to discover and
execute tests, a set of tools which help to efficiently implement
the process and the tests themselves.


* [[CI/Tests|How to run, write and wrap tests]]
== Process ==
* [[Changes/InvokingTests|Standard Spec for Invoking Tests]]
* Helpful [https://pagure.io/standard-test-roles standard-test-roles for wrapping or writing tests].


= Gating =
=== Standard Test Interface ===


When a test fails, continuous integration will be able to prevent the broken package change from affecting other changes. That gating will happen in Bodhi. Packages in the core operating system, such as those in [https://getfedora.org/en/atomic/ Fedora Atomic Host] are most interesting for gating, as when they break they affect the most users.
In order to clearly distinguish test from the CI system running it
the '''Standard Test Interface''' was introduced. It clearly
defines essential terms such as test, test subject, test suite,
test framework, test result, test artifact, test system and
describes what are their responsibilities and requirements.


* Modules and packages will be able to opt into gating.
This general approach gives a nice flexibility as it does not
enforce any specific tools or frameworks to be used. Basically it
only describes how tests are discovered and where the testing
results should be stored to be processed by the automation.
 
* [[CI/Standard_Test_Interface|Standard Test Interface]]
 
=== Gating Updates ===
 
When a test fails, CI can prevent the broken change from affecting
other packages. That gating happens in Bodhi.
 
* [[CI/gating_updates|Gating Updates]]
* [[FedoraAtomicCI/gating|Proposals and options]]
* [[FedoraAtomicCI/gating|Proposals and options]]
* '''Ready'''


While gating is already technically possible we want to improve the integration of the CI pipelines before we turn this on. This involves having a mechanism to re-trigger a test run to ensure that tests failing due to transient errors don't block updates more than they should. We are also dedicated to have a high number of tests passing the pipeline before enabling gating. The progress can be followed in [https://pagure.io/fedora-ci/AtomicCi/issue/2 fedora-ci/AtomicCI#2].
=== Notifications ===
 
[https://apps.fedoraproject.org/notifications Fedora Notifications]
have been adjusted to notify by default every packager when any
step of the CI pipeline fails on one of the package they maintain.
So if you are a kernel maintainer and a commit made to the kernel
dist-git repository fails to compose an OSTree, FMN will notify
you of it.
 
Bodhi includes the CI results in its update page, just as it
already includes tests results from taskotron.
 
== Tools ==
 
=== Standard Test Roles ===
 
'''Standard Test Roles''' were implemented to enable both
automation tools and developers in their local environments to
easily execute tests. This set of ansible roles supports various
frameworks and allows to execute tests against different test
subjects (such as classic rpm package, docker container or Atomic
Host).
 
* [[CI/Standard_Test_Roles|Standard Test Roles]]
* https://pagure.io/standard-test-roles/
 
=== Metadata ===
 
Standard Test Interface defines only a very simple metadata for
selecting which tests should be run. For more complex scenarios
'''Flexible Metadata Format''' can be used:
 
* [[Flexible_Metadata_Format|Flexible Metadata Format]]
* https://github.com/psss/fmf
 
=== Pipeline ===
 
The testing '''Pipeline''' detects tests for enabled
packages, executes the test coverage and gathers the results.
Currently the pipeline is enabled for the Atomic Host packages
only.
 
* [https://github.com/CentOS-PaaS-SIG/ci-pipeline/blob/master/README.md#ci-pipeline-architecture-and-design CI Pipeline Architecture and Design]
* [[FedoraAtomicCI/pipeline|Detailed pipeline description]]
* [[FedoraAtomicCI/KojiBuilds|Build options and ideas]]
* [[FedoraAtomicCI/upstream|Upstream open-source project integration]]
* [[Fedora requirements for CI and CD]]


=== Pagure ===


== Opting-in ==
Test results from the CI pipeline are displayed in '''Pagure'''
web interface. See the commits page of respective package.
Currently tests are scheduled for new commits in branches only.
Support for pull request testing is planned in the near future.


This effort focus on the packages that are part of the Fedora Atomic Host at first, if can prepare tests for your packages but if they are not part of the Fedora Atomic Host, these tests will not be run and their results not be considered for gating. This is to limit the set of packages considered and allow an iterative approach starting small and growing from there. We do aim to provide testing for more than just the Fedora Atomic Host packages but we needed to start somewhere.
* [https://src.fedoraproject.org/ Package Sources]
* [https://pagure.io/ Pagure]


If your packages are part of Fedora Atomic Host, we strongly encourage you to join this effort. The procedure to opt-in is fairly simple
== Tests ==


* read the documentation ([[CI#Test_format| above]]) about the tests format and how to invoke them
The core of the CI success are reliable tests of a good quality,
* add these tests in a ''tests/'' folder in the dist-git repository of your packages
well selected, stable, organized and continuously maintained.


Simply adding the tests in a ''tests/'' folder in the dist-git repository of your packages will be considered as a will to have these tests run and gate your packages.
=== Test Types ===


If you have any questions about the tests or the gating, feel free to reach out to us (cf the Contact section below).
In general it makes sense to store tests as close to the upstream
as possible. So what are the appropriate test types recommended
for testing the Always Ready Operating System?


* Basic functionality tests
* Integration tests


== Notifications ==
For unit tests it usually makes more sense to store them directly
within the upstream project repository. However, in some cases it
might be worth to fetch tests for Fedora CI from the upstream
repository as well.


Notifications about CI results will be available in two ways:
=== Test Code ===


* [https://apps.fedoraproject.org/notifications FMN] has been adjusted to notify by default every packager when any step of the CI pipeline fails on one of the package they maintain. So if you are a kernel maintainer and a commit made to the kernel dist-git repository fails to compose an OSTree, FMN will notify you of it.
Tests may be written all sorts of different ways, but have to be
exposed and invoked in a standard way. Tests are enabled by
including the <code>tests/tests.yml</code> file in the package
dist-git repository as defined by the Standard Test Interface.
Test code itself can be stored directly in the dist-git or fetched
from another repository. Shared tests namespace can be used for
storing test code relevant for multiple packages.


* Bodhi will include the Fedora Atomic CI results in its update page, just as it already includes tests results from taskotron.
* [[CI/Tests|Tests]] ... How to run, write and wrap tests
* [[CI/Share_Test_Code|Share Test Code]] ... Shared tests namespace


= Testing pipeline =
=== Test Execution ===


The pipeline is under development
Executing a test written with the use of Standard Test Roles is as
simple as running an ansible playbook <code>ansible-playbook
tests.yml</code>. However, a set of environment variables needs to
be set properly in order to execute the test against the desired
test subject. The '''Tests''' wiki contains detailed instructions
about running tests and adding new test coverage.


* Source Code: https://github.com/CentOS-PaaS-SIG/ci-pipeline
* [[CI/Tests|Tests]] ... How to run, write and wrap tests
* Documentation: https://github.com/CentOS-PaaS-SIG/ci-pipeline/blob/master/README.md
 
* [[FedoraAtomicCI/pipeline|More pipeline description]]
=== Test Porting ===
* [[FedoraAtomicCI/KojiBuilds|Build options and ideas]]
 
* [[FedoraAtomicCI/upstream|Possibilities for upstream open-source project integration]]
There is an active effort to open source existing internal Red Hat
* [[Fedora requirements for CI and CD]] -- assembled from input from key members of [[Infrastructure]] and [[Release Engineering]]
tests to Fedora called Upstream First. There's a separate upstream
first repository with tests to be ported and a stats page tracking
the progress:
 
* [https://upstreamfirst.fedorainfracloud.org/ Upstream First]
* [[CI/Tests/stat|Progress Status]]
 
= More =
 
== Contact ==
 
If you have questions or would like to get involved:
 
* IRC channel: <code>#fedora-ci</code> on FreeNode
* Mailing list: <code>ci@lists.fedoraproject.org</code>
* Issues: [https://pagure.io/fedora-ci/AtomicCi pagure.io/fedora-ci/AtomicCi]
 
== Links ==


= Contact =
Here's a summary of useful links:


To get involved:
* [[CI/Standard_Test_Interface|Standard Test Interface]] ... definition of the process
* [[CI/Standard_Test_Roles|Standard Test Roles]] ... set of ansible roles
* [[CI/Tests|Tests]] ... executing and adding tests
* [https://upstreamfirst.fedorainfracloud.org/ Upstream First] ... tests to be ported to dist-git


* <code>#fedora-ci</code> on FreeNode
== FAQ ==
* Mailing list: ci@lists.fedoraproject.org
* If you don't know where to go: [https://pagure.io/fedora-ci/AtomicCi pagure.io/fedora-ci/AtomicCi]


[[Category:FedoraAtomicCi]]
* Where do I get the latest Atomic Host images for testing?

Revision as of 13:07, 19 February 2018

Why

Vision

There are hundreds of packages which make up the Operating System. Making sure that they all work together as a whole is not an easy task. This becomes even harder as the number of packages and their inter-dependencies grows. An extensive testing is required before a new version of the operating system is released to ensure it is stable enough. That is the past.

Imagine an Always Ready Operating System which consists of packages which are constantly kept in a good shape. Integrated and stable thanks to an extensive test coverage which is continuously executed upon changes in individual packages, in this way allowing to prepare a new release in much shorter time, or even in no time.

Imagine an operating system distribution which you could release at any moment. This is where we are heading. Here comes the CI, Continuous Integration, as an invaluable tool to ensure everything is working together as expected in every point of time.

Manifesto

Continuous integration aims to ensure broken changes are revealed as soon as possible and do not affect other developers, packagers, maintainers or users. The feedback that continuous integration provides is vital for fast paced agile delivery of software. Late testing, long after a change occurs, does not scale to the pace of Fedora. Learn the goals, terminology and rules for a working CI in the manifesto.

How

There are three main pieces of the puzzle to get this nicely working: A process which clearly defines how to discover and execute tests, a set of tools which help to efficiently implement the process and the tests themselves.

Process

Standard Test Interface

In order to clearly distinguish test from the CI system running it the Standard Test Interface was introduced. It clearly defines essential terms such as test, test subject, test suite, test framework, test result, test artifact, test system and describes what are their responsibilities and requirements.

This general approach gives a nice flexibility as it does not enforce any specific tools or frameworks to be used. Basically it only describes how tests are discovered and where the testing results should be stored to be processed by the automation.

Gating Updates

When a test fails, CI can prevent the broken change from affecting other packages. That gating happens in Bodhi.

Notifications

Fedora Notifications have been adjusted to notify by default every packager when any step of the CI pipeline fails on one of the package they maintain. So if you are a kernel maintainer and a commit made to the kernel dist-git repository fails to compose an OSTree, FMN will notify you of it.

Bodhi includes the CI results in its update page, just as it already includes tests results from taskotron.

Tools

Standard Test Roles

Standard Test Roles were implemented to enable both automation tools and developers in their local environments to easily execute tests. This set of ansible roles supports various frameworks and allows to execute tests against different test subjects (such as classic rpm package, docker container or Atomic Host).

Metadata

Standard Test Interface defines only a very simple metadata for selecting which tests should be run. For more complex scenarios Flexible Metadata Format can be used:

Pipeline

The testing Pipeline detects tests for enabled packages, executes the test coverage and gathers the results. Currently the pipeline is enabled for the Atomic Host packages only.

Pagure

Test results from the CI pipeline are displayed in Pagure web interface. See the commits page of respective package. Currently tests are scheduled for new commits in branches only. Support for pull request testing is planned in the near future.

Tests

The core of the CI success are reliable tests of a good quality, well selected, stable, organized and continuously maintained.

Test Types

In general it makes sense to store tests as close to the upstream as possible. So what are the appropriate test types recommended for testing the Always Ready Operating System?

  • Basic functionality tests
  • Integration tests

For unit tests it usually makes more sense to store them directly within the upstream project repository. However, in some cases it might be worth to fetch tests for Fedora CI from the upstream repository as well.

Test Code

Tests may be written all sorts of different ways, but have to be exposed and invoked in a standard way. Tests are enabled by including the tests/tests.yml file in the package dist-git repository as defined by the Standard Test Interface. Test code itself can be stored directly in the dist-git or fetched from another repository. Shared tests namespace can be used for storing test code relevant for multiple packages.

Test Execution

Executing a test written with the use of Standard Test Roles is as simple as running an ansible playbook ansible-playbook tests.yml. However, a set of environment variables needs to be set properly in order to execute the test against the desired test subject. The Tests wiki contains detailed instructions about running tests and adding new test coverage.

  • Tests ... How to run, write and wrap tests

Test Porting

There is an active effort to open source existing internal Red Hat tests to Fedora called Upstream First. There's a separate upstream first repository with tests to be ported and a stats page tracking the progress:

More

Contact

If you have questions or would like to get involved:

Links

Here's a summary of useful links:

FAQ

  • Where do I get the latest Atomic Host images for testing?
Retrieved from "https://fedoraproject.org/w/index.php?title=CI&oldid=511279"