|
|
| (One intermediate revision by one other user not shown) |
| Line 1: |
Line 1: |
| = Why =
| | #REDIRECT [[CI]] |
| | |
| == Vision ==
| |
| | |
| There are hundreds of packages which make up the Operating System.
| |
| Making sure that they all work together as a whole is not an easy
| |
| task. This becomes even harder as the number of packages and their
| |
| inter-dependencies grows. An extensive testing is required before
| |
| a new version of the operating system is released to ensure it is
| |
| stable enough. That is the past.
| |
| | |
| Imagine an '''Always Ready Operating System''' which consists of
| |
| packages which are constantly kept in a good shape. Integrated
| |
| and stable thanks to an extensive test coverage which is
| |
| continuously executed upon changes in individual packages, in this
| |
| way allowing to prepare a new release in much shorter time, or
| |
| even in no time.
| |
| | |
| Imagine an operating system distribution which you could release
| |
| at any moment. This is where we are heading. Here comes the CI,
| |
| Continuous Integration, as an invaluable tool to ensure everything
| |
| is working together as expected in every point of time.
| |
| | |
| == Manifesto ==
| |
| | |
| Continuous integration aims to ensure broken changes are revealed
| |
| as soon as possible and do not affect other developers, packagers,
| |
| maintainers or users. The feedback that continuous integration
| |
| provides is vital for fast paced agile delivery of software. Late
| |
| testing, long after a change occurs, does not scale to the pace of
| |
| Fedora. Learn the goals, terminology and rules for a working CI
| |
| in the manifesto.
| |
| | |
| * [[CI/Manifesto|Continuous Integration Manifesto]]
| |
| | |
| = How =
| |
| | |
| There are three main pieces of the puzzle to get this nicely
| |
| working: A process which clearly defines how to discover and
| |
| execute tests, a set of tools which help to efficiently implement
| |
| the process and the tests themselves.
| |
| | |
| == Process ==
| |
| | |
| === Standard Test Interface ===
| |
| | |
| In order to clearly distinguish test from the CI system running it
| |
| the '''Standard Test Interface''' was introduced. It clearly
| |
| defines essential terms such as test, test subject, test suite,
| |
| test framework, test result, test artifact, test system and
| |
| describes what are their responsibilities and requirements.
| |
| | |
| This general approach gives a nice flexibility as it does not
| |
| enforce any specific tools or frameworks to be used. Basically it
| |
| only describes how tests are discovered and where the testing
| |
| results should be stored to be processed by the automation.
| |
| | |
| * [[CI/Standard_Test_Interface|Standard Test Interface]]
| |
| | |
| === Gating Updates ===
| |
| | |
| When a test fails, CI can prevent the broken change from affecting
| |
| other packages. That gating happens in Bodhi.
| |
| | |
| * [[CI/gating_updates|Gating Updates]]
| |
| * [[FedoraAtomicCI/gating|Proposals and options]]
| |
| | |
| === Notifications ===
| |
| | |
| [https://apps.fedoraproject.org/notifications Fedora Notifications]
| |
| have been adjusted to notify by default every packager when any
| |
| step of the CI pipeline fails on one of the package they maintain.
| |
| So if you are a kernel maintainer and a commit made to the kernel
| |
| dist-git repository fails to compose an OSTree, FMN will notify
| |
| you of it.
| |
| | |
| Bodhi includes the CI results in its update page, just as it
| |
| already includes tests results from taskotron.
| |
| | |
| == Tools ==
| |
| | |
| === Standard Test Roles ===
| |
| | |
| '''Standard Test Roles''' were implemented to enable both
| |
| automation tools and developers in their local environments to
| |
| easily execute tests. This set of ansible roles supports various
| |
| frameworks and allows to execute tests against different test
| |
| subjects (such as classic rpm package, docker container or Atomic
| |
| Host).
| |
| | |
| * [[CI/Standard_Test_Roles|Standard Test Roles]]
| |
| * https://pagure.io/standard-test-roles/
| |
| | |
| === Metadata ===
| |
| | |
| Standard Test Interface defines only a very simple metadata for
| |
| selecting which tests should be run. For more complex scenarios
| |
| '''Flexible Metadata Format''' can be used:
| |
| | |
| * [[Flexible_Metadata_Format|Flexible Metadata Format]]
| |
| * https://github.com/psss/fmf
| |
| | |
| === Pipeline ===
| |
| | |
| The testing '''Pipeline''' detects tests for enabled
| |
| packages, executes the test coverage and gathers the results.
| |
| Currently the pipeline is enabled for the Atomic Host packages
| |
| only.
| |
| | |
| * [https://github.com/CentOS-PaaS-SIG/ci-pipeline/blob/master/README.md#ci-pipeline-architecture-and-design CI Pipeline Architecture and Design]
| |
| * [[FedoraAtomicCI/pipeline|Detailed pipeline description]]
| |
| * [[FedoraAtomicCI/KojiBuilds|Build options and ideas]]
| |
| * [[FedoraAtomicCI/upstream|Upstream open-source project integration]]
| |
| * [[Fedora requirements for CI and CD]]
| |
| | |
| === Pagure ===
| |
| | |
| Test results from the CI pipeline are displayed in '''Pagure'''
| |
| web interface. See the commits page of respective package.
| |
| Currently tests are scheduled for new commits in branches only.
| |
| Support for pull request testing is planned in the near future.
| |
| | |
| * [https://src.fedoraproject.org/ Package Sources]
| |
| * [https://pagure.io/ Pagure]
| |
| | |
| == Tests ==
| |
| | |
| The core of the CI success are reliable tests of a good quality,
| |
| well selected, stable, organized and continuously maintained.
| |
| | |
| === Test Types ===
| |
| | |
| In general it makes sense to store tests as close to the upstream
| |
| as possible. So what are the appropriate test types recommended
| |
| for testing the Always Ready Operating System?
| |
| | |
| * Basic functionality tests
| |
| * Integration tests
| |
| | |
| For unit tests it usually makes more sense to store them directly
| |
| within the upstream project repository. However, in some cases it
| |
| might be worth to fetch tests for Fedora CI from the upstream
| |
| repository as well.
| |
| | |
| === Test Code ===
| |
| | |
| Tests may be written all sorts of different ways, but have to be
| |
| exposed and invoked in a standard way. Tests are enabled by
| |
| including the <code>tests/tests.yml</code> file in the package
| |
| dist-git repository as defined by the Standard Test Interface.
| |
| Test code itself can be stored directly in the dist-git or fetched
| |
| from another repository. Shared tests namespace can be used for
| |
| storing test code relevant for multiple packages.
| |
| | |
| * [[CI/Tests|Tests]] ... How to run, write and wrap tests
| |
| * [[CI/Share_Test_Code|Share Test Code]] ... Shared tests namespace
| |
| | |
| === Test Execution ===
| |
| | |
| Executing a test written with the use of Standard Test Roles is as
| |
| simple as running an ansible playbook <code>ansible-playbook
| |
| tests.yml</code>. However, a set of environment variables needs to
| |
| be set properly in order to execute the test against the desired
| |
| test subject. The '''Tests''' wiki contains detailed instructions
| |
| about running tests and adding new test coverage.
| |
| | |
| * [[CI/Tests|Tests]] ... How to run, write and wrap tests
| |
| | |
| === Test Porting ===
| |
| | |
| There is an active effort to open source existing internal Red Hat
| |
| tests to Fedora called Upstream First. There's a separate upstream
| |
| first repository with tests to be ported and a stats page tracking
| |
| the progress:
| |
| | |
| * [https://upstreamfirst.fedorainfracloud.org/ Upstream First]
| |
| * [[CI/Tests/stat|Progress Status]]
| |
| | |
| = More =
| |
| | |
| == Contact ==
| |
| | |
| If you have questions or would like to get involved:
| |
| | |
| * IRC channel: <code>#fedora-ci</code> on FreeNode
| |
| * Mailing list: <code>ci@lists.fedoraproject.org</code>
| |
| * Issues: [https://pagure.io/fedora-ci/AtomicCi pagure.io/fedora-ci/AtomicCi]
| |
| | |
| == Links ==
| |
| | |
| Here's a summary of useful links:
| |
| | |
| * [[CI/Standard_Test_Interface|Standard Test Interface]] ... definition of the process
| |
| * [[CI/Standard_Test_Roles|Standard Test Roles]] ... set of ansible roles
| |
| * [[CI/Tests|Tests]] ... executing and adding tests
| |
| * [https://upstreamfirst.fedorainfracloud.org/ Upstream First] ... tests to be ported to dist-git
| |
| | |
| == FAQ ==
| |
| | |
| * Where do I get the latest Atomic Host images for testing?
| |
