|
|
| (One intermediate revision by one other user not shown) |
| Line 1: |
Line 1: |
| = Why =
| | Page moved to: https://docs.fedoraproject.org/en-US/ci/ |
| | |
| == Vision ==
| |
| | |
| There are hundreds of packages which make up the Operating System.
| |
| Making sure that they all work together as a whole is not an easy
| |
| task. This becomes even harder as the number of packages and their
| |
| inter-dependencies grows. An extensive testing is required before
| |
| a new version of the operating system is released to ensure it is
| |
| stable enough. That is the past.
| |
| | |
| Imagine an '''Always Ready Operating System''' which consists of
| |
| packages which are constantly kept in a good shape. Integrated
| |
| and stable thanks to an extensive test coverage which is
| |
| continuously executed upon changes in individual packages, in this
| |
| way allowing to prepare a new release in much shorter time, or
| |
| even in no time.
| |
| | |
| Imagine an operating system distribution which you could release
| |
| at any moment. This is where we are heading. Here comes the CI,
| |
| Continuous Integration, as an invaluable tool to ensure everything
| |
| is working together as expected in every point of time.
| |
| | |
| == Manifesto ==
| |
| | |
| Continuous integration aims to ensure broken changes are revealed
| |
| as soon as possible and do not affect other developers, packagers,
| |
| maintainers or users. The feedback that continuous integration
| |
| provides is vital for fast paced agile delivery of software. Late
| |
| testing, long after a change occurs, does not scale to the pace of
| |
| Fedora. Learn the goals, terminology and rules for a working CI
| |
| in the manifesto.
| |
| | |
| * [[CI/Manifesto|Continuous Integration Manifesto]]
| |
| | |
| = How =
| |
| | |
| There are three main pieces of the puzzle to get this nicely
| |
| working: A process which clearly defines how to discover and
| |
| execute tests, a set of tools which help to efficiently implement
| |
| the process and the tests themselves.
| |
| | |
| == Process ==
| |
| | |
| === Standard Test Interface ===
| |
| | |
| In order to clearly distinguish test from the CI system running it
| |
| the '''Standard Test Interface''' was introduced. It clearly
| |
| defines essential terms such as test, test subject, test suite,
| |
| test framework, test result, test artifact, test system and
| |
| describes what are their responsibilities and requirements.
| |
| | |
| This general approach gives a nice flexibility as it does not
| |
| enforce any specific tools or frameworks to be used. Basically it
| |
| only describes how tests are discovered and where the testing
| |
| results should be stored to be processed by the automation.
| |
| | |
| * [[CI/Standard_Test_Interface|Standard Test Interface]]
| |
| | |
| === Gating ===
| |
| | |
| When a test fails, CI can prevent the broken change from affecting
| |
| other packages. That gating happens in Bodhi. Greewave is used
| |
| together with ResultsDB and WaiverDB to make the gating decision.
| |
| | |
| * [[CI/Gating|Gating]] ... how to enable gating, waiving instructions
| |
| * [https://bodhi.fedoraproject.org/ Bodhi] ... Fedora updates System
| |
| * [https://pagure.io/greenwave Greewave] ... service to evaluate gating policies based on test results
| |
| * [https://pagure.io/taskotron/resultsdb ResultsDB] ... results store engine
| |
| * [https://pagure.io/waiverdb WaiverDB] ... service for recording waivers against test results
| |
| | |
| === Notifications ===
| |
| | |
| [https://apps.fedoraproject.org/notifications Fedora Notifications]
| |
| have been adjusted to notify by default every packager when any
| |
| step of the CI pipeline fails on one of the package they maintain.
| |
| So if you are a kernel maintainer and a commit made to the kernel
| |
| dist-git repository fails to compose an OSTree, FMN will notify
| |
| you of it.
| |
| | |
| Bodhi includes the CI results in its update page, just as it
| |
| already includes tests results from taskotron.
| |
| | |
| === Messages ===
| |
| | |
| Various tools involved in the CI automation are sending updates
| |
| about the progress of the testing to the message bus. Consistent
| |
| format of the CI messages allows to build simpler services which
| |
| will provide useful features for everyone.
| |
| | |
| * [https://pagure.io/fedora-ci/messages Fedora CI Messages] ... CI messages specification
| |
| * [https://apps.fedoraproject.org/datagrepper/ Datagrepper] ... search sent messages (e.g. by [https://apps.fedoraproject.org/datagrepper/raw?topic=org.centos.prod.ci.pipeline.allpackages-pr.complete topic])
| |
| * [https://github.com/CentOS-PaaS-SIG/upstream-fedora-pipeline#fedmsg-bus fedmsg bus] ... messages sent by the pipeline
| |
| * [https://fedora-fedmsg.readthedocs.io/en/latest/topics.html#ci topics] ... list of fedmsg topics related to CI
| |
| | |
| == Tools ==
| |
| | |
| === Standard Test Roles ===
| |
| | |
| '''Standard Test Roles''' were implemented to enable both
| |
| automation tools and developers in their local environments to
| |
| easily execute tests. This set of ansible roles supports various
| |
| frameworks and allows to execute tests against different test
| |
| subjects (such as classic rpm package, docker container or Atomic
| |
| Host).
| |
| | |
| * [[CI/Standard_Test_Roles|Standard Test Roles]]
| |
| * https://pagure.io/standard-test-roles/
| |
| | |
| === Frameworks ===
| |
| | |
| As Standard Test Interface does not define test framework which
| |
| should be used for testing it is possible to choose the most
| |
| suitable framework for your project. Here are some examples:
| |
| | |
| * [https://github.com/beakerlib/beakerlib/wiki/man BeakerLib] and it's [https://pagure.io/beakerlib-libraries Libraries]
| |
| * [https://github.com/avocado-framework/avocado Avocado]
| |
| | |
| === Metadata ===
| |
| | |
| Standard Test Interface defines only a very simple metadata for
| |
| selecting which tests should be run. For more complex scenarios
| |
| '''Flexible Metadata Format''' can be used:
| |
| | |
| * Fedora CI [https://pagure.io/fedora-ci/metadata Metadata Specification]
| |
| * [[Flexible_Metadata_Format|Flexible Metadata Format]]
| |
| * https://github.com/psss/fmf
| |
| | |
| === Pipeline ===
| |
| | |
| The testing '''Pipeline''' detects tests for enabled
| |
| packages, executes the test coverage and gathers the results.
| |
| | |
| * [[CI/Pipeline|Pipeline]]
| |
| * [https://github.com/CentOS-PaaS-SIG/ci-pipeline/blob/master/README.md#ci-pipeline-architecture-and-design CI Pipeline Architecture and Design]
| |
| | |
| === Pagure ===
| |
| | |
| Test results from the CI pipeline are displayed in '''Pagure'''
| |
| web interface. See the commits page and pull request pages of
| |
| respective package to see the results. Pagure integration with
| |
| COPR provides auto-rebuilding and pull request/commit flagging on
| |
| new changes.
| |
| | |
| * [https://src.fedoraproject.org/ Package Sources]
| |
| * [https://pagure.io/ Pagure]
| |
| * [https://docs.pagure.org/copr.copr/user_documentation.html#pagure-integration COPR Integration]
| |
| | |
| == Tests ==
| |
| | |
| The core of the CI success are reliable tests of a good quality,
| |
| well selected, stable, organized and continuously maintained.
| |
| | |
| === Test Types ===
| |
| | |
| In general it makes sense to store tests as close to the upstream
| |
| as possible. So what are the appropriate test types recommended
| |
| for testing the Always Ready Operating System?
| |
| | |
| * Basic functionality tests
| |
| * Integration tests
| |
| | |
| For unit tests it usually makes more sense to store them directly
| |
| within the upstream project repository. However, in some cases it
| |
| might be worth to fetch tests for Fedora CI from the upstream
| |
| repository as well.
| |
| | |
| === Test Code ===
| |
| | |
| Tests may be written all sorts of different ways, but have to be
| |
| exposed and invoked in a standard way. Tests are enabled by
| |
| including the <code>tests/tests.yml</code> file in the package
| |
| dist-git repository as defined by the Standard Test Interface.
| |
| Test code itself can be stored directly in the dist-git or fetched
| |
| from another repository. Shared tests namespace can be used for
| |
| storing test code relevant for multiple packages.
| |
| | |
| * [[CI/Quick_Start_Guide|Quick Start Guide]] ... quick intro for the impatient
| |
| * [[CI/Tests|Tests]] ... how to run, write and wrap tests
| |
| * [[CI/Share_Test_Code|Share Test Code]] ... shared tests namespace
| |
| * [[CI/Pull_Requests|Pull Requests]] ... how to create pull requests
| |
| * [[CI/Examples|Examples]] ... real-life success stories of enabled tests
| |
| | |
| === Test Execution ===
| |
| | |
| Executing a test written with the use of Standard Test Roles is as
| |
| simple as running an ansible playbook <code>ansible-playbook
| |
| tests.yml</code>. However, a set of environment variables needs to
| |
| be set properly in order to execute the test against the desired
| |
| test subject. The '''Tests''' wiki contains detailed instructions
| |
| about running tests and adding new test coverage.
| |
| | |
| * [[CI/Tests|Tests]] ... How to run, write and wrap tests
| |
| | |
| === Test Coverage ===
| |
| | |
| There are statistics pages to track how many Fedora dist-git repositories have tests in the Standard Test Format. These pages are updated automatically.
| |
| | |
| * [[CI/Tests/stat|Basic Operating System Status]]
| |
| | |
| === Test Porting ===
| |
| | |
| There is an active effort to open source existing internal Red Hat
| |
| tests to Fedora called Upstream First. Tests being migrated can be
| |
| found in the upstream first repository:
| |
| | |
| * [https://upstreamfirst.fedorainfracloud.org/ Upstream First]
| |
| | |
| === Shared Responsibility ===
| |
| | |
| The ownership and maintenance of tests should be shared between QE & Devel. For tests that reside in the rpm namespace, QE can use pull requests to create/update tests. Likewise in the tests namespace, both QE and Devel will have commit rights, both QE and Devel should review and sign-off with each commit.
| |
| | |
| = More =
| |
| | |
| == Contact ==
| |
| | |
| If you have questions or would like to get involved:
| |
| | |
| * IRC channel: <code>#fedora-ci</code> on FreeNode
| |
| * Mailing list: [mailto:ci@lists.fedoraproject.org ci@lists.fedoraproject.org] ([https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/ archive])
| |
| * Issues: [https://pagure.io/fedora-ci/general General], [https://pagure.io/fedora-ci/AtomicCi Atomic CI]
| |
| | |
| == Links ==
| |
| | |
| Here's a summary of useful links:
| |
| | |
| * [[CI/Standard_Test_Interface|Standard Test Interface]] ... definition of the process
| |
| * [[CI/Standard_Test_Roles|Standard Test Roles]] ... set of ansible roles
| |
| * [https://pagure.io/fedora-ci/workshop Workshop] ... workshop materials
| |
| * [[CI/Tests|Tests]] ... executing and adding tests
| |
| * [https://upstreamfirst.fedorainfracloud.org/ Upstream First] ... tests to be ported to dist-git
| |
| | |
| == FAQ ==
| |
| | |
| * Where do I get the latest Atomic Host images for testing?
| |
