Why
Vision
There are hundreds of packages which make up the Operating System. Making sure that they all work together as a whole is not an easy task. This becomes even harder as the number of packages and their inter-dependencies grows. An extensive testing is required before a new version of the operating system is released to ensure it is stable enough. That is the past.
Imagine an Always Ready Operating System which consists of packages which are constantly kept in a good shape. Integrated and stable thanks to an extensive test coverage which is continuously executed upon changes in individual packages, in this way allowing to prepare a new release in much shorter time, or even in no time.
Imagine an operating system distribution which you could release at any moment. This is where we are heading. Here comes the CI, Continuous Integration, as an invaluable tool to ensure everything is working together as expected in every point of time.
Manifesto
Continuous integration aims to ensure broken changes are revealed as soon as possible and do not affect other developers, packagers, maintainers or users. The feedback that continuous integration provides is vital for fast paced agile delivery of software. Late testing, long after a change occurs, does not scale to the pace of Fedora. Learn about the steps that are crucial for a working Continuous Integration in the CI Manifesto.
How
There are three main pieces of the puzzle to get this nicely working: A process which clearly defines how to discover and execute tests, a set of tools which help to efficiently implement the process and the tests themselves.
Process
In order to clearly distinguish test from the CI system running it the Standard Test Interface was introduced. It clearly defines essential terms such as test, test subject, test suite, test framework, test result, test artifact, test system and describes what are their responsibilities and requirements.
This general approach gives a nice flexibility as it does not enforce any specific tools or frameworks to be used. Basically it only describes how tests are discovered and where the testing results should be stored to be processed by the automation.
Tools
Standard Test Roles were implemented to enable both automation tools and developers in their local environments to easily execute tests. This set of ansible roles supports various frameworks and allows to execute tests against different test subjects (such as classic rpm package, docker container or Atomic Host).
The testing Pipeline detects tests for enabled packages, executes the test coverage and gathers the results. Currently tests are scheduled for new commits, support for pull request testing is on the way. Results are displayed in Pagure web interface.