Standard Dicsovery, Packaging, Invocation of Integration Tests
This file is incomplete.
Reason
Many Fedora packages have unit tests. These tests are typically run during a
%check RPM build step. These tests run in a BuildRoot.
On the other hand, integration testing happens against a composed system. system. Several upstream projects have integration tests, but these have typically not been used or packaged in Fedora. Both Fedora QA and the Atomic Host team would like to create more integration tests, stored in dist-git.
Summary
What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.
The integration tests are packaged and delivered through Fedora as
%{name}-tests subpackages of the package they are associated with.
This change requires no changes to Fedora infrastructure itself. It is limited to changes in dist-git repos. However certain key infrastructure changes could mitigate usability or side-effects of this change.
Feedback is more than welcome. There's a discussion tab above.
Packaging
Each dist-git repo that has integration tests should package those tests in a
%{name}-tests subpackage. This is similar to the
%{name}-debuginfo or %{name}-docs subpackages we have
today.
The spec file for a dist-git repo may include upstream integration tests into
its %{name}-tests subpackage. The spec file may also include tests
directly from files in the dist-git repo itself.
Staging
The %{name}-test subpackage should depend on all other packages that the
test needs to run.
Some integration tests may choose to test in-situ, on the system on which the %{name}-tests
is installed. In these cases the %{name}-tests subpackage should depend on
the %{name} or other subpackages that are being tested.
More rigourous integration tests test an integrated system from the outside. It is the
responsibility of the %{name}-tests subpackages to provision virtual
machines or containers necessary to do such testing. In almost all cases this will happen
by way of a provisioning framework such as Avocado, Ansible, Module Testing Framework,
linch-pin, etc.
Invocation
TODO: Expand this section.
* Any executable file directly under /usr/tests/ is a test suite to be executed * Zero exit code from the executable indicates testsuite success, a non-zero exit code indicates test suite failure * The stdout/stderr of a test suite is the test log * Additional output artifacts are placed in /run/tests/ * The testsuite is invoked as root, and may drop privileges as desired
Discovery
In a dist-git repo the list of integration tests that should be invoked on a given package
are placed in a file called tests in that dist-git repo. The format of this
file has not yet been defined, but a simple text file similar to sources listing the
package names that include integration tests to use when validating the dist-git repo.
Examples
TODO: Build out this section
Notes
* ... * ...