Changes/InvokingTests

Standard Dicsovery, Packaging, Invocation of Integration Tests

This is incomplete
This file is incomplete.

This is a proposal
Feedback is more than welcome. There's a discussion tab above.

Summary

What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.

Owner

• Name: Stef Walter
• Email: stefw@fedoraproject.org
• Name: Martin Pitt
• Email: martin@piware.de

Terminology

• Test Subject: The items that are to be tested. Typically this is a set of RPMs (updating a package to a new version), a container, or a distribution install ISO/VM image.
• Test: A callable/runnable piece of code and corresponding test data and mocks which exercises and evaluates a subject.
• Test Suite: The collection of all tests that apply to a subject. It's common to split the testing of different aspects into different tests for easier result evaluation, code maintenance, or parallelization.
• Test Result: A boolean output of a test which decides whether the test passed. This is being used for automatic result processing (gating in Continuous Integration).
• Test Artifact: Any additional output of the test such as the stdout/err output, log files, screen shots, core dumps, or TAP/Junit/subunit streams. These are mostly for human consumption (evaluating test failures by developers), but in case of machine readable files like JUnit they can also be used in result browsers for presenting test results.

Detailed Description

Many Fedora packages have unit tests. These tests are typically run during a %check RPM build step. These tests run in the build root in which the package is built and may involve the BuildRequire dependencies.

On the other hand, integration testing happens against a composed system. Several upstream projects have integration tests, but these integration tests have typically not been used or packaged in Fedora. Both Fedora QA and the Atomic Host team would like to create more integration tests, stored in dist-git, and use a CI system to run them on Fedora packages.

What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.

It is important that the CI system which runs the test is independent from the layout of the test themselves.

Goals:

• Implementation details of the integration test suite or its framework should not leak into or rely on the CI system invoking the test.
• CI system implementation details should not leak into the test suite or its metadata.
• It should be possible to change the CI system that runs a test suite.
• It should be possible for a dist-git packager to run a test suite locally.

Requirements:

• Declare and install test suite dependencies cleanly
• Standard way to locally invoke a test process
• Standard location to place test subjects, results, and artifacts

What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.

The integration tests are packaged and delivered through Fedora as %{name}-tests subpackages of the package they are associated with.

Packaging

Each dist-git repo that has integration tests should package those tests in a %{name}-tests subpackage. This is similar to the %{name}-debuginfo or %{name}-docs subpackages we have today.

The spec file for a dist-git repo may include upstream integration tests into its %{name}-tests subpackage. The spec file may also include tests directly from files in tests/ subdirectory of the dist-git repo itself.

Invocation

To invoke the test suite, the subpackage is installed. Each test of the suite installs an executable in the path /usr/tests/.

To invoke the test suite, one would:

1. Create a temporary directory, called $TESTDIR here.
2. Place the subjects being tested in $TESTDIR/subjects/
3. Execute all executable files in /usr/tests/ one at a time.
   1. The test is invoked with a working directory of $TESTDIR
   2. The test is invoked as root, and may drop privileges as desired
   3. Treat the stdout/stderr of each tests process as the test log. This is a standard artifact and written to $TESTDIR/artifacts/testname.log.
   4. Examine the exit code of each test process. Zero exit code is success, non-zero is failure.
4. Tests can put any additional artifacts like screenshots into $TESTDIR/artifacts/.

This ensures that tests can be run on a production system without accidentally clobbering permanent directories, don't require root privileges (simplifies test development), and that CI systems have one unique place from where to collect artifacts. It also avoids collecting temporary files such as downloaded container or VM images as artifacts, as these would usually get stored for a longer time period.

These steps would usually be done through a standard test driver tool (particularly for sensible stdout/err teeing and log capturing), but its usage is not mandatory for developing and calling tests manually.

Multiple subpackages may be installed as long as their dependencies do not conflict.

Staging

The %{name}-test subpackage should Requires: all other packages that the testsuite executable needs in order to run. This includes libraries or frameworks, or subsystems like libvirt.

Some integration tests may choose to test in-situ, on the system on which the test suite is installed. In these cases the %{name}-tests subpackage should directly depend on the package being tested.

More rigourous integration tests test an integrated system from the outside. It is the responsibility of the %{name}-tests subpackages to provision virtual machines or containers necessary to do such testing. In almost all cases this will happen by way of a provisioning framework such as Avocado, Ansible, Module Testing Framework, linch-pin, etc.

Discovery

A check file in a dist-git repo should list the various names of packages that should be installed in order to run the integration tests for that package. If this file is empty then the list of packages will default to %{name}-tests.

The format of this file has not yet been defined, but a simple text file similar to sources listing package N[VR]'s may suffice.

Scope

This change requires no changes to Fedora infrastructure itself. It is limited to changes in dist-git repos. However certain key infrastructure changes could mitigate usability or side-effects of this change.

Benefit to Fedora

Developers benefit by having a consistent target for how to describe tests, while also being able to execute them locally while debugging issues or iterating on tests.

By packaging, staging and invoking tests consistently in Fedora we create an eco-system for the tests that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.

User Experience

Users benefit by having tests that they can reproduce on their own systems. They can install the similar to how they consume %{name}-doc or %{name}-debuginfo subpackages today.

We may choose to avoid having such packages available in the standard repositories. We may choose to only have them in updates-testing or an arrangement similar to debuginfo. These choices will require some markup and/or change to infrastructure.

Upgrade/compatibility impact

Although there may already be subpackages that are named %{name}-tests this is merely a convention, and such packages will not affect the behavior of this proposal.

Examples

TODO: Build out this section

• Add simple downstream integration test for gzip (taken verbatim from Ubuntu package)
• run-fedtest prototype script to run all /usr/tests/*, pass arbitrary subjects to them, and report/capture the results/logs

With this you can install test RPM from above gzip repo:

 $ sudo rpm -i results_gzip/1.8/2.fc27/gzip-tests-1.8-2.fc25.x86_64.rpm

and run the gzip tests on the already installed package (as user) with

 $ ~/run-fedtest
 Subjects/artifacts directory: /tmp/fedtest.vsR
 -----------------------------------------
 Running /usr/tests/test-simple
 -----------------------------------------
 ++ ls 'subjects/*.rpm'
 + echo Bla
 + cp bla.file bla.file.orig
 + gzip bla.file
 + gunzip bla.file.gz
 + cmp bla.file bla.file.orig
 + rm bla.file bla.file.orig
 PASS: /usr/tests/test-simple

 $ ls -l /tmp/fedtest.vsR/artifacts/
 -rw-r--r-- 1 martin martin 156 Mar 28 16:49 test-simple.log

or run them as root (as officially specified) with a subject (locally built gzip RPM):

 $ sudo ~/run-fedtest results_gzip/1.8/2.fc27/gzip-1.8-2.fc25.x86_64.rpm
 Installing subject results_gzip/1.8/2.fc27/gzip-1.8-2.fc25.x86_64.rpm
 Subjects/artifacts directory: /tmp/fedtest.Cck
 -----------------------------------------
 Running /usr/tests/test-simple
 -----------------------------------------
 ++ ls subjects/gzip-1.8-2.fc25.x86_64.rpm
 + '[' -w / ']'
 + rpm --verbose --force -U subjects/gzip-1.8-2.fc25.x86_64.rpm
 Preparing packages...
 gzip-1.8-2.fc25.x86_64
 + echo Bla
 + cp bla.file bla.file.orig
 + gzip bla.file
 + gunzip bla.file.gz
 + cmp bla.file bla.file.orig
 + rm bla.file bla.file.orig
 PASS: /usr/tests/test-simple

Notes

• ...
• ...