CI/Tests

Tests are stored in package or module git repositories along with the packages and modules that they test. The tests are updated together with the software.

Setting up

You can use the fedpkg tool to checkout a package git repository. If a tests/ subdirectory exists, then the repository contains tests. The files ending in .yml in the tests/ subdirectory each represent a test or a part of a test.

The tests are wrapped or written as Ansible playbooks and invoked according to this specification. To invoke the tests you need the following dependencies (as described by the specification) installed on a modern Fedora system:

$ sudo dnf install ansible python2-dnf libselinux-python standard-test-roles

For the following documentation we'll checkout the gzip tests:

$ fedpkg clone gzip
$ cd gzip/tests/
$ sudo -s

Work in progress
The upstreaming of tests is currently ongoing. Use the following command until the end of Summer 2017.

$ git clone https://upstreamfirst.fedorainfracloud.org/gzip.git
$ cd gzip/
$ sudo -s

Although some playbooks may function without sudo, tests are always invoked as root. The test itself may set up users and/or drop permissions if a part of that test. But in general be sure to be root when invoking tests.

Running tests

You can always invoke the tests locally. Many tests modify or change the system they are run against, so take that into account when looking at how to invoke tests. The following tests invoke tests against the same system that the package git repository is checked out on. Below there are further options for invoking tests against another fully formed and integrated system, such as an Atomic Host or container image test subject.

There may be more than one test present in a package git repository, but the file tests.yml is the main entry point. To run it use the following command:

# ansible-playbook tests.yml

You can find output artifacts of the tests in an artifacts/ or specify a specific directory like this:

# ansible-playbook -e artifacts=/tmp/output tests.yml

When run by a CI System the tests are invoked according to a specification. Look there for more details and standard invocation variables.

Testing specific RPMs

When you run the tests as above, the tests assume that the system to be tested is the same as the system invoking the tests. In particular, the test assumes that the thing to be tested is already installed.

A test subject is what we call the thing to be tested. RPMs are a particular kind of test subject. To turn a test subject into a launched, installed system to be tested, we use Ansible dynamic inventory. Lets invoke the tests with an inventory and a specific version of gzip:

# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /etc/ansible/inventory)
# curl -o gzip.rpm https://kojipkgs.fedoraproject.org//packages/gzip/1.8/2.fc26/x86_64/gzip-1.8-2.fc26.x86_64.rpm
# export TEST_SUBJECTS=gzip.rpm
# ansible-playbook tests.yml

You'll notice that the RPM is installed into the testable system before invoking the tests. Some tests contain their own inventory, that is their own instructions for turning a test subject into one or more testable systems. But in this case we use the default standard-test-roles inventory in /etc/ansible/inventory to do this.

Testing an Atomic Host

The former example may seem a bit contrived, but the concept of a test subject starts to make more sense when you want to test a fully formed and integrated deliverable, such as Atomic Host. The test subject again represents the thing to be tested. The test subject in this case is a QCow2 image. To turn a test subject into a launched system ready to be tested, we use Ansible dynamic inventory.

# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /etc/ansible/inventory)
# curl -Lo atomic.qcow2 https://ftp-stud.hs-esslingen.de/pub/Mirrors/alt.fedoraproject.org/atomic/stable/Fedora-Atomic-26-20170707.1/CloudImages/x86_64/images/Fedora-Atomic-26-20170707.1.x86_64.qcow2
# export TEST_SUBJECTS=atomic.qcow2
# ansible-playbook tests.yml

If you watch closely you'll see that the Atomic Host image is booted, and the tests run against the launched image. Not all tests are able to function in the somewhat different environment of Atomic Host, in fact for certain cases the software to be tested may not be included in the Atomic Host test subject. But most of the tests in core packages should work here.

Some tests contain their own inventory, that is their own instructions for turning a test subject into one or more testable systems. But in this case we use the default standard-test-roles inventory to do this.

Testing a Container Image

TODO