CI/Tests

Enabling Tests

Tests may be written in different ways, but are exposed and invoked in a standard way as defined by the Standard Test Interface directly in the package git repository. To start working on tests you can clone a package repo directly:

git clone https://src.fedoraproject.org/rpms/qrencode.git

You can also use the fedpkg to clone the repo. See the Package Maintenance Guide for more info about the tool:

fedpkg clone -a qrencode

Tests are enabled by including the tests.yml file under the tests directory:

cd qrencode/tests
cat tests.yml

Tests are wrapped or written as Ansible playbooks. Here is an example of a simple playbok which enables a single smoke test of the qrencode package:

- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    - container
    - atomic
    tests:
    - smoke
    required_packages:
    - qrencode
    - file

Let's now briefly look at the playbook to see which variables are defined in order to enable the smoke test:

• role — this test uses role standard-test-beakerlib from Standard Test Roles to run a BeakerLib test
• tags — all three test subjects (classic rpm, docker container and atomic host) are relevant for this test
• tests — list of tests to be executed (here we have just a single smoke test)
• required_packages — list of rpm packages required for test execution

There may by multiple files ending in .yml in the tests/ subdirectory and each of them can represent a test or a part of a test. All of them need to be included in the main tests.yaml file. Let's have a look at the gzip example:

> fedpkg clone -a gzip
Cloning into 'gzip'...

> cd gzip/tests/
> ls
test-simple  test_simple.yml  tests.yml

> cat tests.yml
- include: test_simple.yml

Executing Tests

Before running tests make sure you have the following dependencies installed on your system:

dnf install ansible python2-dnf libselinux-python standard-test-roles

Although some playbooks may function without sudo, tests are always invoked as root. The test itself may set up users and/or drop permissions if a part of that test. But in general be sure to be root when invoking tests.

Tests may modify or destroy your environment
It's recommended to use a virtual machine for testing to prevent any unwated changes performed by the test to your system.

Running a test directly on the current system is easy:

ansible-playbook tests.yml

To only run tests that are suited for classic systems installed by yum or dnf use the --tags argument:

ansible-playbook --tags=classic tests.yml

See Standard Test Roles documentation for detailed instructions how to run tests for a specific Rpm Package, Docker Container or Atomic Host.

Adding Tests

Test code itself can be stored directly in the dist-git (recommended as default) or fetched from another repository hosted in the Fedora infrastructure. Test Namespace can be used for storing test code relevant for multiple packages. The simplest way to add a new test is by using one of the existing Standard Test Roles which take care of many implementatin details. If you want to create a custom test follow instructions below.

Once you've identified a dist-git repository you will be adding new tests to (above), you can start to write a new Ansible test. Create an Ansible playbook with a new name. Make sure the extension is .yml. Lets place the following example in test_pid_1.yml file.

---
- hosts: localhost
  vars:
  - artifacts: ./artifacts
  tags:
  - atomic
  - classic
  - container
  tasks:
  - name: Test block
    block:
      - name: Test that /proc/1 exists
        shell: ls /proc > /tmp/test.log && grep -qw 1 /tmp/test.log

    always:
      - name: Pull out the artifacts
        fetch:
          dest: "{{ artifacts }}/"
          src: "/tmp/test.log"
          flat: yes

All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We ensure this directory exists in the test.

By use of tags we note what kind of systems this test is suitable to run on.

The block is the section that runs the actual test. In this example, we use a rather convoluted way of checking that PID 1 exists. However, by doing so, we place an extra test artifact in the artifacts directory.

Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on. Try running this example test against an Atomic Host or Docker Container. It should pass. Try changing the /proc/1 argument to another value, and the test should fail.

You can use most of the Ansible techniques in your playbooks. And take a look at the standard-test-roles for Ansible roles to make writing your tests easier.

See Pull Requests for details about creating pull requests.

Marking the test to be run

Just having a .yml file in the right directory doesn't yet mean it will be invoked. Make sure to reference or add it from a tests.yml playbook. This is the entry point that the testing or CI system will use to invoke all the tests for a given package.

If the tests.yml file doesn't yet exist, create it. Lets continue with our above example and create a tests.yml with the following content:

- import_playbook: test_pid_1.yml

You can now run this test with the standard commands above.

Wrapping Tests

Let's say you have a script that runs a test. Its stdout and stderr is the test output, and an exit status of zero indicates success. Here's how we would wrap that test to be invoked. Lets say we have a simple script like in a file called test-simple

#!/bin/sh
set -ex
# exercise installed gzip/gunzip programs
echo "Bla" > bla.file
cp bla.file bla.file.orig
gzip bla.file
gunzip bla.file.gz
cmp bla.file bla.file.orig
rm bla.file bla.file.orig

We can write an Ansible wrapper for this script like this in test_simple.yml:

---
- hosts: localhost
  vars:
  - artifacts: ./artifacts
  tags:
  - atomic
  - classic
  - container
  remote_user: root
  tasks:
  - name: Install the test files
    copy: src={{ item.file }} dest=/usr/local/bin/{{ item.dest }} mode=0755
    with_items:
    - {file: test-simple, dest: test-simple }

  - name: Test block
    block:
      - name: Execute the tests
        shell: exec > /tmp/test.log 2>&1 && /usr/local/bin/test-simple

    always:
      - name: Pull out the logs
        fetch:
          dest: "{{ artifacts }}/"
          src: "/tmp/test.log"
          flat: yes

All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We create ensure this directory exists in the test.

The block is the section that runs the actual test.

Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on.

If the tests.yml file doesn't yet exist, create it. Lets continue with our above example and create a tests.yml with the following content:

- import_playbook: test_simple.yml

Try running this example test against an Atomic Host or Docker Container. It should pass.

See Standard Test Roles documentation for instructions how to wrap a BeakerLib and RHTS tests.