From Fedora Project Wiki

< CI

mNo edit summary
(Add sections for artifacts and inventory, update and deduplicate debugging)
Line 4: Line 4:


= Setup =
= Setup =
== Packages ==


As the first step install all necessary packages:
As the first step install all necessary packages:
Line 13: Line 15:
  dnf copr -y enable @osci/standard-test-roles
  dnf copr -y enable @osci/standard-test-roles
  dnf update standard-test-roles
  dnf update standard-test-roles
== Artifacts ==
Output of the test (such as the stdout/stderr output, log files or screenshots) is by default saved in the <code>artifacts</code> directory. Use <code>TEST_ARTIFACTS</code> environment variable to choose a different location if desired:
export TEST_ARTIFACTS=/tmp/artifacts
{{admon/important|Artifacts cleanup|Before running tests make sure that all logs /tmp/artifacts/test.* are deleted.}}
== Inventory ==
A ''test subject'' is what we call the thing to be tested. To turn a test subject into a launched, installed system to be tested, we use [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory]. Use the following command to enable it:
export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
As you can see from the way how the inventory is set, tests may contain their own inventory, which defines their own instructions for turning a ''test subject'' into one or more testable systems.


= Testing =
= Testing =
Line 19: Line 37:


You can always invoke the tests locally. Many tests modify or change the system they are run against, so take that into account when looking at how to invoke tests. The following examples invoke tests against the same system that the package git repository is checked out on. Below there are further options for invoking tests against another fully formed and integrated systems, such as an Atomic Host or container image ''test subject''.
You can always invoke the tests locally. Many tests modify or change the system they are run against, so take that into account when looking at how to invoke tests. The following examples invoke tests against the same system that the package git repository is checked out on. Below there are further options for invoking tests against another fully formed and integrated systems, such as an Atomic Host or container image ''test subject''.
{{admon/important|Work in progress|Before running tests make sure that all logs /tmp/artifacts/test.* are deleted.}}


There may be more than one test present in a package git repository, but the file <code>tests.yml</code> is the main entry point. To run it use the following command:
There may be more than one test present in a package git repository, but the file <code>tests.yml</code> is the main entry point. To run it use the following command:
Line 41: Line 56:
</pre>
</pre>


When run by a CI System the tests are [[Changes/InvokingTests|invoked according to a specification]]. Look here for more details and standard invocation variables.
When run by a CI System the tests are invoked according to the [[CI/Standard_Test_Interface|Standard Test Interface]]. Look here for more details and standard invocation variables.


== Package ==
== Package ==
Line 49: Line 64:
A ''test subject'' is what we call the thing to be tested. RPMs are a particular kind of ''test subject''. To turn a test subject into a launched, installed system to be tested, we use [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory]. Lets invoke the tests with an inventory and a specific version of gzip:
A ''test subject'' is what we call the thing to be tested. RPMs are a particular kind of ''test subject''. To turn a test subject into a launched, installed system to be tested, we use [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory]. Lets invoke the tests with an inventory and a specific version of gzip:


<pre>
curl -o gzip.rpm https://kojipkgs.fedoraproject.org//packages/gzip/1.8/2.fc26/x86_64/gzip-1.8-2.fc26.x86_64.rpm
# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
export TEST_SUBJECTS=$PWD/gzip.rpm
# curl -o gzip.rpm https://kojipkgs.fedoraproject.org//packages/gzip/1.8/2.fc26/x86_64/gzip-1.8-2.fc26.x86_64.rpm
ansible-playbook tests.yml
# export TEST_SUBJECTS=$PWD/gzip.rpm
# ansible-playbook tests.yml
</pre>


You'll notice that the RPM is installed into the testable system before invoking the tests. Some tests contain their own inventory, that is their own instructions for turning a ''test subject'' into one or more testable systems. But in this case we use the default <code>standard-test-roles</code> inventory in <code>/usr/share/ansible/inventory</code> to do this.
You'll notice that the RPM is installed into the testable system before invoking the tests. Some tests contain their own inventory, that is their own instructions for turning a ''test subject'' into one or more testable systems. But in this case we use the default <code>standard-test-roles</code> inventory in <code>/usr/share/ansible/inventory</code> to do this.
Line 62: Line 74:
Another example is to use a ''test subject'' of a container image. This is also a fully formed and integrated deliverable. The ''test subject'' again represents the thing to be tested. The container image is pulled from a registry and launched using docker by an [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory].
Another example is to use a ''test subject'' of a container image. This is also a fully formed and integrated deliverable. The ''test subject'' again represents the thing to be tested. The container image is pulled from a registry and launched using docker by an [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory].


<pre>
export TEST_SUBJECTS=docker:docker.io/library/fedora:26
# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
ansible-playbook --tags=container tests.yml
# export TEST_SUBJECTS=docker:docker.io/library/fedora:26
# ansible-playbook --tags=container tests.yml
</pre>


If you watch closely you'll notice the image is pulled if not already local, launched as a container, and then prepared for the tests to run on. The first time this may take a little longer. Not all tests are able to function in the somewhat different environment of a container. In fact, for certain tests, the software to be tested may not be included in the container. But many of the tests for core packages should work here.
If you watch closely you'll notice the image is pulled if not already local, launched as a container, and then prepared for the tests to run on. The first time this may take a little longer. Not all tests are able to function in the somewhat different environment of a container. In fact, for certain tests, the software to be tested may not be included in the container. But many of the tests for core packages should work here.
Line 72: Line 81:
The <code>--tags</code> argument filters out tests that are not suitable for running in a container, either because the system functions differently, or the correct packages are not installable.
The <code>--tags</code> argument filters out tests that are not suitable for running in a container, either because the system functions differently, or the correct packages are not installable.


To diagnose why the tests failed, and log into the running container, you can specify the following environment variable. After the playbook runs, you'll see diagnosis information with a helpful <code>docker exec</code> command to log into the container:
See the [[#Debug|Debug]] section for instructions how to log into a running container and diagnose why the tests failed.
 
<pre>
# export TEST_DEBUG=1
</pre>
 
To increase verbosity you can also use keys  <code>-v</code> or <code>-vvv</code> for full debugging, for example:
<pre>
# ansible-playbook --tags=container tests.yml -v
</pre>
or for full verbosity:
<pre>
# ansible-playbook --tags=container tests.yml -vvv
</pre>


'''Additional arguments for Docker'''
'''Additional arguments for Docker'''
Line 93: Line 89:
It is possible that some tests require additional privileges. In this case specify necessary arguments for Docker using an environment variable '''TEST_DOCKER_EXTRA_ARGS'''. For this create a file '''inventory''' file in '''tests''' directory with the following content:
It is possible that some tests require additional privileges. In this case specify necessary arguments for Docker using an environment variable '''TEST_DOCKER_EXTRA_ARGS'''. For this create a file '''inventory''' file in '''tests''' directory with the following content:


<pre>
#!/bin/bash
#!/bin/bash
export TEST_DOCKER_EXTRA_ARGS="--security-opt seccomp:unconfined"
export TEST_DOCKER_EXTRA_ARGS="--security-opt seccomp:unconfined"
exec merge-standard-inventory "$@"
exec merge-standard-inventory "$@"
</pre>


or
or


<pre>
#!/bin/bash
#!/bin/bash
export TEST_DOCKER_EXTRA_ARGS="--privileged"
export TEST_DOCKER_EXTRA_ARGS="--privileged"
exec merge-standard-inventory "$@"
exec merge-standard-inventory "$@"
</pre>


You can find nformation about '''merge-standard-inventory''' script you can find at: [https://pagure.io/standard-test-roles/blob/master/f/scripts/README.md README file for standard-test-roles]
See [https://pagure.io/standard-test-roles/blob/master/f/scripts/README.md merge-standard-inventory] documentation for details.


== Atomic ==
== Atomic ==
Line 113: Line 105:
The former example may seem a bit contrived, but the concept of a ''test subject'' starts to make more sense when you want to test a fully formed and integrated deliverable, such as Atomic Host. The ''test subject'' again represents the thing to be tested. The ''test subject'' in this case is a QCow2 image. To turn a test subject into a launched system ready to be tested, we use [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory].
The former example may seem a bit contrived, but the concept of a ''test subject'' starts to make more sense when you want to test a fully formed and integrated deliverable, such as Atomic Host. The ''test subject'' again represents the thing to be tested. The ''test subject'' in this case is a QCow2 image. To turn a test subject into a launched system ready to be tested, we use [http://docs.ansible.com/ansible/intro_dynamic_inventory.html Ansible dynamic inventory].


<pre>
curl -Lo /tmp/atomic.qcow2 https://getfedora.org/atomic_qcow2_latest
# export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)
export TEST_SUBJECTS=/tmp/atomic.qcow2
# curl -Lo fedora-atomic-host.qcow2 https://getfedora.org/atomic_qcow2_latest
ansible-playbook --tags=atomic tests.yml
# export TEST_SUBJECTS=$PWD/fedora-atomic-host.qcow2
# ansible-playbook --tags=atomic tests.yml
</pre>


If you watch closely you'll see that the Atomic Host image is booted, and the tests run against the launched image. Not all tests are able to function in the somewhat different environment of Atomic Host, in fact, for certain cases, the software to be tested may not be included in the Atomic Host ''test subject''. But most of the tests in core packages should work here.
If you watch closely you'll see that the Atomic Host image is booted, and the tests run against the launched image. Not all tests are able to function in the somewhat different environment of Atomic Host, in fact, for certain cases, the software to be tested may not be included in the Atomic Host ''test subject''. But most of the tests in core packages should work here.
Line 126: Line 115:
The <code>--tags</code> argument filters out tests that are not suitable for running on an Atomic Host, either because the system functions differently, or the correct packages are not available on that system.
The <code>--tags</code> argument filters out tests that are not suitable for running on an Atomic Host, either because the system functions differently, or the correct packages are not available on that system.


To diagnose why the tests failed, and log into the running Atomic Host, you can specify the following environment variable. After the playbook runs, you'll see diagnosis information with a helpful <code>ssh</code> command to log into the host:
See the [[#Debug|Debug]] section to learn how to diagnose why the tests failed, and log into the running Atomic Host.
 
<pre>
# export TEST_DEBUG=1
</pre>
 
To increase verbosity you can also use keys  <code>-v</code> or <code>-vvv</code> for full debugging, for example:
 
<pre>
# ansible-playbook --tags=atomic tests.yml -v
</pre>
 
or for full verbosity:
 
<pre>
# ansible-playbook --tags=atomic tests.yml -vvv
</pre>


{{admon/warning|Required Packages|Please note that if '''required_packages''' are specified in '''tests.yml''' for Atomic Host, additional packages will be installed using the '''rpm-ostree''' command which is affecting the test subject (it's similar as rebuilding an rpm package to be tested) so this should be used with caution and only when necessary. Also be aware that there are certain limitations for this approach (e.g. it's not possible to install different version of packages that are already part of the tree).}}
{{admon/warning|Required Packages|Please note that if '''required_packages''' are specified in '''tests.yml''' for Atomic Host, additional packages will be installed using the '''rpm-ostree''' command which is affecting the test subject (it's similar as rebuilding an rpm package to be tested) so this should be used with caution and only when necessary. Also be aware that there are certain limitations for this approach (e.g. it's not possible to install different version of packages that are already part of the tree).}}
Line 150: Line 123:
== Debug ==
== Debug ==


For debugging tests option ''TEST_DEBUG'' should be set to ''1''.
To increase output verbosity use option <code>-v</code> or <code>-vvv</code>:
To do that, execute command:
<pre>
# export TEST_DEBUG=1
</pre>
 


After that you can run tests i.e. execute command:
ansible-playbook --tags=container tests.yml -v


<pre>
or for full verbosity:
# ansible-playbook --tags=atomic tests.yml
</pre>
 
You'll see output like this:


<pre>
ansible-playbook --tags=container tests.yml -vvv
PLAY RECAP ****************************************************************************************************************************************
/home/sturi/Programs/Images/atomic.qcow2 : ok=1    changed=0    unreachable=0    failed=1 


(.ansible) [root@kh226 docker]#
To debug tests in a running container or atomic host use the <code>TEST_DEBUG</code> environment variable. After the playbook runs, you'll see diagnosis information with a helpful command to log in.
DIAGNOSE: ssh -p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@127.0.0.3 # password: foobar
DIAGNOSE: kill 13376 # when finished
</pre>


Look for ''DIAGNOSE'' there creds for the Virtual machine, IP address, username, password, and port.
export TEST_DEBUG=1
In our case:


* IP address: 127.0.0.3
For container you'll see output like this:
* username/password: root/foobar
* SSH port: 2222


Now you can connect to the VM using those creds. Open a new terminal and execute:
DIAGNOSE: docker exec -it 56de801f0ddde36fc9770666f7be2a68f89d7f18f52b7b6fe7df7a12b193bf08 /bin/bash
DIAGNOSE: kill 18261 # when finished


<pre>
For atomic host the instructions are a bit different:
# ssh root@127.0.0.3 -p 2222
</pre>


The output:
DIAGNOSE: ssh -p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@127.0.0.3 # password: foobar
DIAGNOSE: export ANSIBLE_INVENTORY=/tmp/inventory-cloudxyhF2M/inventory
DIAGNOSE: kill 16611 # when finished


<pre>
Now you can easily connect using these commands. Use suggested <code>kill</code> command to finish the running instance when done with investigation.
[root@kh226 docker]# ssh root@127.0.0.3 -p 2222
 
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[127.0.0.3]:2222' (ECDSA) to the list of known hosts.
root@127.0.0.3's password:
[root@localhost ~]#
</pre>


= Roles =
= Roles =

Revision as of 10:26, 6 March 2018

Description

Package standard-test-roles provides shared Ansible roles and inventory scripts implementing the Standard Test Interface. It has support for multiple testing frameworks (such as BeakerLib or Avocado) and in this way allows to easily enable existing tests in Fedora CI.

Setup

Packages

As the first step install all necessary packages: 

sudo dnf install fedpkg libselinux-python standard-test-roles

You can also install the latest version from the copr repo: 

dnf copr -y enable @osci/standard-test-roles
dnf update standard-test-roles

Artifacts

Output of the test (such as the stdout/stderr output, log files or screenshots) is by default saved in the artifacts directory. Use TEST_ARTIFACTS environment variable to choose a different location if desired: 

export TEST_ARTIFACTS=/tmp/artifacts
Artifacts cleanup
Before running tests make sure that all logs /tmp/artifacts/test.* are deleted.

Inventory

A test subject is what we call the thing to be tested. To turn a test subject into a launched, installed system to be tested, we use Ansible dynamic inventory. Use the following command to enable it: 

export ANSIBLE_INVENTORY=$(test -e inventory && echo inventory || echo /usr/share/ansible/inventory)

As you can see from the way how the inventory is set, tests may contain their own inventory, which defines their own instructions for turning a test subject into one or more testable systems.

Testing

Classic

You can always invoke the tests locally. Many tests modify or change the system they are run against, so take that into account when looking at how to invoke tests. The following examples invoke tests against the same system that the package git repository is checked out on. Below there are further options for invoking tests against another fully formed and integrated systems, such as an Atomic Host or container image test subject.

There may be more than one test present in a package git repository, but the file tests.yml is the main entry point. To run it use the following command: 

# ansible-playbook tests.yml

You can find output artifacts of the tests in an artifacts/ or specify a specific directory like this: 

# ansible-playbook -e artifacts=/tmp/output tests.yml

You can filter which kinds of tests are run by providing a --tags argument. To only run tests that are suited for classic systems installed by yum or dnf you can use a command like: 

# ansible-playbook --tags=classic tests.yml

When run by a CI System the tests are invoked according to the Standard Test Interface. Look here for more details and standard invocation variables.

Package

When you run the tests as above, the tests assume that the system to be tested is the same as the system invoking the tests. In particular, the test assumes that the thing to be tested is already installed.

A test subject is what we call the thing to be tested. RPMs are a particular kind of test subject. To turn a test subject into a launched, installed system to be tested, we use Ansible dynamic inventory. Lets invoke the tests with an inventory and a specific version of gzip: 

curl -o gzip.rpm https://kojipkgs.fedoraproject.org//packages/gzip/1.8/2.fc26/x86_64/gzip-1.8-2.fc26.x86_64.rpm
export TEST_SUBJECTS=$PWD/gzip.rpm
ansible-playbook tests.yml

You'll notice that the RPM is installed into the testable system before invoking the tests. Some tests contain their own inventory, that is their own instructions for turning a test subject into one or more testable systems. But in this case we use the default standard-test-roles inventory in /usr/share/ansible/inventory to do this.

Container

Another example is to use a test subject of a container image. This is also a fully formed and integrated deliverable. The test subject again represents the thing to be tested. The container image is pulled from a registry and launched using docker by an Ansible dynamic inventory. 

export TEST_SUBJECTS=docker:docker.io/library/fedora:26
ansible-playbook --tags=container tests.yml

If you watch closely you'll notice the image is pulled if not already local, launched as a container, and then prepared for the tests to run on. The first time this may take a little longer. Not all tests are able to function in the somewhat different environment of a container. In fact, for certain tests, the software to be tested may not be included in the container. But many of the tests for core packages should work here.

The --tags argument filters out tests that are not suitable for running in a container, either because the system functions differently, or the correct packages are not installable.

See the Debug section for instructions how to log into a running container and diagnose why the tests failed.

Additional arguments for Docker

Tests for containers are run with a help of Docker. Containers are run within default security context. For more info see Seccomp security profiles for Docker. It is possible that some tests require additional privileges. In this case specify necessary arguments for Docker using an environment variable TEST_DOCKER_EXTRA_ARGS. For this create a file inventory file in tests directory with the following content: 

#!/bin/bash
export TEST_DOCKER_EXTRA_ARGS="--security-opt seccomp:unconfined"
exec merge-standard-inventory "$@"

or 

#!/bin/bash
export TEST_DOCKER_EXTRA_ARGS="--privileged"
exec merge-standard-inventory "$@"

See merge-standard-inventory documentation for details.

Atomic

The former example may seem a bit contrived, but the concept of a test subject starts to make more sense when you want to test a fully formed and integrated deliverable, such as Atomic Host. The test subject again represents the thing to be tested. The test subject in this case is a QCow2 image. To turn a test subject into a launched system ready to be tested, we use Ansible dynamic inventory. 

curl -Lo /tmp/atomic.qcow2 https://getfedora.org/atomic_qcow2_latest
export TEST_SUBJECTS=/tmp/atomic.qcow2
ansible-playbook --tags=atomic tests.yml

If you watch closely you'll see that the Atomic Host image is booted, and the tests run against the launched image. Not all tests are able to function in the somewhat different environment of Atomic Host, in fact, for certain cases, the software to be tested may not be included in the Atomic Host test subject. But most of the tests in core packages should work here.

Some tests contain their own inventory, that is their own instructions for turning a test subject into one or more testable systems. But in this case we use the default standard-test-roles inventory to do this.

The --tags argument filters out tests that are not suitable for running on an Atomic Host, either because the system functions differently, or the correct packages are not available on that system.

See the Debug section to learn how to diagnose why the tests failed, and log into the running Atomic Host.

Required Packages
Please note that if required_packages are specified in tests.yml for Atomic Host, additional packages will be installed using the rpm-ostree command which is affecting the test subject (it's similar as rebuilding an rpm package to be tested) so this should be used with caution and only when necessary. Also be aware that there are certain limitations for this approach (e.g. it's not possible to install different version of packages that are already part of the tree).
Required Packages
Atomic Host is shipped as a base ostree image, however you can install additional packages with a help of rpm-ostree install command. Currently (10.01.2018 ) repo with additional packages is actual only for the latest base-ostree image. Consequence: tests that install additional packages for Atomic Host can fail sometimes with: error: The following base packages would be replaced: xxx Solution: make sure you have the latest Atomic Host image. Additional information you can find rpm-ostree issue 415 and a possible solution in the feature using rpm-ostree jigdo rpm-ostree issue 1081

Debug

To increase output verbosity use option -v or -vvv: 

ansible-playbook --tags=container tests.yml -v

or for full verbosity: 

ansible-playbook --tags=container tests.yml -vvv

To debug tests in a running container or atomic host use the TEST_DEBUG environment variable. After the playbook runs, you'll see diagnosis information with a helpful command to log in. 

export TEST_DEBUG=1

For container you'll see output like this: 

DIAGNOSE: docker exec -it 56de801f0ddde36fc9770666f7be2a68f89d7f18f52b7b6fe7df7a12b193bf08 /bin/bash
DIAGNOSE: kill 18261 # when finished

For atomic host the instructions are a bit different: 

DIAGNOSE: ssh -p 2222 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@127.0.0.3 # password: foobar
DIAGNOSE: export ANSIBLE_INVENTORY=/tmp/inventory-cloudxyhF2M/inventory
DIAGNOSE: kill 16611 # when finished

Now you can easily connect using these commands. Use suggested kill command to finish the running instance when done with investigation.

Roles

Here's the list of currently supported roles:

  • avocado
  • basic
  • beakerlib
  • repo
  • rhts
  • rpm
  • scripts
  • source

BeakerLib

If you have a set of beakerlib tests, it is recommended to place each test in its own subdirectory.

Then, create tests.yml file with contents similar to the following which runs tests using the 'standard-test-beakerlib' role included in the standard-test-roles package. The list provided for the 'tests' parameter should be the list of names of your per-test subdirectories, and the 'required_packages' parameter should contain a list of additional packages that need to be installed to run the tests. 

---
- hosts: localhost
  tags:
  - atomic
  - classic
  - container
  roles:
  - role: standard-test-beakerlib
    tests:
    - cmd-line-options
    required_packages:
    - which         # which package required for cmd-line-options
    - rpm-build     # upstream-testsuite requires rpmbuild command
    - libtool       # upstream-testsuite requires libtool
    - gettext       # upstream-testsuite requires gettext

Note: The 'required_packages' parameter is ignored when running on Atomic Host--since there is no way to install additional packages in that environment.

RHTS

If you have a set of tests you want to run using restraint, each test must be placed in its own subdirectory.

Then, create tests.yml file with contents similar to the following which runs tests using the 'standard-test-rhts' role included in the standard-test-roles package. The list provided for the 'tests' parameter should be the list of names of your per-test subdirectories, and the 'required_packages' parameter should contain a list of additional packages that need to be installed to run the tests. 

---
- hosts: localhost
  tags:
  - classic
  - docker
  roles:
  - role: standard-test-rhts
    tests:
    - cmd-line-options
    required_packages:
    - which         # which package required for cmd-line-options
    - rpm-build     # upstream-testsuite requires rpmbuild command
    - libtool       # upstream-testsuite requires libtool
    - gettext       # upstream-testsuite requires gettext

Note: Tests using the 'standard-test-rhts' role are not compatible with Atomic Host--since it requires the installation of additional packages and there is no way to do so in that environment. We reflect that by omitting atomic from the <tags> section.

More

Learn more about the Standard Test Roles.

Links

Pagure and Copr repositories:

Contact

  • Andrei Stepanov (astepano)
Retrieved from "https://fedoraproject.org/w/index.php?title=CI/Standard_Test_Roles&oldid=512197"