|
|
| (6 intermediate revisions by 2 users not shown) |
| Line 1: |
Line 1: |
| | | Moved to: https://docs.fedoraproject.org/en-US/ci/tests/ |
| == Enabling Tests ==
| |
| | |
| Tests may be written in different ways, but are exposed and invoked in a standard way as defined by the [[CI/Standard_Test_Interface|Standard Test Interface]] directly in the package [https://src.fedoraproject.org/projects/rpms/%2A git repository]. To start working on tests you can clone a package repo directly:
| |
| | |
| git clone https://src.fedoraproject.org/rpms/qrencode.git
| |
| | |
| You can also use the <code>fedpkg</code> to clone the repo. See the [[Package_maintenance_guide|Package Maintenance Guide]] for more info about the tool:
| |
| | |
| fedpkg clone -a qrencode
| |
| | |
| Tests are enabled by including the <code>tests.yml</code> file under the <code>tests</code> directory:
| |
| | |
| cd qrencode/tests
| |
| cat tests.yml
| |
| | |
| Tests are wrapped or written as [http://docs.ansible.com/ansible/playbooks.html Ansible playbooks]. Here is an example of a simple playbok which enables a single <code>smoke</code> test of the <code>qrencode</code> package:
| |
| | |
| - hosts: localhost
| |
| roles:
| |
| - role: standard-test-beakerlib
| |
| tags:
| |
| - classic
| |
| - container
| |
| - atomic
| |
| tests:
| |
| - smoke
| |
| required_packages:
| |
| - qrencode
| |
| - file
| |
| | |
| Let's now briefly look at the playbook to see which variables are defined in order to enable the smoke test:
| |
| | |
| * '''role''' — this test uses role <code>standard-test-beakerlib</code> from [[CI/Standard_Test_Roles|Standard Test Roles]] to run a BeakerLib test
| |
| * '''tags''' — all three test subjects ([[CI/Standard_Test_Roles#Atomic|classic]] rpm, docker [[CI/Standard_Test_Roles#Container|container]] and [[CI/Standard_Test_Roles#Atomic|atomic]] host) are relevant for this test
| |
| * '''tests''' — list of tests to be executed (here we have just a single smoke test)
| |
| * '''required_packages''' — list of rpm packages required for test execution
| |
| | |
| There may by multiple files ending in <code>.yml</code> in the <code>tests/</code> subdirectory and each of them can represent a test or a part of a test. All of them need to be included in the main <code>tests.yaml</code> file. Let's have a look at the <code>gzip</code> example:
| |
| | |
| > fedpkg clone -a gzip
| |
| Cloning into 'gzip'...
| |
| | |
| > cd gzip/tests/
| |
| > ls
| |
| test-simple test_simple.yml tests.yml
| |
| | |
| > cat tests.yml
| |
| - include: test_simple.yml
| |
| | |
| == Executing Tests ==
| |
| | |
| Before running tests make sure you have the following dependencies installed on your system:
| |
| | |
| dnf install ansible python2-dnf libselinux-python standard-test-roles
| |
| | |
| Although some playbooks may function without sudo, tests are always invoked as root. The test itself may set up users and/or drop permissions if a part of that test. But in general be sure to be root when invoking tests.
| |
| | |
| {{admon/important|Tests may modify or destroy your environment|It's recommended to use a virtual machine for testing to prevent any unwated changes performed by the test to your system.}}
| |
| | |
| Running a test directly on the current system is easy:
| |
| | |
| ansible-playbook tests.yml
| |
| | |
| To only run tests that are suited for classic systems installed by <code>yum</code> or <code>dnf</code> use the <code>--tags</code> argument:
| |
| | |
| ansible-playbook --tags=classic tests.yml
| |
| | |
| See [[CI/Standard_Test_Roles|Standard Test Roles]] documentation for detailed instructions how to run tests for a specific [[CI/Standard_Test_Roles#Package|Rpm Package]], [[CI/Standard_Test_Roles#Container|Docker Container]] or [[CI/Standard_Test_Roles#Atomic|Atomic Host]].
| |
| | |
| == Adding Tests ==
| |
| | |
| Test code itself can be stored directly in the dist-git (recommended as default) or fetched from another repository hosted in the Fedora infrastructure. [[CI/Share_Test_Code|Test Namespace]] can be used for storing test code relevant for multiple packages. The simplest way to add a new test is by using one of the existing [[CI/Standard_Test_Roles|Standard Test Roles]] which take care of many implementatin details. If you want to create a custom test follow instructions below.
| |
| | |
| Once you've identified a dist-git repository you will be adding new tests to (above), you can start to write a new Ansible test. Create an [http://docs.ansible.com/ansible/latest/playbooks.html Ansible playbook] with a new name. Make sure the extension is <code>.yml</code>. Lets place the following example in <code>test_pid_1.yml</code> file.
| |
| | |
| <pre>
| |
| ---
| |
| - hosts: localhost
| |
| vars:
| |
| - artifacts: ./artifacts
| |
| tags:
| |
| - atomic
| |
| - classic
| |
| - container
| |
| tasks:
| |
| - name: Test block
| |
| block:
| |
| - name: Test that /proc/1 exists
| |
| shell: ls /proc > /tmp/test.log && grep -qw 1 /tmp/test.log
| |
| | |
| always:
| |
| - name: Pull out the artifacts
| |
| fetch:
| |
| dest: "{{ artifacts }}/"
| |
| src: "/tmp/test.log"
| |
| flat: yes
| |
| </pre>
| |
| | |
| All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We ensure this directory exists in the test.
| |
| | |
| By use of <code>tags</code> we note what kind of systems this test is suitable to run on.
| |
| | |
| The <code>block</code> is the section that runs the actual test. In this example, we use a rather convoluted way of checking that PID 1 exists. However, by doing so, we place an extra test artifact in the artifacts directory.
| |
| | |
| Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on. Try running this example test against an [[CI/Standard_Test_Roles#Atomic|Atomic Host]] or [[CI/Standard_Test_Roles#Container|Docker Container]]. It should pass. Try changing the <code>/proc/1</code> argument to another value, and the test should fail.
| |
| | |
| You can use most of the Ansible techniques in your playbooks. And take a look at the [https://pagure.io/standard-test-roles standard-test-roles] for Ansible roles to make writing your tests easier.
| |
| | |
| See [[CI/Pull_Requests|Pull Requests]] for details about creating pull requests.
| |
| | |
| '''Marking the test to be run'''
| |
| | |
| Just having a <code>.yml</code> file in the right directory doesn't yet mean it will be invoked. Make sure to reference or add it from a <code>tests.yml</code> playbook. This is the entry point that the testing or CI system will use to invoke all the tests for a given package.
| |
| | |
| If the <code>tests.yml</code> file doesn't yet exist, create it. Lets continue with our above example and create a <code>tests.yml</code> with the following content:
| |
| | |
| - import_playbook: test_pid_1.yml
| |
| | |
| You can now run this test with the standard commands above.
| |
| | |
| == Wrapping Tests ==
| |
| | |
| Let's say you have a script that runs a test. Its stdout and stderr is the test output, and an exit status of zero indicates success. Here's how we would wrap that test to be invoked. Lets say we have a simple script like in a file called <code>test-simple</code>
| |
| | |
| #!/bin/sh
| |
| set -ex
| |
| # exercise installed gzip/gunzip programs
| |
| echo "Bla" > bla.file
| |
| cp bla.file bla.file.orig
| |
| gzip bla.file
| |
| gunzip bla.file.gz
| |
| cmp bla.file bla.file.orig
| |
| rm bla.file bla.file.orig
| |
| | |
| We can write an Ansible wrapper for this script like this in <code>test_simple.yml</code>:
| |
| | |
| <pre>
| |
| ---
| |
| - hosts: localhost
| |
| vars:
| |
| - artifacts: ./artifacts
| |
| tags:
| |
| - atomic
| |
| - classic
| |
| - container
| |
| remote_user: root
| |
| tasks:
| |
| - name: Install the test files
| |
| copy: src={{ item.file }} dest=/usr/local/bin/{{ item.dest }} mode=0755
| |
| with_items:
| |
| - {file: test-simple, dest: test-simple }
| |
| | |
| - name: Test block
| |
| block:
| |
| - name: Execute the tests
| |
| shell: exec > /tmp/test.log 2>&1 && /usr/local/bin/test-simple
| |
| | |
| always:
| |
| - name: Pull out the logs
| |
| fetch:
| |
| dest: "{{ artifacts }}/"
| |
| src: "/tmp/test.log"
| |
| flat: yes
| |
| </pre>
| |
| | |
| All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We create ensure this directory exists in the test.
| |
| | |
| The <code>block</code> is the section that runs the actual test.
| |
| | |
| Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on.
| |
| | |
| If the <code>tests.yml</code> file doesn't yet exist, create it. Lets continue with our above example and create a <code>tests.yml</code> with the following content:
| |
| | |
| - import_playbook: test_simple.yml
| |
| | |
| Try running this example test against an [[CI/Standard_Test_Roles#Atomic|Atomic Host]] or [[CI/Standard_Test_Roles#Container|Docker Container]]. It should pass.
| |
| | |
| See [[CI/Standard_Test_Roles|Standard Test Roles]] documentation for instructions how to wrap a [[CI/Standard_Test_Roles#BeakerLib|BeakerLib]] and [[CI/Standard_Test_Roles#RHTS|RHTS]] tests.
| |
| | |
| [[Category:FedoraAtomicCi]]
| |
