|
|
| Line 1: |
Line 1: |
| = Motivation =
| | Moved to: https://docs.fedoraproject.org/en-US/ci/share-test-code/ |
| | |
| In order to make the CI workflow reliable and efficient it is
| |
| crucial to keep the test coverage in a good shape at all times.
| |
| Sharing test code between several packages (even within multiple
| |
| branches of the same package) may significantly help to:
| |
| | |
| * Prevent test code duplication
| |
| * Minimize test maintenance
| |
| * Catch incompatibilities early
| |
| | |
| In general, tests define how the software works and the basic
| |
| functionality of many packages doesn’t change that often. We try
| |
| hard to keep the backward compatibility where possible. Thus it
| |
| seems natural that, for such components, tests guarding the spec
| |
| could change at a slower pace than the distribution branches.
| |
| | |
| See the whole [https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org/thread/55U6V6UHA54MJLD2F6JF46EOLMVRUAE7/ ci-list discussion] for some more context.
| |
| | |
| = Implementation =
| |
| | |
| Store test code in your preferred repository and reference the
| |
| tests from the dist-git yaml file. There is also a special
| |
| <code>tests</code> namespace dedicated for storing Fedora CI
| |
| integration tests:
| |
| | |
| * https://src.fedoraproject.org/projects/tests/*
| |
| | |
| Use <code>fedpkg</code> to quickly clone repositories from the
| |
| tests namespace:
| |
| | |
| fedpkg clone tests/shell
| |
| | |
| Some of the [[CI/Standard_Test_Roles|Standard Test Roles]]
| |
| (currently basic and beakerlib) support fetching test code from
| |
| remote repositories directly in their config in this way:
| |
| | |
| - role: standard-test-beakerlib
| |
| repositories:
| |
| - repo: "https://src.fedoraproject.org/tests/shell.git"
| |
| dest: "shell"
| |
| | |
| It is also possible to specify version (branch, commit hash) which should
| |
| be fetched from the remote repository:
| |
| | |
| - role: standard-test-beakerlib
| |
| repositories:
| |
| - repo: "https://src.fedoraproject.org/tests/shell.git"
| |
| dest: "shell"
| |
| version: "devel"
| |
| | |
| = Testing Tests =
| |
| | |
| It is a good idea to ensure that updating tests in the shared
| |
| repository does not negatively impact packages which they are
| |
| testing. To enable pull request pipeline for tests stored in the
| |
| Fedora dist git tests namespace simply include `tests.yml` file in
| |
| the root of the test repository.
| |
| | |
| - hosts: localhost
| |
| roles:
| |
| - role: standard-test-basic
| |
| tags:
| |
| - classic
| |
| tests:
| |
| - smoke27:
| |
| dir: smoke
| |
| run: VERSION=2.7 METHOD=virtualenv ./venv.sh
| |
| - smoke36:
| |
| dir: smoke
| |
| run: VERSION=3.6 METHOD=virtualenv ./venv.sh
| |
| - smoke37:
| |
| dir: smoke
| |
| run: VERSION=3.7 METHOD=virtualenv ./venv.sh
| |
| required_packages:
| |
| - python27
| |
| - python36
| |
| - python37
| |
| | |
| The example above is a simplified version of the
| |
| [https://src.fedoraproject.org/tests/python/blob/master/f/tests.yml tests.yml]
| |
| file from the Python shared test repo and shows how to enable
| |
| `smoke` test to be executed against three versions of the Python
| |
| interpreter.
| |
| | |
| = Examples =
| |
| | |
| Here are some real-life examples where sharing test code can
| |
| increase long-term efficiency.
| |
| | |
| == Shell ==
| |
| | |
| There are several shells which implement the POSIX specification:
| |
| bash, ksh, mksh, zsh, dash. All of them share a significant amount
| |
| of test coverage and it does not make sense to commit & maintain
| |
| identical tests in five different repositories (+ possible
| |
| branches).
| |
| | |
| Shell tests repository:
| |
| | |
| * https://src.fedoraproject.org/tests/shell
| |
| | |
| Bash [https://src.fedoraproject.org/rpms/bash/blob/master/f/tests/tests.yml tests.yml]:
| |
| | |
| - hosts: localhost
| |
| roles:
| |
| - role: standard-test-beakerlib
| |
| tags:
| |
| - classic
| |
| repositories:
| |
| - repo: "https://src.fedoraproject.org/tests/shell.git"
| |
| dest: "shell"
| |
| tests:
| |
| - shell/func
| |
| - shell/login
| |
| - shell/smoke
| |
| required_packages:
| |
| - expect # login requires expect
| |
| - which # smoke requires which
| |
| | |
| Ksh [https://src.fedoraproject.org/rpms/ksh/blob/master/f/tests/tests.yml tests.yml]:
| |
| | |
| - hosts: localhost
| |
| roles:
| |
| - role: standard-test-beakerlib
| |
| tags:
| |
| - classic
| |
| repositories:
| |
| - repo: "https://src.fedoraproject.org/tests/shell.git"
| |
| dest: "shell"
| |
| tests:
| |
| - shell/func
| |
| - shell/login
| |
| - shell/smoke
| |
| environment:
| |
| PACKAGES: ksh
| |
| SH_BIN: ksh
| |
| required_packages:
| |
| - ksh
| |
| - expect # login requires expect
| |
| - which # smoke requires which
| |
| | |
| == Ruby ==
| |
| | |
| Another example is Ruby: With about 80 packages related to Ruby on
| |
| Rails it would be useful and efficient to have a single place for
| |
| integration tests which verify that the framework is correctly
| |
| working after updating any of these packages. Conversely,
| |
| maintaining those tests in 80 repos would be a tedious task.
| |
| | |
| Currently the shared [https://src.fedoraproject.org/tests/ruby tests/ruby]
| |
| repository hosts these three ruby integration tests:
| |
| | |
| Available integration tests:
| |
| * systemtap-static-probes-in-ruby - exercising ruby's systemtap api
| |
| * bundler-unit-test - run bundler's unit tests
| |
| * run-basic-rails-application - run a simple rails application
| |
| | |
| == SELinux ==
| |
| | |
| Several SELinux user space components are sharing test coverage in
| |
| a single [https://src.fedoraproject.org/tests/selinux.git selinux] test
| |
| repository:
| |
| | |
| * [https://src.fedoraproject.org/rpms/libsepol/blob/master/f/tests/tests.yml libsepol]
| |
| * [https://src.fedoraproject.org/rpms/libselinux/blob/master/f/tests/tests.yml libselinux]
| |
| * [https://src.fedoraproject.org/rpms/libsemanage/blob/master/f/tests/tests.yml libsemanage]
| |
| * [https://src.fedoraproject.org/rpms/libsepol/blob/master/f/tests/tests.yml policycoreutils]
| |
| | |
| = Start =
| |
| | |
| In order to create a new repository in the tests namespace use the fedpkg's request-tests-repo command. For example to create a shared test repository with the name foo, which will be available at [https://src.fedoraproject.org/tests/foo.git]
| |
| | |
| * Setup authentication to pagure according to the help in request-repo command
| |
| | |
| fedpkg request-repo -h
| |
| | |
| * Request a new repository with a sensible decription
| |
| | |
| fedpkg --module-name foo request-tests-repo "Description of the foo repository"
| |
