Motivation
In order to make the CI workflow reliable and efficient it is crucial to keep the test coverage in a good shape at all times. Sharing test code between several packages (even within multiple branches of the same package) may significantly help to:
- Prevent test code duplication
- Minimize test maintenance
- Catch incompatibilities early
In general, tests define how the software works and the basic functionality of many packages doesn’t change that often. We try hard to keep the backward compatibility where possible. Thus it seems natural that, for such components, tests guarding the spec could change at a slower pace than the distribution branches.
See the whole ci-list discussion for some more context.
Implementation
Store test code in your preferred repository and reference the
tests from the dist-git yaml file. There is also a special
tests namespace dedicated for storing Fedora CI
integration tests:
Some of the Standard Test Roles (currently beakerlib and rhts) support fetching test code from remote repositories directly in their config in this way:
- role: standard-test-beakerlib repositories: - repo: "https://src.fedoraproject.org/tests/shell.git" dest: "shell"
Examples
Here are some real-life examples where sharing test code can increase long-term efficiency.
Shell
There are several shells which implement the POSIX specification: bash, ksh, mksh, zsh, dash. All of them share a significant amount of test coverage and it does not make sense to commit & maintain identical tests in five different repositories (+ possible branches).
Shell tests repository:
Bash tests.yml:
- hosts: localhost
roles:
- role: standard-test-beakerlib
tags:
- classic
repositories:
- repo: "https://src.fedoraproject.org/tests/shell.git"
dest: "shell"
tests:
- shell/func
- shell/login
- shell/smoke
required_packages:
- expect # login requires expect
- which # smoke requires which
Ksh tests.yml:
- hosts: localhost
roles:
- role: standard-test-beakerlib
tags:
- classic
repositories:
- repo: "https://src.fedoraproject.org/tests/shell.git"
dest: "shell"
tests:
- shell/func
- shell/login
- shell/smoke
environment:
PACKAGES: ksh
SH_BIN: ksh
required_packages:
- ksh
- expect # login requires expect
- which # smoke requires which
Ruby
Another example is Ruby: With about 80 packages related to Ruby on Rails it would be useful and efficient to have a single place for integration tests which verify that the framework is correctly working after updating any of these packages. Conversely, maintaining those tests in 80 repos would be a tedious task.
SELinux
Several SELinux user space components are sharing test coverage in a single selinux test repository:
Start
In order to create a new repository in the tests namespace file a new issue in the fedora infrastructure queue. Specify desired repository name, main admin and a short description/justification. For example:
Creating new test repositories will be soon supported directly in
the fedpkg tool as well: