Standard Discovery, Packaging, Invocation of Integration Tests via a distgit control file
Feedback is more than welcome. There's a discussion tab above.
First see the Terminogy division of Responsibilities and Requirements
Detailed Description
This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the testing system from the test suite and its framework. It is also important to allow packagers to locally and manually invoke a test suite.
Discovery
The description of the tests are stored in the dist-git repos in the tests/control file. The control file follows the syntax described in the DEP 8 specification.
Here is an example of a control file to describe 2 tests for systemd:
Tests: timedated, hostnamed Depends: systemd, acl, tzdata, Restrictions: needs-root, isolation-container
In the example, it references 2 tests (timedated and hostnamed) with the needed packages to install and the features needed on the system under test (root access and container isolation in the example).
The tests themselves are described in the files named from the Tests field (timedated and hostnamed in the example) and they need to be executable. They can be programmed in any language (shell, python, ansible...).
Functional tests that are included in the source tree of the package could be packaged and driven through the test scripts.
Some tests could be reused from the Debian/Ubuntu packages allowing more collaboration with these Linux distributions. They have more than 6000 packages with tests.
Staging
The system under test needs to be selected according the Restrictions field. This will allow to run tests on light setups to full systems: from chroot, schroot, container, ssh accessible system (vm or bare metal)...
Invocation
We have 2 options: port the autopkgtest tool to support rpm/dnf (a partial port is available) or write a program that reads the tests/control file, copy the needed the files into the system under test and run the corresponding test scripts.
Here is an invocation of autopkgtest to run test timedated on a remote system 192.168.122.215 launched from a Fedora 25 system:
$ autopkgtest -B --test-name timedated $PWD -- ssh --capability isolation-machine --capability root-on-testbed -H 192.168.122.215 -l user autopkgtest [21:16:18]: host localhost.localdomain; command line: /home/fred/external/autopkgtest/runner/autopkgtest -B --test-name timedated /home/fred/external/systemd -- ssh --capability isolation-machine --capability root-on-testbed -H 192.168.122.215 -l user autopkgtest [21:16:19]: testbed architecture: x86_64 autopkgtest [21:16:20]: testbed running kernel: Linux 4.11.0-0.rc5.git4.1.fc27.x86_64 #1 SMP Fri Apr 7 14:30:32 UTC 2017 autopkgtest [21:16:20]: @@@@@@@@@@@@@@@@@@@@ built-tree /home/fred/external/systemd autopkgtest [21:16:20]: test timedated: preparing testbed autopkgtest [21:16:20]: test timedated: publishing Python detected LC_CTYPE=C: LC_ALL & LANG coerced to C.UTF-8 (set another locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion behaviour). Last metadata expiration check: 1:32:45 ago on Thu Apr 13 20:43:35 2017 MSK. Package systemd-233-2.fc27.x86_64 is already installed, skipping. Package acl-2.2.52-13.fc26.x86_64 is already installed, skipping. Package tzdata-2017b-1.fc27.noarch is already installed, skipping. Dependencies resolved. Nothing to do. Complete! autopkgtest [21:16:25]: test timedated: [----------------------- original tz: Europe/Moscow timedatectl works change timezone reset timezone to original autopkgtest [21:16:26]: test timedated: -----------------------] autopkgtest [21:16:27]: test timedated: - - - - - - - - - - results - - - - - - - - - - timedated PASS autopkgtest [21:16:27]: @@@@@@@@@@@@@@@@@@@@ summary timedated PASS
autopkgtest can also invoke tests locally using the null target but at the launcher own risks.
Another interesting feature is to be able to control reboot multiple times during the tests.
Examples
Here is an extract of the timedated test from the systemd example:
#!/bin/sh set -e . `dirname $0`/assert.sh ORIG_TZ=$(timedatectl | grep -F 'Time zone'|sed -e 's@.*: @@' -e 's@ .*@@') echo "original tz: $ORIG_TZ" echo 'timedatectl works' assert_in "Local time:" "`timedatectl --no-pager`" echo 'change timezone' assert_eq "`timedatectl --no-pager set-timezone Europe/Moscow 2>&1`" "" assert_eq "`readlink /etc/localtime | sed 's#^.*zoneinfo/##'`" "Europe/Moscow" [ -n "$TEST_UPSTREAM" ] || assert_in "Europe/Moscow" "`timedatectl | grep -F 'Time zone: '`" assert_in "Time.*zone: Europe/Moscow (MSK, +" "`timedatectl --no-pager`" echo 'reset timezone to original' assert_eq "`timedatectl --no-pager set-timezone $ORIG_TZ 2>&1`" "" assert_eq "`readlink /etc/localtime | sed 's#^.*zoneinfo/##'`" "$ORIG_TZ" [ -n "$TEST_UPSTREAM" ] || assert_eq "`timedatectl | grep -F 'Time zone'|sed -e 's@.*: @@' -e 's@ .*@@'`" "$ORIG_TZ"
dist-git example for systemd
You can look at tests directory that contains 4 tests, one in ansible, one in python3 and 2 in shell.
Upgrade/compatibility impact
There are no real upgrade or compatibility impact. The tests will be branched per release as the dist-git repos are branched.