No edit summary |
Martinpitt (talk | contribs) m (fix formatting) |
||
Line 46: | Line 46: | ||
'''Requirements:''' | '''Requirements:''' | ||
* Stage test suite dependencies cleanly | |||
* Standard way to locally invoke a test suite process | |||
* Standard location to place artifact to test | |||
* Standard way to retrieve test results, log and attachments | |||
What follows is a standard way to discover, package and invoke integration | What follows is a standard way to discover, package and invoke integration |
Revision as of 07:55, 28 March 2017
Standard Dicsovery, Packaging, Invocation of Integration Tests
Summary
What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.
Owner
- Name: Stef Walter
- Email: stefw@fedoraproject.org
- Name: Martin Pitt
- Email: martin@piware.de
Detailed Description
Many Fedora packages have unit tests. These tests are typically run during a
%check
RPM build step. These tests run in the build root in which
the package is built and may involve the BuildRequire
dependencies.
On the other hand, integration testing happens against a composed system. Several upstream projects have integration tests, but these integration tests have typically not been used or packaged in Fedora. Both Fedora QA and the Atomic Host team would like to create more integration tests, stored in dist-git, and use a CI system to run them on Fedora packages.
What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.
It is important that the CI system which runs the test is independent from the layout of the test themselves.
Goals:
- Implementation details of the integration testsuite or its framework should not leak into or rely on the CI system invoking the test.
- CI system implementation details should not leak into the testsuite or its metadata.
- It should be possible to change the CI system that runs a testsuite.
- It should be possible for a dist-git packager to run a testsuite locally.
Requirements:
- Stage test suite dependencies cleanly
- Standard way to locally invoke a test suite process
- Standard location to place artifact to test
- Standard way to retrieve test results, log and attachments
What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.
The integration tests are packaged and delivered through Fedora as
%{name}-tests
subpackages of the package they are associated with.
Packaging
Each dist-git repo that has integration tests should package those tests in a
%{name}-tests
subpackage. This is similar to the
%{name}-debuginfo
or %{name}-docs
subpackages we have
today.
The spec file for a dist-git repo may include upstream integration tests into
its %{name}-tests
subpackage. The spec file may also include tests
directly from files in tests/
subdirectory of the dist-git repo itself.
Invocation
To invoke the testsuite, the subpackage is installed. Each integration testsuite
that is included in installs an executable in the path /usr/tests/
.
To invoke the integration test suites, one would:
- Place the artifact being tested, usually RPM files, in
/var/cache/tests/
- Execute all executable files in
/usr/tests/
one at a time.- The testsuite is invoked with a working directory of
/
- The testsuite is invoked as root, and may drop privileges as desired
- Treat the stdout/stderr of each testsuite process as the testsuite result log.
- Examine the exit code of each testsuite process. Zero exit code is success, non-zero is failure.
- The testsuite is invoked with a working directory of
- Gather any additional result artifacts like screenshots which test suites place in a
/var/log/tests/
directory
Multiple subpackages may be installed as long as their dependencies do not conflict.
Staging
The %{name}-test
subpackage should Requires:
all other packages
that the testsuite executable needs in order to run. This includes libraries or frameworks,
or subsystems like libvirt
.
Some integration tests may choose to test in-situ, on the system on which the testsuite
is installed. In these cases the %{name}-tests
subpackage should directly
depend on the package being tested.
More rigourous integration tests test an integrated system from the outside. It is the
responsibility of the %{name}-tests
subpackages to provision virtual
machines or containers necessary to do such testing. In almost all cases this will happen
by way of a provisioning framework such as Avocado, Ansible, Module Testing Framework,
linch-pin, etc.
Discovery
A check
file in a dist-git repo should list the various names of
packages that should be installed in order to run the integration tests for that
package. If this file is empty then the list of packages will default to
%{name}-tests
.
The format of this file has not yet been defined, but a simple text file similar to sources listing package N[VR]'s may suffice.
Scope
This change requires no changes to Fedora infrastructure itself. It is limited to changes in dist-git repos. However certain key infrastructure changes could mitigate usability or side-effects of this change.
Benefit to Fedora
Developers benefit by having a consistent target for how to describe tests, while also being able to execute them locally while debugging issues or iterating on tests.
By packaging, staging and invoking tests consistently in Fedora we create an create an eco-system for the tests that allows varied testsuite frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.
User Experience
Users benefit by having tests that they can reproduce on their own systems.
They can install the similar to how they consume %{name}-doc
or %{name}-debuginfo
subpackages today.
We may choose to avoid having such packages available in the standard repositories.
We may choose to only have them in updates-testing
or an
arrangement similar to debuginfo
. These choices will require some
markup and/or change to infrastructure.
Upgrade/compatibility impact
Although there may already be subpackages that are named
%{name}-tests
this is merely a convention, and such packages will
not affect the behavior of this proposal.
Examples
TODO: Build out this section
Notes
- ...
- ...