|
|
| Line 1: |
Line 1: |
| = Standard Discovery, Packaging, Invocation of Integration Tests = | | = Standard Discovery, Packaging, Invocation of Integration Tests = |
|
| |
|
| {{admon/warning|This is a proposal|Feedback is more than welcome. | | {{admon/warning|This Page is being reworked|This message will be removed when refactoring is complete}} |
| There's a ''discussion'' tab above.}}
| |
|
| |
|
| == Summary == | | == Current Proposals == |
| | | * [[InvokingTestsPackaged|Tests invoked via RPM packages]] |
| Lets define a clear delineation of between a testsuite (including framework) and the CI system that is running the test suite. This is the standard interface.
| | * [[InvokingTestsAnsible|Tests invoked via Ansible]] |
| | |
| [[File:Invoking-tests-standard-interface.png|800px]]
| |
| | |
| What follows is a standard way to discover, package and invoke integration tests for a package stored in a Fedora dist-git repo.
| |
| | |
| Many Fedora packages have unit tests. These tests are typically run during a <code>%check</code> RPM build step and run in a build root. On the other hand, integration testing should happen against a composed system. Upstream projects have integration tests, both Fedora QA and the Atomic Host team would like to create more integration tests, Red Hat would like to bring integration tests upstream.
| |
| | |
| == Owner ==
| |
| | |
| * Name: [[User:Stefw| Stef Walter]]
| |
| * Email: stefw@fedoraproject.org | |
| * Name: [[User:Pitti| Martin Pitt]]
| |
| * Email: martin@piware.de
| |
| <!-- * Release notes owner: To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
| |
| | |
| == Terminology ==
| |
| | |
| * '''Test Subject''': The items that are to be tested.
| |
| ** Examples: RPMs, OCI image, ISO, QCow2, Module repository ...
| |
| * '''Test''': A callable/runnable piece of code and corresponding test data and mocks which exercises and evaluates a ''test subject''.
| |
| * '''Test Suite''': The collection of all tests that apply to a ''test subject''.
| |
| * '''Test Framework''': A library or component that the ''test suite'' and ''tests'' use to accomplish their job.
| |
| ** Examples: [https://avocado-framework.github.io/ Avocado], [https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests GNOME Installed Tests], [https://pagure.io/modularity-testing-framework/ Modularity Testing Framework], [https://github.com/projectatomic/atomic-host-tests Ansible tests in Atomic Host], [https://tunir.readthedocs.io/en/latest/ Tunir tests], docker test images, ...
| |
| * '''Test Result''': A boolean pass/fail output of a ''test suite''.
| |
| ** ''Test results'' are for consumption by automated aspects of a ''testing systems''.
| |
| * '''Test Artifact''': Any additional output of the test suite such as the stdout/stderr output, log files, screenshots, core dumps, or TAP/Junit/subunit streams.
| |
| ** ''Test artifacts'' are for consumption by humans, archival or big data analysis.
| |
| * '''Testing System''': A CI or other ''testing system'' that would like to discover, stage and invoke tests for a ''test subject''.
| |
| ** Examples: [https://jenkins.io/ Jenkins], [https://taskotron.fedoraproject.org/ Taskotron], [https://docs.openstack.org/infra/zuul/ ZUUL], [https://ci.centos.org/ CentOS CI], Red Hat CI, [https://travis-ci.org/ Travis], [https://semaphoreci.com/ Semaphore], [https://developers.openshift.com/managing-your-applications/continuous-integration.html Openshift CI/CD], [https://wiki.ubuntu.com/ProposedMigration/AutopkgtestInfrastructure Ubuntu CI], ...
| |
| | |
| == Responsibilities ==
| |
| | |
| The '''testing system''' is responsible to:
| |
| * Build or otherwise acquire the ''test subject'', such as package, container image, tree …
| |
| * Decide which ''test suite'' to run, often by using the standard interface to discover appropriate ''tests'' for the dist-git repo that a test subject originated in.
| |
| * Schedule, provision or orchestrate a job to run the ''test suite'' on appropriate compute, storage, ...
| |
| * Stage the ''test suite'' as described by the ''standard interface''.
| |
| * Invoke the ''test suite'' as described by the ''standard interface''.
| |
| * Gather the ''test results'' and ''test artifacts'' as described by the ''standard interface''.
| |
| * Announce and relay the ''test results'' and ''test artifacts'' for gating, archival ...
| |
| | |
| The '''standard interface''' describes how to:
| |
| * Discover a ''test suite'' for a given dist-git repo.
| |
| * Uniquely identify a ''test suite''.
| |
| * Stage a ''test suite'' and its dependencies such as ''test frameworks''.
| |
| * Provide the ''test subject'' to the ''test suite''.
| |
| * Invoke a ''test suite'' in a consistent way.
| |
| * Gather ''test results'' and ''test artifacts'' from the invoked ''test suite''.
| |
| | |
| The '''test suite''' is responsible to:
| |
| * Declare its dependencies such as a ''test framework'' via the ''standard interface''.
| |
| * Execute the ''test framework'' as necessary.
| |
| * Provision (usually locally) any containers or virtual machines necessary for testing the ''test subject''.
| |
| * Provide ''test results'' and ''test subjects'' back according to the standard
| |
| | |
| The format of the textual logs and ''test artifacts'' that come out of a test suite is not prescribed by this document. Nor is it envisioned to be standardized across all possible ''test suites''.
| |
| | |
| == Requirements ==
| |
| | |
| * The ''test suite'' and ''test framework'' SHOULD NOT leak its implementation details into the testing system, other than via the ''standard interface''.
| |
| * The ''test suite'' and ''test framework'' SHOULD NOT rely on behavior of the testing system other than the ''standard interface''.
| |
| * The ''standard interface'' MUST enable a dist-git packager to run a ''test suite'' locally.
| |
| ** ''Test suites'' or ''test frameworks'' MAY call out to the network for certain tasks.
| |
| * It MUST be possible to stage an upstream ''test suite'' using the ''standard interface''.
| |
| * Both ''in-situ tests'', and more rigorous ''outside-in tests'' MUST be possible with the ''standard interface''.
| |
| ** For ''in-situ tests'' the ''test suite'' is in the same file system tree and process space as the ''test subject''.
| |
| ** For ''outside-in tests'' the ''test suite'' is outside of the file system tree and process space of the ''test subject''.
| |
| * The ''test suite'' and ''test framework'' SHOULD be able to provision containers and virtual machines necessary for its testing without requesting them from the ''testing system''.
| |
| * The ''standard interface'' SHOULD describe how to uniquely identify a ''test suite'',
| |
| | |
| == Detailed Description ==
| |
| | |
| This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the ''testing system'' from the ''test suite'' and its framework. It is also important to allow packagers to locally and manually invoke a ''test suite''.
| |
| | |
| === Packaging ===
| |
| | |
| The integration tests are packaged and delivered through Fedora as packages.
| |
| | |
| Each dist-git repo that has integration tests should package those tests in one or more subpackages
| |
| like <code>%{name}-tests</code>. This is similar to the
| |
| <code>%{name}-debuginfo</code> or <code>%{name}-docs</code> subpackages we have
| |
| today.
| |
| | |
| The spec file for a dist-git repo may install upstream integration tests as files in
| |
| its <code>%{name}-tests</code> package. The spec file may also include tests
| |
| directly from files in <code>tests/</code> subdirectory of the dist-git repo itself.
| |
| | |
| The tests package should use <code>Requires:</code> to require any other package, testing framework, or dependency necessary to run the tests. In ''in-situ'' testing cases, the tests package will directly <code>Requires:</code> the package of the ''test subject''.
| |
| | |
| === Invocation ===
| |
| | |
| To invoke the test suite, the test package that contains it is installed. Each test of the suite
| |
| installs an executable in the path <code>/usr/tests/</code>''sourcepackage''<code>/</code> (this will avoid name collisions between packages).
| |
| | |
| To invoke the test suite, one would:
| |
| | |
| # Create a temporary directory, referred to as: <code>$TESTDIR</code>
| |
| # Place the ''test subject(s)'' being tested in <code>$TESTDIR/subjects/</code>
| |
| # Execute all executable files in <code>/usr/tests/*/</code> directories one at a time.
| |
| ## Each executable test is invoked with a working directory of <code>$TESTDIR</code>
| |
| ## Each executable test is invoked as root, and may drop privileges as desired.
| |
| ## Treat the stdout/stderr of the test process as the test log. This is a standard ''test artifact'' and written to <code>$TESTDIR/artifacts/testname.log</code>.
| |
| ## Examine the exit code of each test process. Zero exit code is a successful ''test result'', non-zero is failure.
| |
| # Tests can put any additional ''test artifacts'' like screenshots into <code>$TESTDIR/artifacts/</code>.
| |
| | |
| This ensures that tests can be run on a production system without accidentally clobbering permanent directories,
| |
| don't require root privileges (simplifies test development), and that CI systems have one unique place from where
| |
| to collect artifacts. It also avoids collecting temporary files such as downloaded container or VM images as artifacts,
| |
| as these would usually get stored for a longer time period.
| |
| | |
| These steps would usually be done through a standard test driver tool (particularly for sensible stdout/stderr teeing and log capturing), but its usage is not mandatory for developing and calling tests manually.
| |
| | |
| === Staging ===
| |
| | |
| The <code>%{name}-test</code> package should <code>Requires:</code> all other packages
| |
| that the testsuite executable needs in order to run. This includes libraries or frameworks,
| |
| or subsystems like <code>libvirt</code>.
| |
| | |
| Some integration tests may choose to test ''in-situ'', on the system on which the test suite
| |
| is installed. In these cases the <code>%{name}-tests</code> package should directly
| |
| depend on the package being tested.
| |
| | |
| More rigorous integration tests are ''outside-in''. They test an integrated system without affecting its contents. It is the responsibility of the <code>%{name}-tests</code> packages to provision virtual machines or containers necessary to do such testing. In almost all cases this will happen by way of a provisioning framework such as [http://avocado-framework.readthedocs.org/ Avocado], [https://www.ansible.com/ Ansible], [https://pagure.io/modularity-testing-framework/ Module Testing Framework], [https://linch-pin.readthedocs.io/en/latest/ linch-pin], etc.
| |
| | |
| Multiple tests packages may be installed as long as their dependencies do not conflict.
| |
| | |
| === Discovery ===
| |
| | |
| A testing system needs to be able to efficiently answer the question "does this subject have any tests packages, and if so, what are their names". This should be automatically discoverable to the extent possible.
| |
| | |
| For any RPM ''test subject'' this process requires no additional metadata and can be fully automatic:
| |
| | |
| * It is possible to map a RPM to its SRPM source package (<code><rpm:sourcerpm></code> in the package index <code>*-primary.xml.gz</code>).
| |
| * One can map an SRPM to all the RPMs that it builds (from the same index), and using the <code>*-filelists.xml.gz</code> index one can mechanically tell which of the RPMs are of this test package kind described here.
| |
| | |
| '''TODO''': For other types of test subject cases such as docker images or distribution ISO files this discovery still needs to be discussed.
| |
| * E. g. a <code>Dockerfile</code> might grow a reference to a test package RPM, or at least initially there is a manually maintained map of subject to test package in the testing system.
| |
| | |
| == Scope ==
| |
| | |
| This change requires no initial changes to Fedora infrastructure itself. The change only affects contents spec files in dist-git repos.
| |
| | |
| '''TODO:''' However certain key infrastructure changes could mitigate usability or side-effects of this change. In particular, once this grows beyond the experimental phase, these test packages need to be put into a separate archive, similar to <code>-debuginfo</code>.
| |
| | |
| * How much effort is that to set up?
| |
| * Does this require any additional tags, keywords, or other explicit declaration in the spec file, other than "this RPM ships something in <code>/usr/tests/*</code>"?
| |
| | |
| == Benefit to Fedora ==
| |
| | |
| Developers benefit by having a consistent target for how to describe tests, while also being able to execute them locally while debugging issues or iterating on tests.
| |
| | |
| By packaging, staging and invoking tests consistently in Fedora we create an eco-system for the tests that allows varied test frameworks as well as CI system infrastructure to interoperate. The integration tests outlast the implementation details of either the frameworks they're written in or the CI systems running them.
| |
| | |
| == User Experience ==
| |
| | |
| A standard way to package tests benefits Fedora stability, and makes Fedora better for users.
| |
| | |
| Users could also benefit by having tests that they can reproduce on their own systems. They could install the similar to how they consume <code>%{name}-doc</code> or <code>%{name}-debuginfo</code> subpackages today.
| |
| | |
| We may choose to avoid having such packages available in the standard repositories. We may choose to only have them in <code>updates-testing</code> or an arrangement similar to <code>debuginfo</code>. These choices will require some
| |
| markup and/or change to infrastructure.
| |
| | |
| == Upgrade/compatibility impact ==
| |
| | |
| Although there may already be packages that are named <code>%{name}-tests</code> this is merely a convention, and such packages will not affect the behavior of this proposal.
| |
| | |
| == Comparison with Debian's autopkgtest ==
| |
| | |
| Debian/Ubuntu have used CI with packaged tests (called "autopkgtests") for many years, with
| |
| [https://ci.debian.net/status/ over 7.000 tests]. These are good candidates or at least bases for taking into Fedora
| |
| packages. This compares the structure of autopkgtest with this proposal to learn from autopkgtest's experiences and take
| |
| what works, and justifies the differences. See the
| |
| [https://people.debian.org/~mpitt/autopkgtest/README.package-tests.html format definition] for details.
| |
| | |
| === Packaging ===
| |
| | |
| '''Similarities:''' Both specifications use an existing test metatada format (RPM spec files with <code>Requires:</code> here, Debian RFC822 control files with <code>Depends:</code>in autopkgtest).
| |
| | |
| '''Differences:'''
| |
| * This specification requires packaging tests as binary RPM packages, whereas autopkgtest opted for keeping the test in the source package (equivalent of dist-git) only. The latter avoids the overhead of packaging the tests and having to create a separate archive for them. An important point is also that installing an RPM `-test` package requires root privileges, while invoking autopkgtest doesn't.
| |
| * As autopkgtest uses a separate control file (<code>debian/tests/control</code> instead of <code>debian/control</code> which describes the binary packages), it offers a much richer set of test metadata which cannot be expressed with <code>debian/control</code> or RPM spec files.
| |
| | |
| === Invocation ===
| |
| | |
| '''Simimlarities''': The test interface is very similar: In both specifications, a test is an executable
| |
| (of any script or compiled language), the exit code is the primary indicator of pass/fail, the executable's stdout/err
| |
| is a standard test artifact ("test log"), and tests can write additional artifacts into the
| |
| <code>$AUTOPKGTEST_ARTIFACTS</code> dir (like <code>./artifacts/</code> here).
| |
| | |
| '''Differences:'''
| |
| * By default, autopkgtest considers a test as failed if it produces anything on stderr, for catching unexpected new warnings. This can be disabled with adding <code>Restrictions: allow-stderr</code> to the test metadata. However, this turned out to be not overly useful, and tests which want to intercept warnings should better do that themselves.
| |
| * autopkgtest has no concept of passing test subjects to the test. Tests expect that their subjects are already available/installed, i. e. they get called in a testbed of the desired kind and state. It is the responsibility of the autopkgtest command line tool (the "test driver/executor") to install proposed new package(s) into the testbed (due to its origin of being primarily focussed on testing packages). For testing desktop/cloud images, upgrades, or other non-package subjects, it is instead the testing system's responsibility to produce the desired testbed and call <code>autopkgtest</code> on it. As the scope of this specification puts the staging into the hand of the test instead of the testing system, passing the test subject is a necessary consequence.
| |
| | |
| === Staging ===
| |
| | |
| '''Similarities:''' Both specifications use standard dpkg/rpm package dependencies (<code>Depends:</code> for dpkg, <code>Requires:</code> for rpm) to pull in test dependencies, and both can opt into doing their own provisioning of containers/VMs etc. for doing outside-in tests instead of in-situ. However, of all the ~ 7.000 autopkgtests, only a small handful is actually doing that (known cases are systemd and open-iscsi), as the vast majority of package/upgrade/image tests can (because it's sufficient) and should (because it's magnitudes faster) be run in-situ.
| |
| | |
| '''Differences:''' Here the test itself is responsible for installing the test subjects, while in autopkgtest it's the testing system's responsibility (see above).
| |
| | |
| === Discovery ===
| |
| | |
| '''Similarities:''' The idea is the same in both specifications. Here, as soon as there is a binary package that ships <code>/usr/tests/*</code> it can be discovered through file lists. In autopkgtest, as soon as there is a <code>debian/tests/control</code>, the source package index entry will automatically get a <code>Testsuite: autopkgtest</code> tag. So in both cases the developer does not need to explicitly do anything other than adding the tests.
| |
| | |
| '''Differences:''' None concerning the interface, just technical implementation details due to how rpm/dpkg work.
| |
| | |
| == Examples ==
| |
| | |
| What follows are examples of writing and/or packaging existing tests to this standard.
| |
| | |
| There is a mock ''test system''' which is a simple shell script: [http://piware.de/tmp/run-installed-test run-installed-test]. It runs all <code>/usr/tests/*</code>, can pass arbitrary subjects to them, and report/capture the results/logs. This is purely to study what a CI system would do and whether the standard interface works.
| |
| | |
| === Example: Simple in-situ test ===
| |
| | |
| Add simple downstream integration test for gzip:
| |
| | |
| * Package: '''gzip-tests'''
| |
| * dist-git: https://github.com/martinpitt/fedora-gzip-test
| |
| * Reference: https://patches.ubuntu.com/g/gzip/ (taken verbatim from Ubuntu's autopkgtest)
| |
| | |
| With this you can install test RPM from above gzip repo:
| |
| | |
| $ sudo rpm -i results_gzip/1.8/2.fc27/gzip-tests-1.8-2.fc25.x86_64.rpm
| |
| | |
| and run the gzip tests on the already installed package (as user) with
| |
| | |
| $ ~/run-installed-test
| |
| Subjects/artifacts directory: /tmp/test.vsR
| |
| -----------------------------------------
| |
| Running /usr/tests/gzip/test-simple
| |
| -----------------------------------------
| |
| ++ ls 'subjects/*.rpm'
| |
| + echo Bla
| |
| + cp bla.file bla.file.orig
| |
| + gzip bla.file
| |
| + gunzip bla.file.gz
| |
| + cmp bla.file bla.file.orig
| |
| + rm bla.file bla.file.orig
| |
| PASS: /usr/tests/gzip/test-simple
| |
| | |
| $ ls -l /tmp/test.vsR/artifacts/
| |
| -rw-r--r-- 1 martin martin 156 Mar 28 16:49 test-simple.log
| |
| | |
| or run them as root (as officially specified) with a subject (locally built gzip RPM):
| |
| | |
| $ sudo ~/run-installed-test results_gzip/1.8/2.fc27/gzip-1.8-2.fc25.x86_64.rpm
| |
| Installing subject results_gzip/1.8/2.fc27/gzip-1.8-2.fc25.x86_64.rpm
| |
| Subjects/artifacts directory: /tmp/test.Cck
| |
| -----------------------------------------
| |
| Running /usr/tests/gzip/test-simple
| |
| -----------------------------------------
| |
| ++ ls subjects/gzip-1.8-2.fc25.x86_64.rpm
| |
| + '[' -w / ']'
| |
| + rpm --verbose --force -U subjects/gzip-1.8-2.fc25.x86_64.rpm
| |
| Preparing packages...
| |
| gzip-1.8-2.fc25.x86_64
| |
| + echo Bla
| |
| + cp bla.file bla.file.orig
| |
| + gzip bla.file
| |
| + gunzip bla.file.gz
| |
| + cmp bla.file bla.file.orig
| |
| + rm bla.file bla.file.orig
| |
| PASS: /usr/tests/gzip/test-simple
| |
| | |
| === Example: GNOME style "Installed Tests" ===
| |
| | |
| Add downstream integration test running in gnome installed tests.
| |
| | |
| * Package: '''glib2-tests'''
| |
| * dist-git: https://github.com/martinpitt/fedora-glib2-test
| |
| * Reference: https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests
| |
| | |
| === Example: Tests run in Docker Container ===
| |
| | |
| Add integration test running glib2 installed tests in a docker container. This is also an example of having two different tests packages being created by the same dist-git repo.
| |
| | |
| * Package: '''glib2-container-tests'''
| |
| * dist-git: https://github.com/martinpitt/fedora-glib2-test
| |
| * Based on Debian/Ubuntu's autopkgtest glib2 related script
| |
| | |
| === Example: Modularity testing Framework ===
| |
| | |
| TODO: Port [https://pagure.io/modularity-testing-framework/blob/master/f/examples an example]
| |
| | |
| === Example: Ansible with Atomic Host ===
| |
| | |
| TODO: Port [https://github.com/projectatomic/atomic-host-tests an existing test]
| |
| | |
| === Example: Beakerlib based test ===
| |
| | |
| TODO: Port and shim a beakerlib test
| |
| | |
| === Example: Cockpit upstream test ===
| |
| | |
| Run upstream integration test, which uses VMs through libvirt, in a docker
| |
| container; the entire libvirt/bridge setup is confined to the container, so
| |
| this can be run without interfering with the host system.
| |
| | |
| * Package: '''cockpit-tests'''
| |
| * dist-git: https://github.com/martinpitt/cockpit/commits/packaged-tests - The penultimate commit creates a new <code>cockpit-integration-tests</code> package containing the actual tests and libvirt/qemu etc. dependencies, the topmost commit provides the <code>/usr/tests/</code> script that runs the former in docker.
| |
| * Reference: https://github.com/cockpit-project/cockpit/blob/master/test/README
| |
| | |
| == Evaluation ==
| |
| | |
| ''Instructions:'' Copy the block below, sign your name and fill in each section with your evaluation of that aspect. Add additional bullet points with overall summary or notes.
| |
| | |
| '''Full Name''' -- SignAture
| |
| * ''Summary:'' ...
| |
| * ''Staging:'' ...
| |
| * ''Invocation:'' ...
| |
| * ''Discovery:'' ...
| |
| | |
| '''Stef Walter''' -- Stefw
| |
| * ''Summary:''
| |
| ** Disclaimer: I am one of the owners above.
| |
| ** PRO: RPM is used for staging. RPM and YUM-style reposotiries are a standard part of Fedora. No other technology is involved in the standard.
| |
| ** PRO: Simple Unix invocation mechanism: executable + stdin/stdout + environment variables.
| |
| ** CON: RPM has a learning curve. Although a dist-git maintainer is required to already know about this.
| |
| ** CON: /usr/tests is a new FHS directory, should probably be /usr/libexec/tests.
| |
| ** CON: Only a partial way to describe whether tests are compatible with or conflict with a specific NVR of test subjects.
| |
| ** CON: The *-tests packages may require special handling if the distro does not want to have users able to install/run tests.
| |
| * ''Staging:''
| |
| ** Requires rpm and yum/dnf as well known staging dependencies.
| |
| ** The *-tests suffix is implied by the standard, not required. Is this confusing?
| |
| * ''Invocation:''
| |
| ** The standard describes how multiple test suites can be staged together and executed in one shot.
| |
| * ''Discovery:''
| |
| ** An NVR is the unique identifier for a test suite.
| |
| ** This uses capabilities of how YUM repositories work, but requires no additional technology.
| |
| | |
| | |
| '''Pierre-Yves Chibon''' -- pingou
| |
| * ''Summary:''
| |
| ** PRO: RPM is used and well-know by all packagers.
| |
| ** CON: It would require buy-in by the Fedora Packaging Committee and documentation in the Fedora Package Guidelines
| |
| ** CON: Complexifies the spec file, some of which are already quite complex/un-readable
| |
| ** CON: What about auto-generated spec file? (Think TexLive)
| |
| ** CON: Require local tooling to run the tests (or rpm -ql <foo>-test first to find the executables)
| |
| ** CON: /usr/tests needs to be changed
| |
| ** CON: How are test subject supported?
| |
| ** CON: Will take up space on the mirror
| |
| ** CON: Packagers and QA/testers are working on the same file all the time, high chances PR will conflict, higher changes to disagreement among contributors
| |
| * ''Staging:''
| |
| ** PRO: Requires rpm and yum/dnf as well known staging dependencies.
| |
| ** CON: How is test subject supported?
| |
| * ''Invocation:''
| |
| ** CON: The standard describes how multiple test suites can be staged together and executed but not in one shot, first install then run. -> May need a wrapper tool to do both in one go.
| |
| * ''Discovery:''
| |
| ** PRO: An NVR is the unique identifier for a test suite.
| |
| *** Also meaning that the test suite may change NVR while its content has not changed or we would need to define a EVR just for the -test sub-package
| |
| ** PRO: This uses capabilities of how RPM repositories work, but requires no additional technology.
| |
| ** CON: This implies tracking the dependencies at two distinct places, the main package then the -test sub-package
| |
| | |
| | |
| | |
| '''Tim Flink''' -- Tflink
| |
| * ''Summary:''
| |
| ** PRO: No significantly new technology, no huge requirements for additional software development
| |
| ** PRO: Paradigm works really well for package-specific tests
| |
| ** CON: All involved folks need to learn RPM packaging
| |
| ** CON: RPM packaging overhead once folks learn RPM packaging
| |
| ** CON: Not sure how well the paradigm works for non-package testing (containers, images, etc.)
| |
| * ''Staging:''
| |
| ** Nothing to add here, same concerns about convention over having a standard.
| |
| ** Not clear on how the test subject is found or passed into the framework? How are tests modified during development? How does one find the correct NVR for the test rpm at staging time?
| |
| * ''Invocation:''
| |
| ** What kinds of features would we eventually want to see in the <code>run-installed test</code> script?
| |
| ** How do we differentiate between in-situ and outside-in tests? Is this needed?
| |
| * ''Discovery:''
| |
| ** Are there existing tools to do the RPM to SRPM mapping?
| |
