From Fedora Project Wiki
- AliVigni: In invocation why would I want to hardcode absolute paths for test execution, artifacts, logs, This should be a relative path so where ever you run things it is in the local workspace. My machine, Jenkins, taskotron, etc.
- MartinPitt: I reworked the invocation; it was also impractical for tests that run as non-root, and it would have potentially clobbered the root directory with temporary stuff.
- MartinPitt: In Fedora 25 there are currently 153 packages named
*-tests, none of which match this specification. Thus "If this file [check file in dist-git] is empty then the list of packages will default to %{name}-tests" does not work for discovery, as it would catch these packages. So we either need to change to a suffix that isn't being used yet, or (preferrable) always explicitly declare test packages which follow this standard.
- MartinPitt: It seems to me that shipping tests as RPMs and discovery in dist-git are the wrong way around. In the current spec we require packaged tests and (effectively, see above) declaring the test packages in dist-git.
- Test declarations in dist-git are not discoverable for an automated system that needs to decide which tests to run for a particular package update. For that it needs to be able to tell efficiently which (source) packages have tests and what their names are, and checking out all Fedora package gits is completely impractical an efficient. Thus this either (1) needs to be moved to a tag or special package name or a magic prefix in the package summary such as "FEDTEST:" (we need to give this thing a name!), so that tests can be discovered from the package index; or (2) we need an automated service which indexes all dist-gits regularly.
- Always packaging tests as RPMs doesn't fundamentally restrict the scope of this (we can package things like distro upgrade or Anaconda installer tests as new source packages which only ship test subpackages). However, in the vast majority of cases our tests will be platform independent (i. e. written in Python, shell, or other scripting languages) and not require any compiled bits. Of course sometimes they do, and for those the compiled bits can still be shipped in a
-testshelper RPM. So it would seem prudent to simply ship the tests themselves in the dist-git (tests/*executables), and the test metadata as a separate file. This would then not limit test metadata to what RPMs can express. - So at the very least this needs justification why packaging tests is desirable over putting them into dist-git directly. The main advantage that I see is that it makes test dependency installation a bit more straightforward for human users. But (1) you should really use a tool for this which cleans up after itself (and then it doesn't matter if that installs RPMs or checks out git), and (2) this prevents running tests on production systems and/or when you don't have root privileges.
- In case we do decide to ship tests in dist-git instead of RPMs, we still need to tag the SRPMs or RPMs with some "I have tests" marker for CI system discoverability.