From Fedora Project Wiki
< Changes
During the selection process for a standard test invocation and and layout format for Fedora, several proposals were examined.
The Standard Interface is now documented.
There was an evaluation announced on ci@lists.fedoraproject.org and you can see information about voting and how things were chosen.
Evaluations
Instructions: In depth evaluations should be done in the sub-proposal pages. Read the proposals, you'll find the evaluation sections. Indicate vote below in the voting section
Voting
Every single vote requires an evaluation.
| Contributor | Packaged Tests | Ansible Tests | Control Tests | Notes |
|---|---|---|---|---|
| YourUserName | +1 | This is just an example, please vote for one of the options | ||
| flepied | +1 | Having dependencies at the test granularity plus metadata on the test makes it a more complete solution that is a superset of the 2 other propositions. Added bonus is to be able to re-use and collaborate with other Linux distros. | ||
| Roshi | +1 | |||
| Ausil | +1 | Ansible is a bit more work but I think will give better results and options | ||
| pingou | +1 | Ansible clearly has some down side but I do think it is simpler and can be more powerful than the RPM approach | ||
| Stef | +1 | Tests should be a core part of the distro, hence preference for packaging | ||
| Jenny | +1 | Ansible would be a dependency but it meets the needs better for configuring a system for the tests it is invoking. Packing tests into rpms is an added layer of complexity that is overhead and time consuming. Do not want to go down that path again. | ||
| jscotka | +0.5 | Bigger preference for packaging, because of combining upstream/downstream testing (need proper version of tests for installed packages). But I also like idea of ansible tooling. Not decided yet. There is also third possibility ( I hope it will be added asap) | ||
| miabbott | +1 | The ease of use and amount of available documentation for Ansible are some of it's strongest points for the proposal. Ansible should have better support for provisioning hosts for different kinds of tests. | ||
| alivigni | +1 | The ease of use and amount of available documentation for Ansible are some of it's strongest points for the proposal. Ansible is a general tool used inside and outside of Fedora and is constantly being enhanced with new features. It also allows a common way to drive testing that any CI tool can use. | ||
| dustymabe | +1 | I think ansible gives a balance of simple & sophisticated tooling to enable us to write simple tests or write complex tests. If a user is not familiar with ansible then they can use an example yaml file to just execute a shell script. More advanced users can ramp up to ansible's potential. | ||
| Nick Coghlan | +1 | I started this evaluation expecting to vote for the Ansible option, but changed my mind when I asked myself the question: "Given this approach, how hard would it to be to bootstrap the other?". Given that Fedora and its derivatives are inherently based on RPM, I think the winner on that basis is a packaging based approach, with a helper module and spec file boilerplate to bootstrap Ansible based test environments in libvirt and/or docker for components that can't run their integration tests non-intrusively on the host. This does imply some assumed capabilities for the bootstrapped Ansible environment (1. "Give me a matching local VM"; 2. "Give me a matching container"; 3. "Give me a matching OpenStack VM", 4. "Give me a matching Beaker machine"), but that would be the case even with the Ansible-as-the-baseline option. My rationale for preferring packaged tests to the autopkgtest/control file proposal is similar: given packaged tests as a baseline, I think that can be used to bootstrap control file based tests more easily than the other way around. | ||
| gdk | +1 | Reason one: easier to onboard new test authors because the Ansible approach will be far easier to learn. Reason two: multiple CI options will be easier to unlock down the road with a suite of Ansible tests ready to go. Reason three: you will have a ton of Ansible knowledge right at your fingertips. :) | ||
| contyk | +1 | The RPM approach has so many benefits I don't even know where to start. Every test package states exactly what it requires, along with the exact build it's testing; we get this information automatically and in most cases for free when building the package. Packages from large, standardized ecosystems such as Perl, Python, Ruby and similar make up a large part of our distribution; for most of those, generating the test subpackages could be almost entirely automated; their tests are in a known location, they're invoked in a known way and we have dependency generators for them. Some already have or had macros for exactly this purpose. Currently everyone (and I think this is unlikely to change, and not because of what tools we use) who would be involved in working on tests knows RPM packaging to at least some degree and wouldn't have to learn anything new. I'm sure the package maintainer would be more than willing to help additional interested contributors in extending the suite. Not running tests during the package build would actually simplify packaging as the package author doesn't need to list all the test build dependencies in the main package. The builds run faster since the buildroots are smaller and they don't block builders for the entire duration of the test suite. You could do that with the other two approaches as well but then you also need to modify the RPM package which is what you're trying to avoid. Also, the packager can switch from testing during build to async testing whenever they feel like it without the tests running twice or not at all at any point in time. It just feels natural to me. | ||
| tflink | +1 | I think that there are down-sides to all of these proposals but considering our constraints, I think that starting with ansible is the best option right now. | ||
| cevich | +1 | Proving something works or doesn't, requires eliminating variables. This is FAR easier if the equation starts out with a minimum set. Any tooling brought in at a low level, hauls in a non-trivial number of dependencies (and therefor variables). Testing the dependent packages is then ripe for circular-dependency problems which can be very difficult to spot. I don't know much about the 'control' file approach, but ansible is a HUGE hammer for a relatively small collection of nails (local commands). |