Changes/InvokingTestsAnsible

Ansible: Standard Discovery, Staging, Invocation of Integration Tests

This is a proposal
Feedback is more than welcome. There's a discussion tab above.

First see the Terminogy division of Responsibilities and Requirements

Detailed Description

This standard interface describes how to discover, stage and invoke tests. It is important to cleanly separate implementation details of the testing system from the test suite and its framework. It is also important to allow Fedora packagers to locally and manually invoke a test suite.

Staging

Tests files will be added into the tests folder of a dist-git repository branch. The structure of the files and folders is left to the liberty of the packagers but there should be a run_tests.yml playbook at the top level of the tests folder to set up and run all the tests.

1. The testing system should stage the tests on Fedora operating system appropriate for the branch name of the dist-git repository containing the tests.
2. The testing system must have installed the following packages:
   1. ansible python2-dnf libselinux-python
3. The testing system clones the dist-git repository for the test, and checks out the appropriate branch.
4. The contents of /etc/yum.repos.d is replaced with repository information that reflects the known good Fedora packages the test may require.
   1. The testing system may use multiple repositories, including updates or updates-testing to ensure this.

Invocation

The test can be invoke simply by calling ansible-playbook run_tests.yml on the run_tests.yml playbook of interest.

To invoke the test suite, one would:

1. Create a folder and place the test subject(s) in that folder.
2. Set an enivonment variable $TEST_SUBJECT_FOLDER to the path of the folder.
3. Execute the following command ansible-playbook run-tests.yml
   1. The command is invoked as root, and the roles/modules may drop privileges as desired
   2. Examine the exit code of each ansible-playbook command. Zero exit code is a successful test result, non-zero is failure.
   3. The ansible playbook must put the output of the tests into a file named output in the current directory.
4. Tests can put any additional test artifacts like screenshots, core dumps, into the current directory.

Discovery

A testing system needs to be able to efficiently answer the question "does this subject have any tests packages, and if so, what are their names". This should be automatically discoverable to the extent possible.

Use repoquery, basically I propose we rely on the dependency chain of the RPMs itself instead of trying to replicate it differently.

repoquery --whatrequires or an equivalent relying on mdapi: https://apps.fedoraproject.org/mdapi/ (which I need to adjust to support back walking (ie find which packages requires "foo" instead of what packages "foo" requires which we currently have) and we should be able to build a list of dependencies.

In addition, a test suite can be uniquely identified using the git hash of the commit of the git repo.

Scope

Since the tests are added in a sub-folder of the dist-git repo, there are no changes required to the Fedora infrastructure and will have no impact on the packagers' workflow and tooling.

Only the testing system will need to be taught to install the requirements and run the playbooks.

User Experience

A standard way to package, store and run tests benefits Fedora stability, and makes Fedora better for users.

• This structure makes it easy to run locally thus potentially reproducing an error triggered on the test system.
• Ansible is being more and more popular, thus making it easier for people to contribute new tests
• Used by a lot of sys-admin, ansible could help sys-admin to bring test-cases to the packagers and developers about situation where something failed for them.

Upgrade/compatibility impact

There are no real upgrade or compatibility impact. The tests will be branched per release as spec files are branched dist-git is now.

Examples

What follows are examples of writing and/or packaging existing tests to this standard.

TODO: Put general example notes here.

Example: Simple in-situ test

A simple downstream integration test for gzip can be found at: https://pagure.io/ansible_based_tests/blob/master/f/tests/gzip

This is how the folder structure looks like:

 .
 ├── files
 │   └── test-simple
 └── run_tests.yml

And the content of run_tests.yml is:

---
- hosts: localhost
  remote_user: root
  tasks:
  - name: Install the requirements
    package: 
      name: "{{item}}" 
      state: latest
    with_items:
      - coreutils
      - /sbin/install-info
      - gzip
 
  - name: Create the folder where we will store the tests
    file: 
      state: directory
      path: "{{ item }}"
      owner: root 
      group: root
    with_items:
     - /usr/libexec/tests/gzip/

  - name: Install the test files
    copy: 
      src: "{{ item.file }}"
      dest: "/usr/libexec/tests/gzip/{{ item.dest }}"
      mode: 0755
    with_items:
     - {file: test-simple, dest: test-simple }

  - name: Execute the tests
    shell: /usr/libexec/tests/gzip/test-simple 2>&1 | tee output

Example: GNOME style "Installed Tests"

A downstream integration test running in gnome installed tests can be found at: https://pagure.io/ansible_based_tests/blob/master/f/tests/gzip full example structure: https://pagure.io/ansible_based_tests/blob/master/f/tests/gzip/playbooks

Example: Tests run in Docker Container

An integration test running tests in a docker container can be found at: https://pagure.io/ansible_based_tests/blob/master/f/tests/glib2 full example structure: https://pagure.io/ansible_based_tests/blob/master/f/tests/glib2/playbooks

Example: Modularity testing Framework

TODO: Port an example

Example: Ansible with Atomic Host

TODO: Port an existing test

Example: Beakerlib based test

TODO: Port and shim a beakerlib test

Example: Full Structure

 .
 └── tests
    └── test-case
    └── config
    └── playbooks
        └── group_vars
        └── roles
        │   └── configure
        │   │   └── defaults
        │   │   └── files
        │   │   └── handlers
        │   │   └── meta
        │   │   └── tasks
        │   │   └── templates
        │   │   └── vars
        │   └── run_tests
        │   │   └── defaults
        │   │   └── files
        │   │   └── handlers
        │   │   └── meta
        │   │   └── tasks
        │   │   └── templates
        │   │   └── vars
        └── configure.yml
        └── run_tests.yml

Tests will live under tests directory in a dist-git repo. The playbooks directory will define the roles for configuration and execution of the tests. The run_tests.yml will call roles necessary and dependencies of other roles can be defined there or in the meta of another role. (Well documented on writing ansible playbooks) I put the config as a place holder for configuration files needed or for provisioning (thinking of linch-pin https://github.com/CentOS-PaaS-SIG/linch-pin) Note :This does not mean all these role sub-directories are required this just shows a full example case

Evaluation

Instructions: Copy the block below, sign your name and fill in each section with your evaluation of that aspect. Add additional bullet points with overall summary or notes.

Full Name -- SignAture

• Summary: ...
• Staging: ...
• Invocation: ...
• Discovery: ...

Stef Walter -- Stefw

• Summary:
  • PRO: Ansible is readable and approachable
  • PRO: Tests are stored in same repo as tests
  • PRO: Inclusion of upstream tests seems to require packaging them as RPMs.
  • CON: Ansible is another technology (in addition to RPM spec files, etc.) that packager is required to learn in order to maintain a package in dist-git.
  • CON: If tests become a core Fedora concept (which we hope), Ansible becomes a core technology that Fedora requires and is built upon.
  • CON: Most Ansible modules require Python 2.x while the distro is trying to move to Python 3.x
  • CON: No standard mechanism for passing a test subject to a test suite implementing the standard test interface
  • CON: No standard mechanism for reporting test log, or test artifacts from standard interface
  • CON: No way to describe whether tests are compatible with or conflict with specific NVR of test subjects.
• Staging:
  • No mechanism for passing a test subject (eg: a built package, a module, or a container) to the test suite to operate on.
  • No guidance on what Ansible modules should be used to install test dependencies
  • No mechanism for a test system to control which repo of known-good packages to pull test or test suite dependencies from.
  • Requires sudo, dnf, git, ansible, python2-dnf, libselinux-python as well known staging dependencies
• Invocation:
  • Seems that zero exit code from sudo means success, non-zero exit code means failure? Not described explicitly in standard.
  • The use of sudo seems to imply invocation should happen as a non-root user. Is this correct?
  • Does the standard assume sudo is guaranteed to work? Should the sudo part just be dropped and require invocation as root?
  • No mechanism for reporting logs, or test artifacts has been described.
• Discovery:
  • Mechanism is simple, but no concrete description of how exactly this works. How does a testing system find tests given a test subject such as an RPM or NVR?
  • MDAPI link is broken: https://apps.fedoraproject.org/mdapi/
    • This has been fixed --Pingou (talk) 08:03, 12 April 2017 (UTC)

Martin Pitt -- mpitt

• Summary:
  • I agree to what Stef said above, so I just add my "delta" review.
  • PRO: I prefer keeping tests in the sources (like in this proposal) over packaging tests, as it's much less overhead for the packager and avoids having to create a new kind of package archive.
  • CON: My main concern is that the Ansible format/tool might be replaced with something else in a few years, but the test format should be stable for a long time to avoid having to port hundreds/thousands of tests.
  • CON: The ansible format is relatively verbose and too procedural for my taste; I prefer a purely declarative syntax and avoiding boilerplate for installing test deps and invoking the tests.
• Staging:
  • Not supporting test subjects is a major gap in the prototype - this is one of the core requirements here!
  • Installing the actual tests is unnecessary overhead in the playbook, and clutters the host system with files in /usr that don't belong to a package; this can be rectified though with dropping the "Create folder"/"Install" tasks and replacing the run part with

- name: Execute the tests
  script: files/test-simple

• Invocation:
  • Getting live logs from the test and also saving it as an artifact is crucial, this is a major gap in the prototype. Can ansible do this somehow?
• Discovery:
  • Checking out and inspecting hundreds/thousands of dist-gits whether they contain tests does not meet "able to efficiently answer the question..."; this needs a new service which regularly indexes all dist-gits and creates list of source packages that have tests.

Pierre-Yves Chibon -- pingou

• Disclaimer: I am one of the owners above.
• Summary:
  • PRO: Ansible is a well-know technology for sys-admin making it easier for them to contribute tests
  • CON: While being well-know for some people, it will be new for others
  • PRO: Very flexible it gives the packagers all the flexibility to install/configure/run their tests as they wish
  • PRO: We could use --tag to allow running just a part of the test suite at certain time (-t PR to run on pull-request -t updates to run on bodhi updates...)
  • CON: We may need to "regulate" the flexibility to suggest a set of standard/gold practices to be used in the test system (using different tags or playbook if we want)
• Staging:
  • PRO: its flexibility makes it easy to test anything
  • CON: we will need to write policies/guidelines on how to test the different subject (RPM, container, images...)
• Invocation:
  • PRO: easy to run locally
  • PRO: easy to run as root and switch to a local user or vice-versa
  • PRO: easy to couple with something like vagrant to allow running locally destructive tests
  • CON: May require policy to set expectations and document how to move from one to the other
  • CON: Inter-package dependencies is a challenge that can be overcome with a custom ansible module allowing to git clone other dist-git repo and while allowing us to block other network accesses (to avoid downloading random things from the internet that may be gone tomorrow and thus kill the reproducibility aspect).
• Discovery:
  • Git hash uniquely identifies a test suite
    • Meaning the identifier may change while the test suite itself hasn't
  • PRO: Relies on the same dependency chain as the artefacts themselves
  • QUESTION: What is the aim here? Do we really want to run all the tests of every perl module for every change made to the perl package? If so, good luck, otherwise repoquery --whatrequires <pkg> should do what we want.
    • MartinPitt: That's what Debian/Ubuntu do, and indeed that triggers thousands of tests (times 5 architectures). This allows landing new Perl versions with confidence and points out modules that need to be adjusted (and believe me, pretty much every new Perl version breaks some module or two!). That said, it should be possible to discover tests for that reason - I don't expect our infra to be scalable and fast enough right from the start to actually do testing at that depth.

Tim Flink -- Tflink

• Disclaimer: I am one of the owners of this proposal
• Summary:
  • PRO: Storing tests in this way decouples them from the build process
  • PRO: Ansible has better docs and more examples than Fedora packages or RPM do
  • PRO: non-packager testers don't have to learn RPM syntax
  • PRO: Able to provide a lot more in the way of convenience functions to the test author - galaxy, roles/modules that we provide
  • PRO: easy to change tests during devel, does not require a dedicated path in the filesystem
  • PRO/CON: More easily extendable
  • CON: Adds ansible et. al as a dependency for the test process - what happens if ansible changes or if it becomes unattractive 5-10 years from now?
  • CON: Adds additional thing that packagers have to learn
  • CON: We would have no control over when/how ansible changes
  • It's not incredibly clear what all would be distributed (ansible modules, plugins) or how those would be distributed (galaxy-ish, package, etc.)
• Staging:
  • There is no obvious way to say what NVR is under test other than looking at what's installed or what's locally available pre-build
• Invocation:
  • Not sure sudo is required, it would likely be easier to have a plugin (if required) that ran things in a temp dir kind of how we do with libtaskotron today
• Discovery:
  • While arguably more complex than the -tests package proposal, the additional complexity in terms of code to be written doesn't seem to be much more complex
  • There are systems already doing some parts of this discovery and could likely be re-used to a certain extent (Taskotron's trigger)

Dennis Gilmore -- Ausil

• Summary:
  • PRO: we could have unique git repos for collections, gnome-desktop, KDE, Atomic Host, Server, etc
  • PRO: Docs are good as is support ofr the format across platforms
  • PRO: Branching could be separate from package branching, simplifying workflows
    • I believe the idea is to store the tests in dist-git next to the spec files and patches, so branching would be at the same time --Pingou (talk) 08:03, 12 April 2017 (UTC)
  • PRO: should be simple to write validation testing of tests, making sure that people are in compliance.
  • CON: Not clear how we should store tests for same package with different git namespaces. for example Cockpit rpm and cockpit container
    • If they are stored in dist-git the tests for the rpm would be stored next to the spec file and the tests for the container next to the Dockerfile or equivalent --Pingou (talk) 08:03, 12 April 2017 (UTC)
  • CON: getting started with Ansible for those who do not now it is a steep learning curve
  • CON: can not reuse tools like rpmlint, rpmdiff etc
    • Could you expand on why? I don't see anything preventing using these tools. --Pingou (talk) 08:03, 12 April 2017 (UTC)
  • PRO: seems like we should be able to easily setup a template for a tests repo
  • PRO: We should be able to easily put a web interface for adding and editing tests for people not familiar with git
• Staging:
  • Using VM's and containers seems to have a much clearer path than the -tests package proposal
• Invocation:
  • use of sudo seems very suboptimal.
• Discovery:
  • indexing, searching and mapping of tests seems uncovered. Likely we will need to write some tooling to make it useful and easy to find and get for people.

Micah Abbott -- miabbott

• Summary:
  • Disclaimer: I was pulled into this evaluation later in the game and may be missing some context/pieces of the larger effort.
  • PRO: Ansible feels easier to read/understand/learn
  • PRO: Ansible appears to give more flexibility and options to packagers
  • CON: New requirement on Ansible; not a standard install option like rpm/yum/dnf
  • CON: Easy to do bad things with Ansible + root user
• Staging:
  • Using Ansible here seems to better support the in-situ and outside-in test approaches. There may still be the issue of multiple, conflicting provisioning solutions.
• Invocation:
  • Using root has risks, although widely used when running Ansible playbooks.
• Discovery:
  • Using reqoquery seems reasonable enough, although I'd like to see a more concrete example of the whole process.