CI/Metadata/Examples

SELinux

Here's an example of how Flexible Metadata Format can simplify test execution when number of tests grows. We would like to execute all available tests relevant to the libselinux component plus all Tier1 selinux tests.

List

The old way is the provide a full list of tests to be executed:

- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    repositories:
    - repo: "https://src.fedoraproject.org/tests/selinux.git"
      dest: "selinux"
    tests:
    - selinux/policycoreutils/restorecon
    - selinux/libselinux/realpath_not_final-function
    - selinux/libselinux/selinux_boolean_sub-function
    - selinux/libselinux/selinux_restorecon-functions
    - selinux/libselinux/setenforce
    - selinux/libselinux/getsebool
    - selinux/libselinux/selabel-functions
    - selinux/libselinux/selinux_restorecon
    - selinux/libselinux/selinux_sestatus-functions
    - selinux/libselinux/selinux_set_callback
    - selinux/checkpolicy/checkpolicy
    - selinux/libsepol/sepol_check_context
    - selinux/libsemanage/semanage-seuser-functions

This list would have to be kept up-to-date after each test addition.

Command

Another slightly better option would be to provide an easy way how to generate the list of tests to be executed, for example directly with the fmf command line tool:

- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    repositories:
    - repo: "https://src.fedoraproject.org/tests/selinux.git"
      dest: "selinux"
    tests_command: "fmf selinux --brief --key test --filter 'component:libselinux | tags:Tier1'"

Test name does not always have to map to the directory where the test is stored so the following command would have to be used to make it more general:

fmf --key test --format "{}/{}\n" --value "root" --value "data.get(path) or name" --filter 'component:libselinux | tags:Tier1'

Which seems to be a bit complicated.

Integration

The best solution probably will be to integrate Flexible Metadata Format directly in the Standard Test Roles. The tests.yaml file could then be much more readable:

- hosts: localhost
  roles:
  - role: standard-test-beakerlib
    tags:
    - classic
    repositories:
    - repo: "https://src.fedoraproject.org/tests/selinux.git"
      dest: "selinux"
    test_directories:
       - "selinux"
    test_filters:
       - "component:libselinux | tags:Tier1"

Standard Test Roles would then internally call fmf to filter relevant tests and directly executed them. Plus it would be possible to integrate Test Case Relevancy to offer filtering based on the particular environment.