Introduction
In order to keep test execution efficient when number of test cases grows, it is crucial to maintain corresponding metadata, which define some aspects of how the test coverage is executed.
This document proposes a flexible format for defining metadata in plain text files which can be stored close to the test code and structured in a hiearchical way with support for inheritance.
Although the proposal initially originated from user stories centered around test execution, the format is general and thus can be used in broader scenarios, e.g. test coverage mapping.
Using this approach it's also possible to combine both test execution metadata and test coverage information. Thanks to elasticity and hiearchy it provides ability to organize data into well-sized text documents while preventing duplication.
Contact: Petr Šplíchal
Stones
These are essential corner stones for the design:
- Text files under version control
- Keep common uses cases simple
- Use hiearchy to organize content
- Prevent duplication where possible
- Metadata close to the test code
- Solution should be open source
- Focus on essential use cases
Stories
Important user stories to be covered:
- As a tester or developer I want to easy read and modify metadata and see history.
- As a tester I want to select a subset of test cases for execution by specifying a tag.
- As a tester I want to define a maximum time for a test case to run.
- As a tester I want to specify which environment is relevant for testing.
- As a user I want to easily define common metadata for multiple cases to simplify maintenance.
- As a user I want to provide specific metadata for selected tests to complement common metadata.
- As an individual tester and test contributor I want to execute specific single test case.
- As an automation tool I need a metadata storage with good api, extensible, quick for reading.
Choices
- Use git for version control and history of changes.
- Yaml format easily readable for both machines and humans.
Naming
A dedicated file name extension fmf as an
abbreviation of Flexible Metadata Format to easily find all
metadata files:
- main.fmf
- smoke.fmf
The format does not define attribute naming in any way. This is up
to individual projects. The only exception is the special name
main which is reserved for main directory index.
Attribute namespacing can be introduced as needed to prevent collisions between similar attributes. For example:
- test-description, requirement-description
- test:description, requirement:description
- test_description, requirement_description
Objects
Objects are identified by path from the git root directory.
Special file name main.fmf works similarly as
index.html:
| Location | Identifier |
|---|---|
| wget/main.fmf | wget |
| wget/download/main.fmf | wget/download |
| wget/download/smoke.fmf | wget/download/smoke |
Features
Let's demonstrate the features on a simple wget example with the following directory structure:
wget ├── download ├── protocols │ ├── ftp │ ├── http │ └── https ├── recursion └── smoke
Simple
The most common use cases super simple to read & write. Test metadata for a single test look like this:
description: Check basic download options tester: Petr Šplíchal <psplicha@redhat.com> tags: [Tier2, TierSecurity] test: runtest.sh time: 3 min
Hiearchy
Hiearchy is defined by directory structure (see example above) and
explicit nesting using attributes starting with /.
Defining metadata for several tests in a single file is
straightforward:
/download:
description: Check basic download options
tester: Petr Šplíchal <psplicha@redhat.com>
tags: [Tier2, TierSecurity]
test: runtest.sh
time: 3 min
/recursion:
description: Check recursive download options
tester: Petr Šplíchal <psplicha@redhat.com>
tags: [Tier2, TierSecurity]
test: runtest.sh
time: 20 min
Content above would be stored in wget/main.fmf file.
Inheritance
Metadata is inherited from parent objects:
tester: Petr Šplíchal <psplicha@redhat.com>
tags: [Tier2, TierSecurity]
test: runtest.sh
/download:
description: Check basic download options
time: 3 min
/recursion:
description: Check recursive download options
time: 20 min
This nicely prevents unnecessary duplication.
Elasticity
Use a single file or scatter metadata across the hiearchy, whatever is more desired for the project.
File wget/main.fmf:
tester: Petr Šplíchal <psplicha@redhat.com> tags: [Tier2, TierSecurity] test: runtest.sh
File wget/download/main.fmf:
description: Check basic download options time: 3 min
File: wget/recursion/main.fmf:
description: Check recursive download options time: 20 min
This allows reasonable structure for both small and large projects.
Scatter
Thanks to elasticity, metadata can be scattered across several
files. For example wget/download metadata can be
defined in the following three files:
File wget/main.fmf:
/download:
description: Check basic download options
test: runtest.sh
File wget/download.fmf:
description: Check basic download options test: runtest.sh
File wget/download/main.fmf:
description: Check basic download options test: runtest.sh
Parsing is done from top to bottom (in the order of examples above). Later/lower defined attributes replace values defined earlier/higher in the structure.
Leaves
When searching, key content is used to define which leaves
from the metadata tree will be selected. For example, every test
case to be executed must have the test attribute
defined, every requirement to be considered for test coverage
evaluation must have the requirement attribute
defined. Otherwise object data is used for inheritance only.
description: Check basic download options test: runtest.sh time: 3 min
The key content attributes are not supposed to be hard-coded in the Flexible Metadata Format but freely configurable. Multiple key content attributes (e.g. script & backend) could be used as well.
Virtual
Using a single test code for testing multiple scenarios can be easily implemented using leaves inheriting from the same parent:
description: Check basic download options
test: runtest.sh
/fast:
description: Check basic download options (quick smoke test)
environment: MODE=fast
tags: [Tier1]
time: 1 min
/full:
description: Check basic download options (full test set)
environment: MODE=full
tags: [Tier2]
time: 3 min
In this way we can efficiently create virtual test cases.
Examples
Relevancy
Test Case Relevancy can be naturaly integrated:
description: Check basic download options tags: [Tier2, TierSecurity] relevancy: - "distro < rhel-7: False" - "arch = s390x: False"
Note that, because of YAML parsing, relevancy rules have to be enclosed in quotes. Another option is to use text format:
description: Check basic download options
tags: [Tier2, TierSecurity]
relevancy: |
distro < rhel-7: False
arch = s390x: False
Which seems a bit more clear.
Coverage
Test coverage information can be stored in a single file, for
example wget/requirements.fmf:
/protocols:
priority: high
/ftp:
requirement: Download a file using the ftp protocol.
coverage: wget/protocols/ftp
/http:
requirement: Download a file using the http protocol.
coverage: wget/protocols/http
/https:
requirement: Download a file using the https protocol.
coverage: wget/protocols/https
/download:
priority: medium
/output-document-pipe:
requirement: Save content to pipe.
coverage: wget/download
/output-document-file:
requirement: Save content to a file.
coverage: wget/download
/upload:
priority: medium
/post-file:
requirement: Upload a file to the server
coverage: wget/protocols/http
/post-data:
requirement: Upload a string to the server
coverage: wget/protocols/http
Or split by functionality area into separate files as desired, for
example wget/download/requirements.fmf:
priority: medium
/output-document-pipe:
requirement: Save content to pipe.
coverage: wget/download
/output-document-file:
requirement: Save content to a file.
coverage: wget/download
Or integrated with test case metadata, e.g.
wget/download/main.fmf:
description: Check basic download options
tags: [Tier2, TierSecurity]
test: runtest.sh
time: 3 min
/requirements
requirement: Various download options working correctly
priority: low
/get-file:
coverage: wget/download
/output-document:
coverage: wget/download
/continue:
/timestamping:
/tries:
/no-clobber:
coverage: wget/download
/progress:
/quota:
/server-response:
/bind-address:
/spider:
In the example above three requirements are already covered, the rest still await for test coverage (attributes value is null).
Strategist
Here's an example implementation of test-strategist data for openscap using the Flexible Metadata Format:
/probes:
description: Probes
/offline:
description: Offline scanning
/online:
description: Online scanning
/scanning:
description: Reading and understanding source datastreams
/oval:
influencers:
- openscap/probes/offline
- openscap/probes/online
/ds:
influencers:
- openscap/scanning/oval
- openscap/scanning/cpe
/cpe:
influencers:
- openscap/scanning/oval
Implementation
A Python module and a simple command line tool implementing the features described in this draft are available for experimenting: