|
|
| (5 intermediate revisions by one other user not shown) |
| Line 1: |
Line 1: |
| = Introduction = | | = Introduction = |
|
| |
|
| Handle test metadata in an efficient way.
| | In order to keep test execution efficient when number of test |
| | cases grows, it is crucial to maintain corresponding metadata, |
| | which define some aspects of how the test coverage is executed. |
| | |
| | This concept proposes a flexible format for defining metadata in |
| | plain text files which can be stored close to the test code and |
| | structured in a hiearchical way with support for inheritance. |
|
| |
|
| '''Contact:''' [[User:Psss|Petr Šplíchal]] | | '''Contact:''' [[User:Psss|Petr Šplíchal]] |
|
| |
| == Stones ==
| |
|
| |
| These are essential corner stones for the design:
| |
|
| |
| * Text files under version control
| |
| * Keep common uses cases simple
| |
| * Use hiearchy to organize content
| |
| * Prevent duplication where possible
| |
| * Metadata close to the test code
| |
| * Solution should be open source
| |
| * Focus on essential use cases
| |
|
| |
| == Stories ==
| |
|
| |
| Important user stories to be covered:
| |
|
| |
| * As a tester or developer I want to easy read and modify metadata and see history.
| |
| * As a tester I want to select a subset of test cases for execution by specifying a tag.
| |
| * As a tester I want to specify which environment is relevant for testing.
| |
| * As a user I want to easily define common metadata for multiple cases to simplify maintenance.
| |
| * As a user I want to provide specific metadata for selected tests to complement common metadata.
| |
| * As an individual tester and test contributor I want to execute specific single test case.
| |
| * As an automation tool I need a metadata storage with good api, extensible, quick for reading.
| |
|
| |
| == Choices ==
| |
|
| |
| * Use git for version control and history of changes.
| |
| * Yaml format easily readable for both machines and humans.
| |
|
| |
| == Naming ==
| |
|
| |
| A dedicated file name extension <code>fmf</code> as an
| |
| abbreviation of Flexible Metadata Format to easily find all
| |
| metadata files:
| |
|
| |
| * main.fmf
| |
| * smoke.fmf
| |
|
| |
| The format does not define attribute naming in any way. This is up
| |
| to individual projects. The only exception is the special name
| |
| <code>main</code> which is reserved for main directory index.
| |
|
| |
| Attribute namespacing can be introduced as needed to prevent
| |
| collisions between similar attributes. For example:
| |
|
| |
| * test-description, requirement-description
| |
| * test:description, requirement:description
| |
| * test_description, requirement_description
| |
|
| |
| == Objects ==
| |
|
| |
| Objects are identified by path from the git root directory.
| |
| Special file name <code>main.fmf</code> works similarly as
| |
| <code>index.html</code>:
| |
|
| |
| {| class="wikitable"
| |
| !Location
| |
| !Identifier
| |
| |-
| |
| |wget/main.fmf
| |
| |wget
| |
| |-
| |
| |wget/download/main.fmf
| |
| |wget/download
| |
| |-
| |
| |wget/download/smoke.fmf
| |
| |wget/download/smoke
| |
| |}
| |
|
| |
|
| = Features = | | = Features = |
|
| |
|
| Let's demonstrate the features on a simple wget example with the
| | Here are some of the key features of the Flexible Metadata Format: |
| following directory structure:
| |
| | |
| wget
| |
| ├── download
| |
| ├── protocols
| |
| │ ├── ftp
| |
| │ ├── http
| |
| │ └── https
| |
| ├── recursion
| |
| └── smoke
| |
| | |
| == Simple ==
| |
| | |
| The most common use cases super simple to read & write. Test
| |
| metadata for a single test look like this:
| |
| | |
| description: Check basic download options
| |
| tester: Petr Šplíchal <psplicha@redhat.com>
| |
| tags: [Tier2, TierSecurity]
| |
| test: runtest.sh
| |
| time: 3 min
| |
| | |
| == Hiearchy ==
| |
| | |
| Hiearchy is defined by directory structure (see example above) and
| |
| explicit nesting using attributes starting with <code>/</code>.
| |
| Defining metadata for several tests in a single file is
| |
| straightforward:
| |
| | |
| /download:
| |
| description: Check basic download options
| |
| tester: Petr Šplíchal <psplicha@redhat.com>
| |
| tags: [Tier2, TierSecurity]
| |
| test: runtest.sh
| |
| time: 3 min
| |
| /recursion:
| |
| description: Check recursive download options
| |
| tester: Petr Šplíchal <psplicha@redhat.com>
| |
| tags: [Tier2, TierSecurity]
| |
| test: runtest.sh
| |
| time: 20 min
| |
| | |
| Content above would be stored in <code>wget/main.fmf</code> file.
| |
| | |
| == Inheritance ==
| |
| | |
| Metadata is inherited from parent objects:
| |
| | |
| tester: Petr Šplíchal <psplicha@redhat.com>
| |
| tags: [Tier2, TierSecurity]
| |
| test: runtest.sh
| |
|
| |
| /download:
| |
| description: Check basic download options
| |
| time: 3 min
| |
| /recursion:
| |
| description: Check recursive download options
| |
| time: 20 min
| |
| | |
| This nicely prevents unnecessary duplication.
| |
| | |
| == Elasticity ==
| |
| | |
| Use a single file or scatter metadata across the hiearchy,
| |
| whatever is more desired for the project.
| |
| | |
| File <code>wget/main.fmf</code>:
| |
| | |
| tester: Petr Šplíchal <psplicha@redhat.com>
| |
| tags: [Tier2, TierSecurity]
| |
| test: runtest.sh
| |
| | |
| File <code>wget/download/main.fmf</code>:
| |
| | |
| description: Check basic download options
| |
| time: 3 min
| |
| | |
| File: <code>wget/recursion/main.fmf</code>:
| |
| | |
| description: Check recursive download options
| |
| time: 20 min
| |
| | |
| This allows reasonable structure for both small and large
| |
| projects.
| |
| | |
| == Scatter ==
| |
| | |
| Thanks to elasticity, metadata can be scattered across several
| |
| files. For example <code>wget/download</code> metadata can be
| |
| defined in the following three files:
| |
| | |
| File <code>wget/main.fmf</code>:
| |
| | |
| /download:
| |
| description: Check basic download options
| |
| test: runtest.sh
| |
| | |
| File <code>wget/download.fmf</code>:
| |
| | |
| description: Check basic download options
| |
| test: runtest.sh
| |
| | |
| File <code>wget/download/main.fmf</code>:
| |
| | |
| description: Check basic download options
| |
| test: runtest.sh
| |
| | |
| Parsing is done from top to bottom (in the order of examples
| |
| above). Later/lower defined attributes replace values defined
| |
| earlier/higher in the structure.
| |
| | |
| == Leaves ==
| |
| | |
| When searching, '''key content''' is used to define which leaves
| |
| from the metadata tree will be selected. For example, every test
| |
| case to be executed must have the <code>test</code> attribute
| |
| defined, every requirement to be considered for test coverage
| |
| evaluation must have the <code>requirement</code> attribute
| |
| defined. Otherwise object data is used for inheritance only.
| |
| | |
| description: Check basic download options
| |
| test: runtest.sh
| |
| time: 3 min
| |
| | |
| The key content attributes are not supposed to be hard-coded in
| |
| the Flexible Metadata Format but freely configurable. Multiple key | |
| content attributes (e.g. script & backend) could be used as well.
| |
| | |
| == Virtual ==
| |
| | |
| Using a single test code for testing multiple scenarios can be
| |
| easily implemented using leaves inheriting from the same parent:
| |
| | |
| description: Check basic download options
| |
| test: runtest.sh
| |
|
| |
| /fast:
| |
| description: Check basic download options (quick smoke test)
| |
| environment: MODE=fast
| |
| tags: [Tier1]
| |
| time: 1 min
| |
| /full:
| |
| description: Check basic download options (full test set)
| |
| environment: MODE=full
| |
| tags: [Tier2]
| |
| time: 3 min
| |
| | |
| In this way we can efficiently create virtual test cases.
| |
| | |
| == API ==
| |
| | |
| A python module is planned to be implemented which would take care
| |
| of parsing and merging the metadata files and providing an easy
| |
| API for automation scripts. A corresponding simple command line
| |
| tool should allow to easily investigate metadata and could be used
| |
| for example for adhoc test execution.
| |
| | |
| fmf --key test --search tag=Tier1
| |
| fmf --key requirement --search priority=high
| |
| | |
| | |
| = Examples =
| |
| | |
| == Relevancy ==
| |
| | |
| Test Case Relevancy can be naturaly integrated:
| |
| | |
| description: Check basic download options
| |
| tags: [Tier2, TierSecurity]
| |
| relevancy:
| |
| - "distro < rhel-7: False"
| |
| - "arch = s390x: False"
| |
| | |
| Note that, because of YAML parsing, relevancy rules have to be
| |
| enclosed in quotes. Another option is to use text format:
| |
| | |
| description: Check basic download options
| |
| tags: [Tier2, TierSecurity]
| |
| relevancy: |
| |
| distro < rhel-7: False
| |
| arch = s390x: False
| |
| | |
| Which seems a bit more clear.
| |
| | |
| == Coverage ==
| |
| | |
| Test coverage information can be stored in a single file, for
| |
| example <code>wget/requirements.fmf</code>:
| |
| | |
| /protocols:
| |
| priority: high
| |
| /ftp:
| |
| requirement: Download a file using the ftp protocol.
| |
| coverage: wget/protocols/ftp
| |
| /http:
| |
| requirement: Download a file using the http protocol.
| |
| coverage: wget/protocols/http
| |
| /https:
| |
| requirement: Download a file using the https protocol.
| |
| coverage: wget/protocols/https
| |
|
| |
| /download:
| |
| priority: medium
| |
| /output-document-pipe:
| |
| requirement: Save content to pipe.
| |
| coverage: wget/download
| |
| /output-document-file:
| |
| requirement: Save content to a file.
| |
| coverage: wget/download
| |
|
| |
| /upload:
| |
| priority: medium
| |
| /post-file:
| |
| requirement: Upload a file to the server
| |
| coverage: wget/protocols/http
| |
| /post-data:
| |
| requirement: Upload a string to the server
| |
| coverage: wget/protocols/http
| |
| | |
| Or split by functionality area into separate files as desired, for
| |
| example <code>wget/download/requirements.fmf</code>:
| |
|
| |
|
| priority: medium
| | * [https://fmf.readthedocs.io/en/latest/features.html#simple Simple] ... common use cases super simple to read & write |
| /output-document-pipe:
| | * [https://fmf.readthedocs.io/en/latest/features.html#hierarchy Hierarchy] ... metadata structured in a hierarchy |
| requirement: Save content to pipe.
| | * [https://fmf.readthedocs.io/en/latest/features.html#inheritance Inheritance] ... data inheritance prevent duplication |
| coverage: wget/download
| | * [https://fmf.readthedocs.io/en/latest/features.html#elasticity Elasticity] ... single file or data scattered across hierarchy |
| /output-document-file:
| | * [https://fmf.readthedocs.io/en/latest/features.html#virtual Virtual] ... support for maintaining virtual test cases |
| requirement: Save content to a file.
| |
| coverage: wget/download
| |
|
| |
|
| Or integrated with test case metadata, e.g.
| | = Specification = |
| <code>wget/download/main.fmf</code>:
| |
|
| |
|
| description: Check basic download options
| | We are currently working on a Metadata specification which |
| tags: [Tier2, TierSecurity]
| | describes in detail individual attributes to be used for selecting |
| test: runtest.sh
| | and executing tests in the Fedora CI (L1 metadata) and additional |
| time: 3 min
| | information which is necessary for executing multiple test cases |
|
| | (L2 metadata): |
| /requirements
| |
| requirement: Various download options working correctly
| |
| priority: low
| |
| /get-file:
| |
| coverage: wget/download
| |
| /output-document:
| |
| coverage: wget/download
| |
| /continue:
| |
| /timestamping:
| |
| /tries:
| |
| /no-clobber:
| |
| coverage: wget/download
| |
| /progress:
| |
| /quota:
| |
| /server-response:
| |
| /bind-address:
| |
| /spider:
| |
|
| |
|
| In the example above three requirements are already covered,
| | * Metadata: https://pagure.io/fedora-ci/metadata |
| the rest still await for test coverage (attributes value is null).
| |
|
| |
|
| == Strategist == | | = Links = |
|
| |
|
| Here's an example implementation of
| | A Python module and a simple command line tool are available for |
| [https://github.com/dahaic/test-strategist test-strategist] data for
| | experimenting. Detailed documentation is included as well: |
| openscap using the Flexible Metadata Format:
| |
|
| |
|
| /probes:
| | * GitHub: https://github.com/psss/fmf |
| description: Probes
| | * Docs: https://fmf.readthedocs.io/ |
| /offline:
| | * Fedora: https://apps.fedoraproject.org/packages/fmf/ |
| description: Offline scanning
| | * Copr: https://copr.fedorainfracloud.org/coprs/psss/fmf/ |
| /online:
| | * PIP: https://pypi.python.org/pypi/fmf |
| description: Online scanning
| |
| /scanning:
| |
| description: Reading and understanding source datastreams
| |
| /oval:
| |
| influencers:
| |
| - openscap/probes/offline
| |
| - openscap/probes/online
| |
| /ds:
| |
| influencers:
| |
| - openscap/scanning/oval
| |
| - openscap/scanning/cpe
| |
| /cpe:
| |
| influencers:
| |
| - openscap/scanning/oval
| |
