From Fedora Project Wiki

Disablement of STI tests

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

Drop support for Standard Test Interface (STI) tests in Fedora CI and Zuul CI for Fedora 43 and later.

For Fedora 43 and later:

  • STI tests will not be run

For EPEL:

  • keep running STI tests for EPEL dist-git pull requests until Fedora 43 branch date.
  • STI tests have not been included in Bodhi updates, and support for it will not be added.

For Fedora 42 and earlier:

  • keep running the STI tests in both dist-git and Bodhi updates.

All users of STI tests will need to migrate to the new tmt format during the Fedora 43 development.

The change will affect 281 components. The list of components affected can be found via this sourcegraph query.

All references to STI will be removed from the Fedora CI documentation.

Owner


Current status

Detailed Description

For some time CI testing in Fedora can be defined using two different formats:

  • Standard Test Interface (STI)
  • Test Management Tool (tmt)

As tmt has matured to a state it can fully replace the STI functionality, we are proposing to drop STI as the supported format. STI format is not actively developed and is causing us a maintenance burden. More and more STI test failures are starting to appear as the Fedora packages evolve out of sync with the test scripts.

Tmt provides various advantages over STI that would make it easier to manage in the long term:

  • Better organization of tests and test environments (referred to as plans in tmt)
  • Local reproducible environment thanks to testing-farm reproducer script, allowing to thinker locally without pushing to dist-git
  • Tests can be defined in the local dist-git repo, inside the srpm archive, tests/* dist-git namespace, or in the upstream repo, allowing it to be reused more freely between packages and with upstream directly
  • Tmt tests are integrated with packit allowing it to be executed with upstream and better migrate to the new dist-git environment

Hopefully this list can help inspire some better ways of reorganizing the tests during the migration.

Feedback

Benefit to Fedora

Having two formats for executing the tests is an unnecessary duplication and causes confusion for the Fedora maintainers and community.

STI tests have limited functionality and are harder to develop and maintain, when compared to tmt.

Scope


  • Other developers:
  • Policies and guidelines: N/A (not needed for this Change)
  • Trademark approval: N/A (not needed for this Change)
  • Alignment with the Fedora Strategy:

Upgrade/compatibility impact

This is a change to the development experience, no changes to Fedora distribution are made.

How To Migrate

To find out if you package has STI tests, check if tests/tests*.yml files are present in your dist-git repository.

Follow the sti to tmt migration guide. For the majority of tests involve just executing the already provided script you only need to:

1. Initialize tmt structure using tmt init

2. Create a minimal plan 

# /plan.fmf
# Boilerplate to indicate tmt to run any tests files it finds in the repo
discover:
  how: fmf
execute:
  how: tmt
# Prepare test environment, e.g. install packages
# Note: the current packages in the spec file will already be installed by default
prepare:
  how: install
  packages:
    - foo
    - bar

3. Migrate individual tests 

# /tests/foo.fmf
test: ./foo.sh

For more complex needs feel free to discuss in the #fedora-ci or #tmt matrix channels. You can also explore other project's tmt tests (identified by having a .fmf folder) using tmt tests show and tmt plans show

User Experience

For Fedora 43 and later, packagers who have setup gating on STI tests will have to migrate to be able to push their packages to stable repositories.

For Fedora 42, 41 and 40, STI tests will continue to run. The developers are encouraged to backport the STI test migration to the earlier branches, or remove them from those branches.

Dependencies

N/A

Contingency Plan

  • Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
  • Contingency deadline: N/A (not a System Wide Change)
  • Blocks release? No (not a System Wide Change)

We will keep the STI support until all references to STI tests are gone in case all users have not migrated to tmt until Fedora 43. See the description for the list of the affected components.

Documentation

Release Notes

Retrieved from "https://fedoraproject.org/w/index.php?title=Changes/DisableSTI&oldid=729164"