Wikitcms

Wikitcms is a term used to refer to Fedora's use of this MediaWiki instance as an ad hoc test management system ('TCMS' stands for 'Test Case Management System', but that precise initialism is in fact quite rarely used). It is also the name of a Python module which provides an interface to the 'system'.

Background

The use of the Fedora wiki to track test results dates back to at least the page Releases/5/FC5FinalTreeTesting - at that time, the Fedora wiki was a MoinMoin instance, not this Mediawiki instance. The rough form of a table with different tests as the rows and results as columns is visible even there. Since then, this basic format has been gradually elaborated.

Detailed history

• Separate test case pages were introduced in QA/TestResults/Fedora9Install/Alpha
• Live CD testing was introduced in QA/TestResults/Fedora9LiveCD/FinalRelease (and dropped after Fedora 10, until the introduction of the Desktop test type with Fedora 13)
• The first test result templates - Template:Testresult/fail, Template:Testresult/pass, and Template:Testresult/warn - were introduced in QA/TestResults/Fedora10Install/Final
• Result columns per architecture also first appeared in QA/TestResults/Fedora10Install/Final
• The first effort at associating bug reports with results was in QA:Fedora_11_Alpha_Install_Test_Results
• The first 'template' page (then used for manual copy/paste templating) for results pages was QA:Fedora_11_Install_Results_Template
• Separation of results pages by compose first occurred in Fedora 11, with TCs first appearing with Fedora 12
• The Test Results namespace (allowing result submission without a FAS account) was first used for Fedora 12
• The current Template:Result result template first appeared in Test_Results:Fedora_13_Alpha_TC1_Install
• The first of the current 'test types' other than Installation - Desktop - appeared in Test_Results:Fedora_13_Alpha_TC_Desktop (this also marked the introduction of result 'environments' other than architecture)
• The TC/RC result page naming scheme was more or less settled as of Fedora 13
• The Current (testtype) Results redirect pages first appeared with Fedora 13
• The Installation page was first split into sections in Test_Results:Fedora_14_Alpha_TC1_Install
• The Base and Security Lab test types appeared in Fedora 16 Alpha
• The first 'two-dimensional' result table (where the rows each represent the same test case in a different context, effectively using both rows and columns to represent environments) appeared in Test_Results:Fedora_20_Alpha_TC1_Install#ARM_disk_images
• The first (manual) result pages for nightly builds appeared in Fedora 21's pre-Alpha phase
• The Cloud and Server test types first appeared in Fedora 21 Alpha
• The template system for generating result pages, the CurrentFedoraCompose template, the wikitcms module, and the relval utility for generating result pages and reporting results were all introduced during the Fedora 21 cycle
• Individual nightly compose result pages and automatic generation of nightly result pages were introduced in Fedora 22
• The page naming scheme was changed slightly, nightly events interspersed with 'candidate' compose events appeared, and automated creation of 'candidate' compose events was introduced with the shift to Pungi 4 as the compose tool for Fedora 24
• The system was extended to allow for a parallel track of 'Fedora Modular' events during the Fedora 27 cycle, where the first Fedora Modular Server release was made on a different schedule and from a different compose track to the rest of the Fedora 27 release

• The system was modified to handle dists other than "Fedora" more generically (rather than hard-coded, special-cased handling of "Modular" composes) during the Fedora 33 cycle, for use with IoT composes

Features

The following can be considered as features of the Wikitcms 'system':

• Storage and organization (as pages, in categories) of test cases
• Advanced formatting and templating for test case content (via the mediawiki engine)
• Result reporting without authentication
• Capability to distinguish between test case and 'test instance'
• Results sorted by environment (with arbitrary 'environments')
• Tests for a given build grouped by 'test type' and section within test type

Implementation

Fundamentally, Wikitcms consists of a set of conventions and expectations about the names, contents and categorization of wiki pages related to test results. There are also several critical 'auxiliary' pages which are used in the generation of the primary pages; these are considered a part of the system and should be modified only with care.

General caveat for things interfacing with Wikitcms

The component pages of the 'system' can often be human edited, and so variation must be expected in the content within the requirements specified below. Two considerations apply to most data you may want to take from or enter into the system:

• Mediawiki tends to tolerate and ignore leading and trailing whitespace: for instance, mediawiki displays {{result|fail|adamwill|123456}} and {{ result | fail|adamwill | 123456 }} identically, and things interfacing with the system should treat them as equivalent
• In page names underscore (_) and space ( ) characters should be treated as identical and interchangeable: Foo_Bar and Foo Bar lead to the same page, and therefore any code handling page names should expect to cope with either format and consider them as identical

The python-wikitcms module follows these principles and handles most variations in the formatting of the data it can extract, so using it is recommended.

Naming conventions

• Release validation result pages are in the Test Results namespace, and are named DIST RELEASE MILESTONE COMPOSE TESTTYPE, where:

  DIST is usually "Fedora", but may be e.g. "Fedora-Modular" or "Fedora-IoT" in the case of validation events for a different compose track (e.g. Fedora 27 Modular Server composes, Fedora 33+ IoT composes)

  RELEASE is a Fedora release version number

  MILESTONE is a valid Fedora milestone for the release, or a nightly compose type, currently Rawhide or Branched

  Prior to Fedora 24, COMPOSE was a TC or RC compose identifier, e.g. TC1 or RC2 (nb: RC2.1 is a valid format here), or a nightly compose date, in YYYYMMDD format

  From Fedora 24 onwards, COMPOSE is a candidate compose label, e.g. 1.1 or 1.2, or a nightly compose identifier in YYYYMMDD.n.S format (where S is the respin increment), e.g. 20161025.n.0

  TESTTYPE is one of the validation test types (see below)

• Test types for the "Fedora" dist are defined by the existence of a template page (in the Template namespace) named TESTTYPE test matrix. Test types for other dists are defined by the existence of a template page named TESTTYPE SHORTDIST test matrix, where SHORTDIST is the dist name with "Fedora-" removed, e.g. "Modular" or "IoT"
• Validation test category pages are named as follows:

  The top-level Category:Test_Results exists

  Each Fedora release has a Fedora RELEASE Test Results category that is a member of Category:Test_Results

  There is a category for each milestone for each Fedora release, which is a member of that release's category, with the name Fedora RELEASE MILESTONE Test Results

  If there are nightly validation pages for the release, there is a category with the name Fedora RELEASE Nightly Test Results, which is a member of the that release's category

  Where events exist for a release for dists other than Fedora, there will be an equivalent category hierarchy with the dist name (e.g. Fedora-IoT RELEASE Nightly Test Results, and so on)

• Test case pages are in the QA namespace and their name begins with Testcase_ (which, as noted above, is effectively equivalent to Testcase )

Contents

Result pages

The result pages contain a section heading, Test Matrix, which acts as a separator. The page wikitext beyond that separator is considered the result text. Results take the form of one or more wiki tables, but separating tests into different tables is only cosmetic.

Result tables must follow these rules:

• They have a single header row, which appears as a single line in the wikitext. The format where each cell in the header row appears on a newline in the wikitext is not allowed for Wikitcms purposes
• Each row other than the header row - a result row - represents a single test instance (usually just a test case, but see elsewhere in this section for the way in which test names and page sections can produce multiple test instances for a single test case)
• In contrast to the header row, result rows place each of their cells on a new line in the wikitext. The format where the entire row appears on a single line of wikitext is not allowed for Wikitcms purposes
• The first column indicates the level (usually milestone) of the test
• One of the columns must be used for the purpose of identifying the test instance with which that row is associated

  For each row, the contents of that column must be formatted to begin with a link to the test case associated with the row, e.g. [[QA:Testcase_Mediakit_Checksums]]

  The link may be named, in which case the name portion is considered the test name, e.g. [[QA:Testcase_Boot_default_install|Workstation live]]. This is an example of a test instance which is not simply a single test case. It is considered a test instance with the test case QA:Testcase_Boot_default_install and the test name Workstation live. The same test case could appear in another row with a different test name, which would constitute another test instance

• Each column after the identifier column is a result column: its column title represents a test environment, and its contents is one or more test results
• Test results must use the Template:Result template, e.g. {{result|fail|adamwill|123456}} (to indicate a failure from the user adamwill with the related bug #123456)

  Comments may appear following the test result with which they are associated; all text between one test result and the next (or the end of the line) is considered a comment associated with that test result

The result text for a given page may be separated by section, and if it is, the section in which a test instance appears is considered an identifying attribute of that test instance: that is, if there are two rows for the same test case (with no test name, or with the same test name) but they appear in separate sections, they are considered two separate test instances.

Test case pages

The contents of test case pages is not currently used by the Wikitcms 'system'; it effectively simply expects that what they contain will be some sort of content from which a human being can generate a test result. The contents are 'parsed' only by the human testers.

In practice, it is strongly established that each page should contain a single test case using the Template:QA/Test_Case template, according to QA:SOP_test_case_creation. Some test cases used in Wikitcms take advantage of mediawiki's advanced templating features, including, for example, content that changes depending on the nature of the current compose according to Template:CurrentFedoraCompose, or simply sharing common content between many test cases for consistency and convenience of editing. This is a significant difference from most test management systems, which have no such capabilities for their test cases.

Auxiliary pages

Auxiliary pages are those which are used in the process of generating the other pages in the system. They are not used as stores of results data directly. These pages are critical to the functioning of the system and care must be taken when editing them. Sensible simple edits to the text in the templates are usually fine, but please be careful about touching any of the critical wiki syntax. Please contact QA before making changes to any of these pages if you are at all unsure of the consequences.

Not all template pages and so on that are used by pages that form a part of the system are really crucial to its operation; only those that would cause the system described here not to function correctly are considered auxiliary pages.

Result page generation system

The result page generation system consists of a set of nested templates. It is set up such that one complete result page can be generated using a single template, e.g. {{subst:Validation results|testtype=Installation|release=42|milestone=Beta|compose=1.1}} to generate the Beta 1.1 Installation results page for Fedora 42. As long as it is invoked with correct parameters (i.e. a sensible Fedora release number, existing test type, valid milestone and compose or date), it should always generate result pages that comply with the requirements listed above. This style of invocation is the only intended interface to this mechanism; any other use of the component templates is not expected and may have unintended consequences. The python-wikitcms module, and the relval tool based upon it, generates result pages using this system.

The pages which make up the system are:

• Template:Validation_results and Template:Modular_validation_results
• Template:Release_validation_instructions
• The result matrix templates for each test type: those are found in the Category:QA_test_matrix_templates and Category:QA_Modular_test_matrix_templates (for Modular events) categories and must be named as specified in #Naming_conventions

  Each result matrix template, when substituted, should produce the result text (not including the Test Matrix section separator) for a result page in the format described above, with all result cells containing a single {{result|none}} result

Any situation in which the result page generation system produces pages that do not meet the expectations documented here should be considered a bug in the system and reported.

CurrentDISTCompose (e.g. CurrentFedoraCompose) pages

Template:CurrentFedoraCompose, Template:CurrentFedora-ModularCompose etc. define the current compose0s for a given dist: the candidate or nightly composes which are considered to be under active testing by the QA team at present. They can be used as templates with various parameters to display various elements of the current compose's identity, and are also intended to be used as the authoritative source of that data by things interfacing with Wikitcms as a 'system'.

The python-wikitcms module and relval use them in various ways, and their contents are usually expected to be updated by passing the --current parameter to relval when creating a new validation event. Hand editing the page is not recommended; its format is quite specific and mistakes may cause it to be processed incorrectly.

Things interfacing with Wikitcms as a 'system' should expect each line in the page's wikitext that begins with a pipe symbol (|) to represent a different attribute of the 'current' compose's identification. The pipe symbol may be followed by any number of spaces (including none), then a string containing no spaces that identifies an attribute, then any number of spaces (including none), then an equals sign (=), then any number of spaces (including none), then a string which may contain spaces that specifies the value for that attribute. The value ends at the end of the line, or - if any number of spaces directly precedes the end of the line - at the last character before the first of those spaces. For example, the line:

|full   =22 Rawhide 20150107  

(which ends with two space characters) means the full attribute has the value 22 Rawhide 20150107.

The following attributes should always be present: full, release, milestone, compose, date. The value for one of compose or date should always be empty; compose being empty indicates that the current compose is a nightly build (in which case the value of milestone should be either Rawhide or Nightly), while date being empty indicates that the current compose is a candidate compose (in which case the value of milestone should be a valid release milestone). release should always be a valid Fedora release number. full combines all the non-empty attributes in a human-readable string defining the complete identification of the current compose.

Template:FedoraVersion and Template:FedoraVersionNumber

These two pages are used throughout the wiki and are not maintained as part of the Wikitcms 'system', but they are used extensively by pages that form a part of Wikitcms and so they are considered auxiliary pages for its purposes. Changes to their behaviour could cause unexpected results for Wikitcms.

Supplementary pages

Supplementary pages are those that are provided only as conveniences for human interaction with the system, not for storage or organization of results data. Things that treat Wikitcms as a 'system' should not usually rely on their existence, though in practice this assumption is usually safe.

Results summary page

For each validation test event (a candidate, or nominated nightly compose), there is a page named as above, but with Summary in place of the TESTTYPE. It is produced by transclusion of the result pages, which are set up (by the generation system) such that transcluding them shows only their result text. Thus the summary page combines the result text from all the result pages.

Current redirect pages

For each test type and also for the results summary page, a page exists in the Test Results namespace, named Current TESTTYPE Test (or Current Summary for the summary page). There are Current SHORTDIST TESTTYPE Test pages, and a Current SHORTDIST Summary page, for event tracks for other dists. These pages should always redirect to the result page for the current compose, as defined by the Template:CurrentFedoraCompose or Template:CurrentDISTCompose page defined above. In the case of a conflict, Template:CurrentFedoraCompose should be treated as the authoritative definition of what is current, not the destinations of these redirect pages, though conflicts should happen rarely if at all so long as the relval utility is used - it updates both Template:CurrentFedoraCompose and the redirects, or neither.

Reference implementation

The python-wikitcms module is considered the 'reference implementation' for things that interface with the Wikitcms 'system'. When reading existing content, it is intended to follow the expectations described in this page: that is, any wiki content consistent with the expectations specified here should be correctly processed by python-wikitcms. Similarly, when creating content in the Wikitcms 'system' - creating new result pages, or entering results into existing pages - it is intended to create them in accordance with these expectations.

Any divergence between the expectations described here and the behaviour of python-wikitcms should be considered a bug in one or the other, and reported as such.

Note that python-wikitcms is tolerant of a wider range of behaviours than described above in many cases. For instance, it understands several alternatives to Test Matrix as the result text separator in a result page, it can process the older test result templates as well as the current one, and it accepts several variants on the test case page name convention described above. It does this for the purpose of backwards compatibility with content created for older releases, and these behaviours should not be used in newly-created content. Please ensure all newly-created content in the system follows the expectations described in this page.

Alternatives and comparison

It is possible that by this point the reader might be wondering why this rather odd 'test management system' exists in the first place, and why it has not been discarded in favour of a dedicated alternative. QA has considered adopting a dedicated test management system of some kind at several points; the pages Tcms_Comparison, Tcms_feature_requirements and Tcms_use_cases relate to some of these efforts (note they were written in 2010-2011 and discuss rather older versions of Wikitcms and its alternatives).

In general, we have found two major areas where Wikitcms differs from alternatives, which have made it difficult to switch:

• Test case content: as discussed above, Wikitcms uses mediawiki's advanced templating capabilities for test case content in many cases, and alternatives usually have no similar capabilities
• Result reporting: this can be considered a weakness of Wikitcms as (at least until the introduction of relval) it requires reporters to edit wiki tables directly and enter the {{result|...}} template format correctly; human error is not uncommon. However, alternatives tend to have an entirely different set of expectations for user interaction with the system; they tend to be based on a design where a known and implicitly quite small set of testers is tasked with running complete 'test runs' by a higher authority who then synthesizes the results. Wikitcms' design, by contrast, is for an unknown and potentially large set of testers to input results for single test cases, in a format which naturally produces a visual synthesis of results for a specific set of test cases (the display format of a test result page)
• Authentication: Wikitcms allows anonymous result submission (the Test Results namespace can be edited without logging in to the wiki). Dedicated systems generally do not allow this

In general, the major advantages of just about any dedicated test management system over Wikitcms are:

• More structured data: Wikitcms is very vulnerable to human error in creation of its data, as mediawiki is a general purpose system and there can be no enforcement of compliance with the expectations documented on this page except through an intermediate system like python-wikitcms
• Separation of data from its visual representation: Wikitcms innately ties together its content with its primary visual representation - the mediawiki table provides both the 'system's' organization of results, and the human-visible representation of them. This obviously means processing results is much more complex than if they were stored in something sensible like a database, and makes it unnecessarily difficult to produce alternative representations of the data
• Efficiency: obviously, using a general-purpose wiki as a data store is not the most efficient possible design, and dedicated systems are pretty much universally more efficient in general

Here are some specific alternative systems, and some notes on comparison of Wikitcms with them.

Moztrap

Moztrap is Mozilla's current test management system. It has many attractive features, including a lot of organizational flexibility and powerful search capabilities. It has an active developer team which is receptive to outside feedback and contribution. The primary public implementation of Moztrap is Mozilla's own, used for various testing performed on Mozilla products.

It is a Django-based web application; it has some somewhat outdated pinned dependencies.

It allows Markdown-style formatting for test cases, but no templating. Test cases expect a strict 1:1 match between actions and results - each 'action' step must have a single corresponding 'result' step (Wikitcms has no such requirement, and many of its test cases do not meet it).

Its approach to environments, while very powerful, is a bit painful for conversion from Wikitcms, as it generally expects all possible combinations of environments specified for a given test run to be valid for all test cases in it, while Wikitcms often has different sets of environments for different sections (or even just different tables) within a single test type. This means that converting a single Wikitcms test type to a single Moztrap test run requires a lot of manual editing of test case / environment associations; alternatively each table in a Wikitcms result page could become a separate test run in Moztrap, but that complicates the conversion process and may overwhelm testers in the result submission process.

Its result submission process is quite strongly tied to the idea that each tester will perform a complete test run. It is possible to simply enter a result for a single test case, but this is awkward and clearly not expected. A significant problem is that the tester must select from all possible environments for the test run before submitting any results - so if the tester only wants to submit a result for a test case where the only relevant environment choice is arch, but other test cases in the same test run have other environment sets, the tester must make choices for all the irrelevant environment sets before submitting their result.

The interfaces for viewing and submitting results are quite strongly separated, unlike in Wikitcms.

It requires some form of authentication for all 'write'-type operations; anonymous users cannot submit results, create test cases or runs, or anything along those lines.

It is possible to integrate its authentication with FAS (this has been tested in a demo instance).

Contact: discussing and reporting bugs in Wikitcms

The Wikitcms 'system' is maintained by the QA team. If you have any questions or concerns about its behaviour, or want to report anything that can be considered a 'bug' in the system, please contact the QA group via one of the methods described at QA#Communicate. At present Adam Williamson is the primary 'maintainer' of the 'system', and is happy to discuss it via IRC (adamw) or email.