MinGW Testsuite

Introduction

At the moment there is no automated method to check the state of the MinGW toolchain which is in Fedora. There's a good possibility that regressions get slipped in and only get discovered after some time.

To prevent these kind of issues we need to have an automated testsuite which can test various parts of the Fedora MinGW toolchain.

Such a testsuite can eventually be hooked in AutoQA so automated testing will be done automatically as soon as something is changed in one of the Fedora MinGW packages.

Requirements

Must support both the current mingw.org-based toolchain as well as the new mingw-w64-based toolchain
Must support both win32 as well as win64 binaries
Must be able to run on Fedora 16 and above and RHEL6/CentOS6
Must be easily extendable (without re-compilation of the testsuite tool)

Type of tests

RPM macros

Goal: regression testing of the mingw-filesystem package
Method: parse rpm code and make sure the parsed output is what we expect
Input: text file containing rpm code which needs to be parsed
Output: the parsed rpm code

Compile and runtime checks (single file)

Goal: test the toolchain
Method: testing if compilation of a .c / .cpp file succeeds without compiler warnings/errors
Input: .ini file describing which .c / .cpp file should be used, any additional CFLAGS, boolean whether the binary should be executed using wine and expected output
Output: gcc output, exit code of gcc, runtime output, runtime exit code

Custom script

Goal: test the toolchain
Method: call a custom script which can set the exit code to indicate whether the test has failed or succeeded
Input: .ini file describing the script which needs to be executed and its command line parameters, the script which contains the testcase
Output: the output of the script and its exit code

File format

A testcase always consists of one textfile which describes the testcase. This file is using the INI-file style for its contents and must have the suffix '.testcase' in order to be recognized by the testsuite tool.

Here's an example of such a testcase (the file shown is named macro_mingw32_configure.testcase):

[mingw_testcase]
type=rpm_parser
input_text=%{_mingw32_configure}
expected_output_file=mingw32_configure_output.txt

All testcases belonging to the Fedora MinGW Testsuite must contain a section named 'mingw_testcase' . In this section a key named 'type' is also required. The value belonging to this key can be used to identify the type of test which needs to be executed. At the moment the following types are valid:

rpm_parser
compile_and_run
custom

Depending on the type of the testcase one or more additional key/value pairs need to be set

Type 'rpm_parser'

Testcases for the type 'rpm_parser' needs to provide an input text and an output text. The input text is the RPM code which needs to be processed by the RPM parser. The output text is the RPM code as returned after being processed by the RPM parser. The testcase will be successful when the output from the RPM parser is the same at the expected output.

The following keys are supported:

input_text - The RPM code which needs to be processed by the RPM parser
input_file - A textfile containing the RPM code which needs to be processed by the RPM parser
expected_output_text - The output which is expected
expected_output_file - A textfile containing the output which is expected

Only one input key-value pair and one output key-value pair are required. It is recommended to use the '*_text' keys for single-line strings and the '*_file' keys for multi-lines.

Type 'compile_and_run'

Type 'custom'

Source code

All development is being done in a SVN repository for the time being. The checkout URL is http://svn.openftd.org/svn/fedora_mingw_testsuite/trunk A SVNWeb instance for this SVN repository can be found at http://svn.openftd.org/viewvc/Fedora%20MinGW%20Testsuite/

Search

MinGW/Testsuite

Contents