MinGW/Testsuite

MinGW Testsuite

Introduction

At the moment there is no automated method to check the state of the MinGW toolchain which is in Fedora. There's a good possibility that regressions get slipped in and only get discovered after some time.

To prevent these kind of issues we need to have an automated testsuite which can test various parts of the Fedora MinGW toolchain.

Such a testsuite can eventually be hooked in AutoQA so automated testing will be done automatically as soon as something is changed in one of the Fedora MinGW packages.

Requirements

• Must support both the current mingw.org-based toolchain as well as the new mingw-w64-based toolchain
• Must support both win32 as well as win64 binaries
• Must be able to run on Fedora 16 and above and RHEL6/CentOS6
• Must be easily extendable (without re-compilation of the testsuite tool)

Type of tests

RPM macros

• Goal: regression testing of the mingw-filesystem package
• Method: parse rpm code and make sure the parsed output is what we expect
• Input: text file containing rpm code which needs to be parsed
• Output: the parsed rpm code

Compile and runtime checks (single file)

• Goal: test the toolchain
• Method: testing if compilation of a .c / .cpp file succeeds without compiler warnings/errors
• Input: .ini file describing which .c / .cpp file should be used, any additional CFLAGS, boolean whether the binary should be executed using wine and expected output
• Output: gcc output, exit code of gcc, runtime output, runtime exit code

Custom script

• Goal: test the toolchain
• Method: call a custom script which can set the exit code to indicate whether the test has failed or succeeded
• Input: .ini file describing the script which needs to be executed and its command line parameters, the script which contains the testcase
• Output: the output of the script and its exit code