MinGW/Testsuite

MinGW Testsuite

Introduction

At the moment (November 2011) there is no automated method to check the state of the MinGW toolchain which is in Fedora. There's a good possibility that regressions get slipped in and only get discovered after some time.

To prevent these kind of issues we need to have an automated testsuite which can test various parts of the Fedora MinGW toolchain.

Such a testsuite can eventually be hooked in AutoQA so automated testing will be done automatically as soon as something is changed in one of the Fedora MinGW packages.

Requirements

• Must support both the current mingw.org-based toolchain as well as the new mingw-w64-based toolchain
• Must support both win32 as well as win64 binaries
• Must be able to run on Fedora 16 and above and RHEL6/CentOS6
• Must be easily extendible (without re-compilation of the testsuite tool)
• Must the able to hook into AutoQA

Fedora MinGW Testsuite tool

In order to create such a testsuite a small tool was created by Erik van Pienbroek (epienbro). This is a small (about 1000 lines of code) program written in the C programming language which uses GLib's GTester framework. With this framework it is possible to easily create test cases and also to make the results available in both machine friendly form (XML) as user-friendly form (HTML reports).

On startup this tool searches it's testcases folder for available testcases and tries to prepare them for execution. After all available testcases are prepared the testsuite is executed and the results are printed to the console or in a XML/HTML file (depending on how the tool was executed).

During the prepare phase the following tasks are done for each testcase:

• Parse the contents of the testcase file
• Make sure that the host is able to execute the testcase (it doesn't make sense to run mingw-w64 specific testcases when there are no mingw-w64 available on the host)
• Replace any possible placeholders with the truly expected data (like %TARGET% and %TESTCASESDIR%)

Another task which is done during the prepare phase of the tool is the creation of a clean wine prefix. This wine prefix will be used by testcases of the type compile_and_run and custom and is created in a temporary folder (/tmp/wine_mingw_testsuite_XXXXXX) which will automatically be cleaned up once all testcases are executed. While creating this wine prefix several symbolic links will also be created from %{mingw32_bindir}/*.dll to $WINEPREFIX/drive_c/windows/syswow64/ and %{mingw64_bindir}/*.dll to $WINEPREFIX/drive_c/windows/system32/. Due to this testcases can also be executed which depend on dll's provided by mingw packages (like libstdc++-6.dll and libglib-2.0-0.dll)

The Subversion repository containing the source code of this tool can be found at http://svn.openftd.org/svn/fedora_mingw_testsuite/trunk/ A Subversion webinterface can be found at http://svn.openftd.org/viewvc/Fedora MinGW Testsuite/trunk

Usage

1. Install required dependencies: yum -y install glib2-devel rpm-devel wine cmake autoconf automake
2. Download the code using: svn co http://svn.openftd.org/svn/fedora_mingw_testsuite/trunk fedora_mingw_testsuite
3. Go to the folder containing the source code: cd fedora_mingw_testsuite
4. Generate the configure script: ./autogen.sh
5. Execute the configure script for the desired target:
   • For Fedora <= 16: ./configure --enable-mingw-old-target
   • For RHEL 6: ./configure --enable-mingw-el6-target
   • For users of the mingw-w64 testing repo: ./configure --enable-mingw-w64-target
6. Build the tool: make

Once these steps are completed, you should see a binary called 'fedora_mingw_testsuite' in the folder which contains the source code.

To run the testsuite, just call ./fedora_mingw_testsuite from the folder containing the source code. Executing this command should give the following output:

$ ./fedora_mingw_testsuite 
/hello_mingw64_configure: OK
/hello_c++: OK
/macro_mingw32_configure_w64: OK
/hello_c: OK
/hello_mingw32_cmake: OK
/hello_mingw64_cmake: OK
/macro_mingw32_configure: OK
/hello_mingw32_configure: OK
/macro_mingw64_configure: OK

This shows all testcases which are available for the target mentioned in the ./configure call and the result of each testcase. Failed testcases can also be seen here.

With just the command ./fedora_mingw_testsuite only brief output is shown. To show more detailed information, you can call the program with the --verbose argument.

To generate the XML and HTML reports you can use the command make full-report. This generates two files called full-report.xml and full-report.html. These files can be processed further with other tools/scripts.

Testcase file format

A testcase always consists of one text file which describes the testcase. This file is using the INI-style for its contents and must have the suffix '.testcase' in order to be recognized by the testsuite tool. Testcases have to be placed in the 'testcases' folder which is part of the Fedora MinGW Testsuite tool.

Examples for the various testcase types are shown in the sections below

All testcases belonging to the Fedora MinGW Testsuite must contain a section named 'mingw_testcase' . In this section a key named 'type' is also required. The value belonging to this key can be used to identify the type of test which needs to be executed. At the moment the following types are valid:

• rpm_parser
• compile_and_run
• custom

Depending on the type of the testcase one or more additional key/value pairs need to be set

Type 'rpm_parser'

Action: Parse the given RPM code using rpmlib and verify the output

Testcases for the type 'rpm_parser' need to provide an input text and an output text. The input text is the RPM code which needs to be processed by the RPM parser. The output text is the RPM code as returned after being processed by the RPM parser. The testcase will be successful when the output from the RPM parser is the same at the expected output.

The following keys are supported:

• required: input_text - string - The RPM code which needs to be processed by the RPM parser
• required: input_file - string - A textfile containing the RPM code which needs to be processed by the RPM parser
• required: expected_output_text - string - The output which is expected
• required: expected_output_file - string - A textfile containing the output which is expected
• optional: expected_output_text_el6 - string - The output which is expected (for the RHEL-6 target only)
• optional: expected_output_file_el6 - string - A textfile containing the output which is expected (for the RHEL-6 target only)
• optional: expected_output_text_w64 - string - The output which is expected (for the mingw-w64 target only)
• optional: expected_output_file_w64 - string - A textfile containing the output which is expected (for the mingw-w64 target only)

Only one *_input key-value pair and one *_output key-value pair are required. It is recommended to use the '*_text' keys for single-line strings and the '*_file' keys for multi-lines.

Here's an example of such a testcase (the file shown is named macro_mingw32_configure.testcase):

[mingw_testcase]
type=rpm_parser
input_text=%{_mingw32_configure}
expected_output_file=mingw32_configure_output.txt
expected_output_file_el6=mingw32_configure_output_el6.txt
expected_output_file_w64=mingw32_configure_output_w64.txt

The testcase tries to parse the RPM macro '%{_mingw32_configure}' (using rpmlib) and verifies the output as mentioned in the mentioned output files (where the output is different on each target, hence the different expected_output entries)

Type 'compile_and_run'

Action: Test if the compilation of a .c / .cpp file succeeds without compiler warnings/errors and (optionally) execute the file using wine and verify its output

The following keys are supported:

• required: source_file - string - The file name of the file which needs to be compiled (relative to the testcases folder)
• required: run_binary - boolean - Whether this testcase only needs to compile the binary (false) or whether it needs to be compiled and executed (true)
• optional: cflags - string - Any additional CFLAGS which need to be used to compile this testcase
• optional: ldflags - string - Any additional LDFLAGS which need to be used to compile this testcase
• optional: expected_stdout_text - string - The stdout output which is expected when the binary is executed
• optional: expected_stdout_file - string - A textfile containing the stdout output which is expected when the binary is executed
• optional: expected_stderr_text - string - The stderr output which is expected when the binary is executed
• optional: expected_stderr_file - string - A textfile containing the stderr output which is expected when the binary is executed

Only one input key-value pair and one output key-value pair are required. It is recommended to use the '*_text' keys for single-line strings and the '*_file' keys for multi-lines.

Here's an example of such a testcase (the file shown is named hello_c.testcase):

[mingw_testcase]
type=compile_and_run
source_file=hello.c
run_binary=true
expected_stdout_text=Hello world!
expected_stderr_text=
cflags=
ldflags=

Type 'custom'

Action: Execute a custom script which performs some tests, check its exit code and verify its output

The following keys are supported:

• required: command - string - Command which needs to be executed
• optional: target - string - Indicate whether this testcase uses the mingw32 or mingw64 toolchain (used to replace placeholders in the output files with correct values). Valid values are 'mingw32' and 'mingw64'
• optional: expected_stdout_text - string - The stdout output which is expected when the binary is executed
• optional: expected_stdout_file - string - A textfile containing the stdout output which is expected when the binary is executed
• optional: expected_stderr_text - string - The stderr output which is expected when the binary is executed
• optional: expected_stderr_file - string - A textfile containing the stderr output which is expected when the binary is executed
• optional: expected_stdout_text_el6 - string - The stdout output which is expected when the binary is executed (for the RHEL-6 target only)
• optional: expected_stdout_file_el6 - string - A textfile containing the stdout output which is expected when the binary is executed (for the RHEL-6 target only)
• optional: expected_stderr_text_el6 - string - The stderr output which is expected when the binary is executed (for the RHEL-6 target only)
• optional: expected_stderr_file_el6 - string - A textfile containing the stderr output which is expected when the binary is executed (for the RHEL-6 target only)
• optional: expected_stdout_text_w64 - string - The stdout output which is expected when the binary is executed (for the mingw-w64 target only)
• optional: expected_stdout_file_w64 - string - A textfile containing the stdout output which is expected when the binary is executed (for the mingw-w64 target only)
• optional: expected_stderr_text_w64 - string - The stderr output which is expected when the binary is executed (for the mingw-w64 target only)
• optional: expected_stderr_file_w64 - string - A textfile containing the stderr output which is expected when the binary is executed (for the mingw-w64 target only)

Only one *_input key-value pair and one *_output key-value pair are required. It is recommended to use the '*_text' keys for single-line strings and the '*_file' keys for multi-lines.

The various output files and the command key can contain some special placeholder tags. These tags can be used to cope with output which can be different between environments. Right now there are two placeholder tags available:

• %TARGET%
• %TESTCASESDIR%
• %GCC_VERSION%

The %TARGET% placeholder can be used in places were the compiler triplet is used. For example, take this line of output (from some configure script):

checking whether i686-pc-mingw32-gcc accepts -g... yes

As the compiler triplet is different between mingw.org and mingw-w64 environments this line can be rewritten to:

checking whether %TARGET%-gcc accepts -g... yes

The same also applies to messages like these:

make[1]: Entering directory /home/erik/fedora_mingw_testsuite/trunk/testcases/hello_configure'

This line can be rewritten to:

make[1]: Entering directory %TESTCASESDIR%/hello_configure'

The %GCC_VERSION% placeholder can be used in places where the gcc version is used

Here's an example of such a testcase (the file shown is named hello_mingw32_cmake.testcase):

[mingw_testcase]
type=custom
command=./hello_mingw32_cmake.sh
target=mingw32
expected_stdout_file=hello_cmake.stdout
expected_stderr_file=hello_cmake.stderr
expected_stdout_file_el6=hello_cmake_el6.stdout
expected_stderr_text_el6=

The _el6 entries are set here because RHEL 6 uses an older version of CMake than Fedora and this older version of CMake produces a slightly different output than the newer Fedora version

An example of an output file using placeholders can be found at hello_configure_mingw_w64.stdout