MinGW/CrossCompilerFramework

Introduction

One of the new features of F16 will be the introduction of a Cross Compiler Framework. This framework will obsolete the Fedora MinGW toolchain which can be found in current versions of Fedora.

Initially, this framework will have support for building binaries for Win32 and Win64 targets. The mingw-w64 toolchain will be used for this. However support for other targets (like Mac OS X) can be added quite easily.

The goal of this framework is to provide a generic method to cross-compile binaries for multiple targets with as little packaging overhead as possible.

Namespaces

The framework will use several namespaces. Each namespace has it own characteristics. These will be explained here

Package names:

cross-            Used for packages which can be built for multiple targets
mingw-            Used for packages which can be built for both Win32 and Win64 but not for other targets
mingw32-          Used for packages which can only be built for Win32
mingw64-          Used for packages which can only be built for Win64
darwinx-          Used for packages which can only be built for Mac OS X (future work)

Filelists:

%{_mingw32_...}   Used for files which are only used on Win32
%{_mingw64_...}   Used for files which are only used on Win64
%{_darwinx_...}   Used for files which are only used on Mac OS X (future work)

Development and testing repository

All development of this toolchain is currently being done in a SVN repository at http://svn.nntpgrab.nl/svn/fedora_cross (SVNWeb: http://svn.nntpgrab.nl/viewvc/Fedora%20Cross%20Compiler%20Framework/) It has anonymous read-only access. For write access, please contact epienbro on FreeNode #fedora-mingw.

At the moment there are two testing repositories available. The first one contains Win32 and Win64 packages while the other one contains Win32, Win64 and Mac OS X packages. The packages from the first one are expected to land in Fedora 16.

NOTE: If you want to test, please use only one of the below methods, not both

Testing repository containing Win32 and Win64 packages:

The packages from this repository are planned to be added to Fedora 16. It contains all mingw32-* packages which have been in Fedora as of December 30 2010.

All packages from this repository contain the dist tag '.fc15_cross'.

To enable this repository, drop the file http://build1.vanpienbroek.nl/fedora-cross/fedora-cross.repo in /etc/yum.repos.d and perform a 'yum update'. Note that this testing repository will obsolete any mingw32 packages you might have installed!

If you're still running on a Fedora 14 environment you need to un-comment the 'exclude=..' line in the /etc/yum.repos.d/fedora-cross.repo file. Otherwise you'll get a dependency resolving conflict.

Testing repository containing Win32, Win64 and Mac OS X packages:

Due to legal issues and the fact that various binary blobs are used these packages can't be added to Fedora (yet). Please only use this repository when you really need Mac OS X support.

All packages from this repository contain the dist tag '.fc15_cross_darwinx'.

To enable this repository, drop the file http://build1.vanpienbroek.nl/fedora-cross-darwinx/fedora-cross-darwinx.repo in /etc/yum.repos.d and perform a 'yum update'. Note that this testing repository will obsolete any mingw32 packages you might have installed!

If you're still running on a Fedora 14 environment you need to un-comment the 'exclude=..' line in the /etc/yum.repos.d/fedora-cross-darwinx.repo file. Otherwise you'll get a dependency resolving conflict.

Issues with these testing repositories can be reported on the fedora-mingw mailing list or #fedora-mingw on FreeNode

Package versions

In the development/testing repository several changes have been applied which aren't part of the mingw32 packages which are currently in rawhide. Here's a list of the changes which aren't in rawhide yet:

Binutils / GCC:

The mingw-w64 developers have indicated that the latest HEAD versions of binutils and GCC (4.6) are stable for generating binaries for Win32 and Win64 targets. I The latest HEAD versions of binutils and gcc are currently already used in rawhide for the native toolchain. For the cross compiler framework the same package versions of binutils and gcc are used as their native fedora counterparts (which are recent snapshots).

The latest GCC also enables LTO support.

mingw-w64 headers/crt:

To build binaries for Win32/Win64 you need headers and some runtime libraries. Upstream maintains two sets of packages containing the mingw-w64 headers and the crt. For the cross compiler framework we decided to use the 'v1.0' branch and not the trunk version. If there's a demand (or upstream indicates that the trunk version is stable enough) we can switch to the trunk version instead. At the time of writing the latest v1.0 release is a snapshot dated 20101003 which can be found at http://sourceforge.net/projects/mingw-w64/files/

New packages:

• libiconv has been replaced by win-iconv (review request)
• libjpeg has been replaced by jpeg-turbo

Changes in packages:

• filesystem: fixed a bug which triggered invalid declarare statement in the %{_mingw32_env} macro (BZ#657478)
• filesystem: the mingw32_cmake macro now supports out-of-source builds
• gcc: dropped the pthreads dependency on gcc by moving the libgomp pieces (which require pthreads) to a seperate package (BZ#599567)
• gcc: the testsuite (for the i686-w64-mingw32 target) will be run (using wine) every time the package is built
• gettext: make libintl-8.dll a soft-dependency (already in rawhide)
• cairo: enable 'tee' support

Regular package updates:

• Update to gdk-pixbuf 2.23.0
• Update to pixman 0.21.2
• Update to glib 2.27.95
• Update to libsoup 2.33.5
• Update to gtk2 2.23.90
• Update to gnutls 2.10.4
• Update to cairo 1.10.2

Roadmap

To provide a clean upgrade path and as little breakage as possible the introduction of this framework can be split in several phases.

Phase 1: Package reviews

The 5 packages which form the toolchain have to be reviewed and approved first mingw-filesystem: https://bugzilla.redhat.com/show_bug.cgi?id=673784 mingw-binutils: https://bugzilla.redhat.com/show_bug.cgi?id=673786 mingw-gcc: https://bugzilla.redhat.com/show_bug.cgi?id=673788 mingw-headers: https://bugzilla.redhat.com/show_bug.cgi?id=673790 mingw-crt: https://bugzilla.redhat.com/show_bug.cgi?id=673792

Phase 2: Request a separate buildroot

To avoid breakage in the rawhide tree it's recommended to perform large changes in a separate buildroot first and when everything is ready merge everything back in one go

Phase 3: Bootstrap the mingw-w64 toolchain

Once the 5 packages mentioned in phase 1 are approved they can be imported. In order to bootstrap the mingw-w64 toolchain this has to be done in a specific order.

1. Import the mingw-filesystem package, build it and wait for it to enter the buildroot
2. Import the mingw-binutils package, build it and wait for it to enter the buildroot
3. Import the mingw-headers package, build it and wait for it to enter the buildroot
4. Import the mingw-gcc package, build it (with the %bootstrap flag set to 1 and the %enable_libgomp flag set to 0) and wait for it to enter the buildroot
5. Import the mingw-crt package, build it and wait for it to enter the buildroot
6. Rebuild the mingw-gcc package with the %bootstrap flag set to 0 (and the %enable_libgomp flag set to 0) and wait for it to enter the buildroot

Phase 4: Rebuild all mingw32 packages

The mingw-w64 toolchain is now operational and can build binaries for both Win32 and Win64. As the name of the triplet has been changed (in the cross-filesystem package) from i686-pc-mingw32 to i686-w64-mingw64 as mingw32 packages have to be rebuild. All mingw32 packages should have been using macros like %{_mingw32_bindir} as mentioned in the MinGW Packaging Guidelines. As all these macros have been moved to the cross-filesystem package a plain rebuild should be sufficient to get mingw32 packages built against the new toolchain.

Packages which have the triplet 'i686-pc-mingw32' hardcoded in their .spec files need to updated as they aren't conforming to the packaging guidelines

Any packages which failed to build need to be fixed first before continuing

Phase 5: Merge the separate buildroot back in the rawhide tree

As all packages are rebuilt against the new toolchain all updated packages can be merged back in the rawhide tree. From here on, all changes can be applied directly in the rawhide tree

Phase 6: Drop the old mingw32 toolchain packages

Now that the mingw-w64 based toolchain is imported the original mingw32 based toolchain has been obsoleted. The packages mingw32-filesystem, mingw32-binutils, mingw32-gcc, mingw32-w32api and mingw32-runtime can be dropped from the repository without breaking anything

Phase 7: Rename and port all mingw32-packages to the new framework

This phase is also the most time-consuming one. Right now all mingw32 packages use the prefix 'mingw32-'. All packages (which want to be built for both win32 and win64) have to be renamed to use the prefix 'cross-' instead of 'mingw32-'. In additional several macros will have to be replaced. See the porting guide for more details.

This phase isn't time limited. Package maintainers don't have to port to the new framework if they don't want to add support for additional targets. They can leave their packages as is

In the testing repository there are 32 packages which have already been ported to the new cross compiler framework. These changes can be applied to rawhide as well

A special case is the pthreads package. Once the mingw-pthreads package has been imported and built then we will be able to (re-)enable libgomp support in GCC. Enabling this feature is just switching a flag called %enable_libgomp in the mingw-gcc package.

Porting guide

First thing to keep in mind is that the name of the triplet for Win32 has been changed. Originally this was i686-pc-mingw32 while with the new cross compiler framework this has changed to i686-w64-mingw32. This has been done to indicate that the new mingw-w64 toolchain is used instead of the old mingw.org one. All paths have been updated to use this new name as well. So now all files are kept in /usr/i686-w64-mingw32 instead of /usr/i686-pc-mingw32 for Win32. After the initial rebuild (as mentioned in phase 4) all files will be moved to this new location.

If you want your package to be built for multiple targets instead of just Win32 you need to rename the package to use the 'cross-' or 'mingw-' prefix (depending on the package) and apply some changes in the .spec file. If you don't want to add support for other targets you can leave the .spec file as is and ignore the rest of this porting guide.

Generic macros

Most Fedora MinGW packages contain this set of instructions at the top of the .spec file:

%global __strip %{_mingw32_strip}
%global __objdump %{_mingw32_objdump}
%global _use_internal_dependency_generator 0
%global __find_requires %{_mingw32_findrequires}
%global __find_provides %{_mingw32_findprovides}
%define __debug_install_post %{_mingw32_debug_install_post}

These have to be adjusted so that the generic cross compilation macros are used:

%global __strip %{_cross_strip}
%global __objdump %{_cross_objdump}
%global _use_internal_dependency_generator 0
%global __find_requires %{_cross_findrequires}
%global __find_provides %{_cross_findprovides}
%define __debug_install_post %{_cross_debug_install_post}

Several macros belonging to the cross compiler framework need to know the name of the package (without it's prefix). This can be indicated with this line:

%global _cross_pkg_name mypkg

(Replace 'mypkg' with the name of the package, for example 'glib2' or 'gnutls')

To indicate which targets you want to have the package build you need to add one of these (or all) lines to your .spec file. To build for Win32:

%global _cross_build_win32 1

To build for Win64:

%global _cross_build_win64 1

To build for Mac OS X (future work):

%global _cross_build_darwinx 1

Each package must contain these two lines BEFORE the first occurrence of the %description tag:

%{?_cross_default_requires}
%{?_cross_debug_package}

The first macro expands to multiple 'Requires: xxx' tags (depending on the configured build targets, see above). This makes sure then whenever the user wants to install the package 'cross-mypkg' that also the packages 'mingw32-mypkg' and 'mingw64-mypkg' get pulled in.

The second macro is needed to generate a debuginfo subpackage. This replaces the %{?_mingw32_debug_package} macro which was used in several original mingw32 packages

Per-target packages

The goal of the cross compiler framework is to build binaries for multiple targets from a single source RPM. In the first iteration of the cross compiler framework binaries for all compiled targets were placed in the same package. In response to that there has been a demand to split this single binary RPM into per-target RPMs.

To be able to do that one has to define %package sections for all targets (and possibly for other subpackages like a '-static' one). Once the %package sections have been added for all targets you also need to add %files sections for all these packages. In these file lists you can use macros which are mentioned in the file lists section below

It is known that this introduces a lot of duplication, but we haven't been able to rewrite this into proper RPM macros yet. Help is welcome in this area to reduce duplication!

An example can be found at http://svn.openftd.org/viewvc/Fedora%20Cross%20Compiler%20Framework/darwinx/cross-glib2/cross-glib2.spec?view=markup

Upgrade path

Once a package is ported to the cross compiler framework (using per-target binary RPMs) then there are no additional steps needed (like obsoletes tags) to provide an upgrade path. If an user only has mingw32-* packages installed then only packages will be pulled in for the mingw32 target. Packages belonging to other targets will only get installed when the user explicitly installs one of when the user installs the 'cross-mypkg' package (which pulls in the binaries for all available targets, see the %{?_cross_default_requires} remarks in the generic macros section above).

Description and Summary

Most packages mention in their description and/or summary that the package in question is targeted for Win32. Though not strictly required, it's recommended to update the description and summary tags to indicate the targets which this package supports

BuildRequires

As a new 'cross-' prefix has been introduced for the base toolchain packages the BuildRequires have to be updated:

Original                             New
------------------------             ----------------------
mingw32-filesystem >= xx             cross-filesystem >= 65
mingw32-binutils                     cross-binutils
mingw32-cpp                          cross-cpp
mingw32-gcc                          cross-gcc
mingw32-gcc-c++                      cross-gcc-c++
mingw32-runtime                      mingw-runtime
mingw32-w32api                       mingw-headers

For other dependencies it depends on whether that specific package has been ported to the new cross compiler framework

Running the ./configure script

The original Fedora Mingw toolchain contained a macro called %{_mingw32_configure} which could be used to call the ./configure script with the proper arguments set. For cross compilation to multiple targets a new macro has been introduced called %{_cross_configure}. The macro creates a separate folder for each target (out of source compilation) and executes the configure script for every target (set with the flags mentioned earlier).

Almost all packages support out of source compilation or require slight patching. The only known exceptions to date are zlib and openssl. Packages which don't support out of source compilation may require a different approach like performing everything in the %install phase. If you happen to stumble across a package which requires a different approach feel free to contact us on the Fedora MinGW mailing list

If you want to supply arguments to the ./configure call, you have to use a slightly different approach than before. Say you've got something like the call below:

%{_mingw32_configure} --disable-xlib --enable-win32

This has to be changed to something like this:

%{_cross_configure "--disable-xlib" "--enable-win32"}

The quotes mentioned above are mandatory! If you forget them then rpmbuild will fail or the arguments might not get passed to the ./configure calls. Also note that the '}' character needs to be placed after all the arguments

Some packages need to be built multiple times for each target. Examples of this are packages which have to be built once for a static version and once for a shared version. Such packages can add a custom suffix to the build directory used. Say you've got something like below:

mkdir build_shared
pushd build_shared
%{_mingw32_configure} --enable-shared
popd
mkdir build_static
pushd build_static
%{_mingw32_configure} --enable-static
popd

This can be rewritten to something like this:

%{_cross_configure -s shared "--enable-shared"}
%{_cross_configure -s static "--enable-static"}

Building and installing the package

Most packages used the command 'make %{?_smp_mflags}' to build the package. In the cross compiler framework you have to use '%{_cross_make %{?_smp_mflags}}' to build the package for all configured targets As with the %{_cross_configure} macro you can also use the '-s' argument to indicate a custom suffix to the build directory used

To install the package the command 'make install DESTDIR=$RPM_BUILD_ROOT' was used in almost all cases. This can be rewritten to '%{_cross_make_install DESTDIR=$RPM_BUILD_ROOT}' to install the package for all configured targets The '-s' argument can also be used here

Custom %install instructions

Some packages require some custom instructions before the files are ready to be packaged. Such code can remain as is. However, you may need to duplicate these instructions multiple times (for all configured targets).

Translations

Normally all translations should be collected using the %find_lang macro. As this macro generates a list of all available translations (for all targets) it isn't really useful for us. To fix this situation a new macro has been added called %_cross_find_lang. This macro generates multiple file lists. Each file list contains the translations for a given target. Instead of a file named 'mypkg.lang' multiple files are created called 'mingw32-mypkg.lang', 'mingw64-mypkg.lang' and 'darwinx-mypkg.lang' (if enabled). These file lists can be used in the %files section for the various per-target packages.

File lists

All the original filelist macros from the original Fedora MinGW toolchain (like %{_mingw32_bindir}) are still available. The following macros exists for use in filelists:

%{_mingw32_sysroot}
%{_mingw32_prefix}
%{_mingw32_exec_prefix}
%{_mingw32_bindir}
%{_mingw32_libdir}
%{_mingw32_includedir}
%{_mingw32_datadir}
%{_mingw32_sysconfdir}
%{_mingw32_docdir}
%{_mingw32_sbindir}
%{_mingw32_libexecdir}
%{_mingw32_infodir}
%{_mingw32_mandir}
%{_mingw32_sharedstatedir}
%{_mingw32_localstatedir}

%{_mingw64_sysroot}
%{_mingw64_prefix}
%{_mingw64_exec_prefix}
%{_mingw64_bindir}
%{_mingw64_libdir}
%{_mingw64_includedir}
%{_mingw64_datadir}
%{_mingw64_sysconfdir}
%{_mingw64_docdir}
%{_mingw64_sbindir}
%{_mingw64_libexecdir}
%{_mingw64_infodir}
%{_mingw64_mandir}
%{_mingw64_sharedstatedir}
%{_mingw64_localstatedir}

These are only available if you're using the Mac OS X based testing repository:
%{_darwinx_sysroot}
%{_darwinx_prefix}
%{_darwinx_exec_prefix}
%{_darwinx_bindir}
%{_darwinx_libdir}
%{_darwinx_includedir}
%{_darwinx_datadir}
%{_darwinx_sysconfdir}
%{_darwinx_docdir}
%{_darwinx_sbindir}
%{_darwinx_libexecdir}
%{_darwinx_infodir}
%{_darwinx_mandir}
%{_darwinx_sharedstatedir}
%{_darwinx_localstatedir}

Dropped packages

Some packages aren't needed anymore to compile software these days so they won't be ported to the new cross compiler framework and will be dropped from the repository eventually:

mingw32-dlfcn