Architectures/AArch64/Bootstrap

The bootstrap process is now complete, and this page is largely for archive purposes (how we did it). AArch64 is now being built in koji (Fedora Rawhide/F21). Most packages are now built, but a few still require attention. If you would like to help, please check for failed builds in Fedora ARM koji.

Bootstrap Process

Essentially, AArch64 bootstrap is following the same process as that used for Fedora 15 on ARM: ARM HardFP Bootstrap. The older F15 process was intentionally designed with AArch64 bootstrap in mind, and so this similarity is also intended.

The purpose of the effort is to enable Fedora on ARMv8, using the standard 64-bit ABI as defined by ARM. The good news is that unlike ARMv7 and earlier, there is only one ABI (see AAPCS for 64-bit).

Everything Is Saved In One Place
Please note that all of the files and repositories for the ARMv8 bootstrap work are stored at http://fedorapeople.org/groups/armv8. Contact the maintainer Al Stone if you need to have write access.

Goals

Our goals are to complete the bootstrap "stages" (1,2,3,4,5,6) at which we anticipate certain levels of functionality from the root filesystem, beginning with very basic cross compilation, proceeding to native builds, and ultimately to a working mock and Koji configuration able to build production packages.

CURRENT STAGE: We have completed stages 1-3, and are progressing through stage4.

Bootstrap Tools

Again capitalizing on the HardFP port work previously done, we re-used and modified as necessary several scripts written earlier by DJ Delorie. These scripts are stored in a git repository and can be retrieved as follows:

$ git clone http://fedorapeople.org/groups/armv8/bootstrap.git

Patches are of course welcome.

Short term git problem
If the URI above does not work, please try:

$ git clone git://fedorapeople.org/~ahs3/bootstrap.git

Stage 1 - make

Status: mostly complete

Stage 1 was an initial bootstrap using cross compilation from source (not from SRPMs) on x86_64 hosts, for a minimal set of aarch64 packages. The goal was to create a minimal chroot environment which was sufficient for building the more complex packages of stage2. This implies providing native versions of GCC, binutils, glibc, plus some basic tools such as gzip, patch and awk, along with a few others. Stage1 results include these tools and a BusyBox environment (in lieu of requiring the use of a chroot, which is another possibility, using the OpenEmbedded Linaro images and running the rootfs within those, for example) that provides most of the normal and expected Linux commands. It does not include packages beyond the very minimal set required to get to a native building stage (still without RPM).

If you have access to cross-compilers for ARMv8 in your PATH, and those cross-compilers are named aarch64-redhat-linux-gnu-*, you should be able to recreate all of stage1 with the following:

$ git clone http://fedorapeople.org/groups/armv8/bootstrap.git
$ cd bootstrap
$ mkdir f17-local
$ scp username@fedorapeople.org:/project/armv8/SRPMs/f17/changed/* f17-local
$ ln -s f17-local SRPMlocal
$ mkdir f17-sources
$ scp username@fedorapeople.org:/project/armv8/SRPMs/f17/unchanged/* f17-sources
$ ln -s f17-sources SRPMs
$ ./stage1

If you do not have cross-compilers with names of the form given above, you will need to edit the stage1 script to point at the right places.

Further information:

• Stage1 Notes
• Task list:
  • [ahs3] Package and make available the Linaro aarch64 cross-toolchain [INPROGRESS]
  • Kernel needs to be brought up to date -- either upstream 3.7 with patches, Linaro AArch64, or something similar [TODO]
  • Binutils has had several aarch64 patches added in the last few weeks [TODO]
  • GCC patches should be re-examined for possible updates [TODO]
  • Glibc patches probably need to be updated to latest upstream [TODO]
  • Investigate and resolve additional rpath info needed for cloog. Why? [TODO]
  • Investigate and resolve additional rpath info needed for libselinux. Why? [TODO]
  • Investigate and resolve additional rpath info needed for sed. Why? [TODO]
  • [ahs3] Make sure any changes made go back into upstream, where appropriate [INPROGRESS]
  • [jcm, ahs3] Resolve cross-distribution compatibility issues, switch fully to /lib64; now rebuilding with updated toolchain [DONE]
  • [jcm] Resolve issues with architecture features in binutils/toolchain

Stage 2 - rpmbuild

Status: DONE

Stage 2 has the goal of getting a working native build of RPM (rpmbuild) running within the rootfs. This includes enough packages built from source (not from SRPMs, but on the target) to get a working "rpmbuild" binary sufficient to rebuild RPM and other packages. The output from this is the ability to build native RPM packages.

Further information:

• Stage2 Notes
• How To Use the ARM Foundation Model -- instructions for using what is currently the only publicly available platform simulator for ARMv8 with the rootfs built in stage1 (which is how one builds stage2 results)
• Task list (these all need to be built in dependency order):
  • [ahs3] publish patchify scripts [DONE] -- see git://fedorapeople.org/~ahs3/patchify.git
  • [ahs3] Gdb FTBFS -- newer upstream patches available? NB: latest rebuild worked fine for no obvious reason [DONE]
  • [ahs3] Upstream patches? Perl, nspr, to date [INPROGRESS]
  • [ahs3] Gettext FTBFS [DONE]
  • [ahs3] Build chkconfig [DONE]
  • [ahs3] Build sqlite [DONE]
  • [ahs3] Build pth [DONE]
  • [ahs3] Build openssl [DONE]
  • [ahs3] Build popt [DONE]
  • [ahs3] Build libffi [DONE]
  • [ahs3] Build pkgconfig [DONE]
  • [ahs3] Build glib2 [DONE]
  • [ahs3] Build nss-util [DONE]
  • [ahs3] Build nss-softokn [DONE]
  • [ahs3] Build nss [DONE]
  • [msalter] Build elfutils [DONE]
  • [msalter] Build rsync [DONE]
  • [msalter] Build fakechroot [DONE]
  • [msalter] Build file [DONE]
  • [msalter] Build ustr [DONE]
  • [msalter] Build libusb1 [DONE]
  • [msalter] Build libusb [DONE]
  • [msalter] Build tzdata [DONE]
  • [msalter] Build redhat-rpm-config DONE]
  • [msalter] Build rpm [DONE]
  • [msalter] Build cracklib [DONE]
  • [msalter] Build libsepol [DONE]
  • [msalter] Build attr [DONE]
  • [msalter] Build acl [DONE]
  • [msalter] Build libsemanage [DONE]
  • [msalter] Build pam [DONE]
  • [msalter] Build shadow-utils [DONE]
  • [msalter] Build strace [DONE]
  • [msalter] Build filesystem [DONE]
  • [msalter] Build iso-codes [DONE]

Stage 3 - mock

Status: Done. Not all packages built but close enough to start building with mock.

Stage 3 is the effort to get a minimal set of useful buildroot RPMs sufficient to rebuild the buildroot and have a working yum, mock configuration. RPMs will be manually built with rpmbuild and uploaded to a shared central rootfs repository.

At stage3, it may be necessary to intentionally disable certain unnecessary features of RPMs, and GNU autotools (and so on) will automatically exclude certain features dependent on non-present functionality. Consequently, these RPMs will not be final and must be rebuilt in the following stages.

Further information:

• Stage3 Notes

Stage3 rootfs currently contains dependencies needed to run mock. Current effort is focused on adding packages needed in the mock chroot. Complicating this is a desire to update to glibc 2.17 and gcc 4.8 which are the first upstream releases with AArch64 support. See glibc notes in Stage3 Notes for details. Because of switch to glibc 2.17, all previously built stage3 packages need to be rebuilt. Also, packages built without rpmbuild will need to be rebuilt as RPMs so they can be part of the stage4 repo used by mock. In parallel with building packages needed for base mock chroot, some packages are being built in an effort to be able to build documentation. This generally needs a number of X packages, etc. These packages are not strictly necessary but will allow more packages to be built without patches to avoid docs.

The current rootfs still uses busybox for /sbin/init and is a general mess because of the nature of the bootstrap. Packages have been installed with --nodeps and --force and in non-standard order. The rootfs does include a build of systemd but /sbin/init is still linked to busybox. Using the systemd /sbin/init does boot to a login prompt but the login fails for some reason. This could be something simple or something more complicated due to missing packages or other issues with the rootfs. It may not be worth fixing in the current rootfs because we have, or almost have, enough packages to replace the current root with one created by a yum install of aarch64 RPMs.

• List of desired packages:
  • [msalter] kernel-headers [DONE]
  • [msalter] glibc-2.17 [DONE]
  • [msalter] gcc-4.8 [DONE]
  • [msalter] filesystem [DONE]
  • [msalter] bzip2 [DONE]
  • [msalter] coreutils [DONE]
  • [msalter] bash [DONE]
  • [msalter] cpio [DONE]
  • [msalter] swig [DONE]
  • [msalter] libcap-ng [DONE]
  • [msalter] audit [DONE]
  • [msalter] shadow-utils [DONE]
  • [msalter] diffutil [DONE]
  • [msalter] dbus [DONE]
  • [msalter] libxslt [DONE]
  • [msalter] libutempter [DONE]
  • [msalter] util-linux [DONE]
  • [msalter] opensp [DONE]
  • [msalter] openjade [DONE]
  • [msalter] glib2 [DONE]
  • [msalter] ncurses [DONE]
  • [msalter] acl [DONE]
  • [msalter] attr [DONE]
  • [msalter] byacc [DONE]
  • [msalter] bison [DONE]
  • [msalter] zlib [DONE]
  • [msalter] chrpath [DONE]
  • [msalter] check [DONE]
  • [msalter] cracklib [DONE]
  • [msalter] curl [DONE]
  • [msalter] deltarpm [DONE]
  • [msalter] e2fsprogs [DONE]
  • [msalter] binutils [DONE]
  • [msalter] elfutils [DONE]
  • [msalter] expat [DONE]
  • [msalter] fipscheck [DONE]
  • [msalter] gawk [DONE]
  • [msalter] procps-ng [DONE]
  • [msalter] make [DONE]
  • [msalter] libICE [DONE]
  • [msalter] libSM [DONE]
  • [msalter] libXau [DONE]
  • [msalter] gettext [DONE]
  • [msalter] gnupg2 [DONE]
  • [msalter] libxcb [DONE]
  • [msalter] libXdmcp [DONE]
  • [msalter] gpgme [DONE]
  • [msalter] texinfo [DONE]
  • [msalter] keyutils [DONE]
  • [msalter] libX11 [DONE]
  • [msalter] freetype [DONE]
  • [msalter] libXt [DONE]
  • [msalter] krb5 with ldap [DONE]
  • [msalter] libarchive [DONE]
  • [msalter] libassuan [DONE]
  • [msalter] libcap [DONE]
  • [msalter] libcroco [DONE]
  • [msalter] fontconfig [DONE]
  • [msalter] libXrender [DONE]
  • [msalter] libXft [DONE]
  • [msalter] libXext [DONE]
  • [msalter] libXfixes [DONE]
  • [msalter] libXdamage [DONE]
  • [msalter] libdb [DONE]
  • [msalter] libXmu [DONE]
  • [msalter] libgcrypt [DONE]
  • [msalter] libgpg-error [DONE]
  • [msalter] tk [DONE]
  • [msalter] libksba [DONE]
  • [msalter] gdbm [DONE]
  • [msalter] libpng [DONE]
  • [msalter] libsepol [DONE]
  • [msalter] readline [DONE]
  • [msalter] libtool [DONE]
  • [msalter] libffi [DONE]
  • [msalter] libunistring [DONE]
  • [msalter] libuser [DONE]
  • [msalter] libxml2 [DONE]
  • [msalter] lua [DONE]
  • [msalter] lzo [DONE]
  • [msalter] m4 [DONE]
  • [msalter] nss-util [DONE]
  • [msalter] nss-softokn [DONE]
  • [msalter] ruby [DONE]
  • [msalter] nss [DONE]
  • [msalter] libselinux [DONE]
  • [msalter] popt [DONE]
  • [msalter] qrencode [DONE]
  • [msalter] pciutils [DONE]
  • [msalter] pth [DONE]
  • [msalter] usbutils [DONE]
  • [msalter] sed [DONE]
  • [msalter] gperf [DONE]
  • [msalter] sharutils [DONE]
  • [msalter] pigz [DONE]
  • [msalter] pkgconfig [DONE]
  • [msalter] systemd [DONE]
  • [msalter] pcre [DONE]
  • [msalter] libidn [DONE]
  • [msalter] imake [DONE]
  • [msalter] gc [DONE]
  • [msalter] cmake [DONE]
  • [msalter] wayland [DONE]
  • [msalter] libpciaccess [DONE]
  • [msalter] openssl [DONE]
  • [msalter] cyrus-sasl [DONE]
  • [msalter] openldap [DONE]
  • [msalter] libXpm [DONE]
  • [msalter] libjpeg-turbo [DONE]
  • [msalter] gd [DONE]
  • [msalter] xz [DONE]
  • [msalter] patch [DONE]
  • [msalter] tar [DONE]
  • which
  • [msalter] chkconfig [DONE]
  • cloog
  • [msalter] llvm [DONE]
  • [msalter] dwz [DONE]
  • [msalter] file [DONE]
  • gdb
  • [msalter] gnutls [DONE]
  • kmod
  • [msalter] libssh2 [DONE]
  • p11-kit
  • [msalter] pam [DONE]
  • [msalter] perl [DONE]
  • pinentry
  • policycoreutils
  • [msalter] python [DONE]
  • [msalter] shared-mime-info [DONE]
  • [msalter] sqlite [DONE]
  • [msalter] tcp_wrappers [DONE]
  • [msalter] ustr [DONE]
  • ? others ?

Stage 4 - distribution bootstrap

Status: Done.

At Stage 4 we will run a complete mock rebuild of the distribution, identifying and solving dependency and platform build issues. With those out of the way, we will switch over to koji while waiting for the koji and yum changes needed to correctly support the new architecture type. This stage will absolutely require the presence of real ARMv8 hardware for completion.

The result of Stage 4 is threefold:

1. The set of packages needed for running Koji
2. A set of SRPMs which actually can build some form of F19, or sufficiently large part of it.
3. A set of binary RPMs that can be used initially by koji while doing the first koji rebuild of the distribution

Further information:

• AArch64 Quickstart
• Stage 4 Patched Packages
• Stage 4 Problem Packages

Stage 5

Status: Done

Complete rebuild of the distribution using koji

Stage 6

Status: Done

At this point we should have koji-shadow running automatically shadowing the main koji.

Getting Involved

There are various ways you can get involved. If in doubt, visit the Fedora ARM IRC (#fedora-arm on irc.freenode.net) and ask one of us for advice.

Who's involved?

• Al Stone
• Jon Masters
• Chris Tyler
• DJ Delorie
• David Marlin
• Dennis Gilmore
• Brendan Conoboy
• Peter Robinson
• Chen Baozi
• Mark Salter
• Jaromír Cápík
• Paul Whalen
• Your Name Could Go Here!

Task Lists

In the sections above are task lists for the various stages. These are the ones we currently know about. If you have further suggestions, let us know on IRC or the fedora-arm mailing list where we can discuss them and figure out how they fit into the overall scheme of things. As part of that discussion, we'll decide if and how it gets added to the lists. As you'll notice, there is a marker in front of each task (well, sometimes) and a marker at the end of each task (always). The marker at the end should be one of TODO (meaning the task has not been started by anyone), INPROGRESS (implying someone is actively working on it), or DONE (um, well, the task is done :). If the task is INPROGRESS or DONE, there should be a marker at the front indicating who is doing the work (e.g., ahs3 is Al Stone's nick on IRC).

If you're looking for something to do, find a task that is marked TODO, and edit this page to show that you're working on it. If someone has put their nick on the task, please ask them first and share nicely like grown-ups. For example, change the line:

   * grizzle the fingo [TODO]

to this:

   * [myNick] grizzle the fingo [INPROGRESS]

And when you've completed the task, change it to:

   * [myNick] grizzle the fingo [DONE]

Git Based Rootfs Workflow

For stages 1 through 3, we are coordinating work using the git repository at http://fedorapeople.org/groups/armv8/rootfs.git.

IMPORTANT: Please do work using only the ARM Foundation model. While cross-compilers are possible, we are trying to stay as close as possible to doing a target-based (i.e., real hardware) bring up at this time. We use cross-compilation in stage1, but we are trying to be very close to the "Fedora way" of operating, which is build on target. For the time being, the ARM Foundation model is as good as it gets. As soon as hardware becomes available, we will make this more stringent and insist all stage2 and later work being done on real hardware.

IMPORTANT: Build flags, and similar options are defined in the stage1 and stage2 scripts. In stage1/stage2, please do not enable or alter architecture features that are not required for minimal bootstrap, and especially DO NOT do not change build flags and such as those that are defined globally in the RPM packages (rpm, redhat-rpm-config, fedora-release...).

• http://fedorapeople.org/groups/armv8/rootfs.git -- just pull the master branch

Our workflow is centered around the above git repository -- a copy of the current state of the rootfs. You will need to clone this git repository and then build a disk image from it; instructions can be found in How To Use the ARM Foundation Model. If you can do so and make your git clone publicly visible, you can then send pull requests to the fedora-arm list for review and incorporation. If you cannot make your clone publicly visible, please send a git diff of the SRPM changes made to the fedora-arm list; after review and incorporation, we will build from the source changes and incorporate the binaries back into the rootfs also (sending the binaries to the list would just be painful).

Using the ARMv8 Models

ARM provides two different platform simulators that can be used to run AArch64 binaries; the FAST model is under strict license and not generally available. The Foundation model is binary only, but still accessible to anyone who wishes to use it (see http://www.arm.com/fvp). Instructions for using the models are here:

• Using the ARMv8 Model with an NFS Root
• Using the ARMv8 Model with a Disk Image

Getting one or the other of these working is a prerequisite for assisting in this effort.

Kernel command lines are hard-coded
In both of the cases above, the models use a '.axf' file. This file provides a basic boot environment for the model. However, the kernel parameters in the .axf file are hard-coded when it is created. If you need to change them, you can either ask on #fedora-arm and if someone has time, they can build you a custom one. Or, the more straightforward (and likely quicker) way is to create your own custom .axf using the patch tool found at git://fedorapeople.org/~ahs3/patch-axf.git. Do a git clone of that repository and follow the instructions in the README.

Assisting with Stage1

The results of the first three stages of the bootstrap are being kept in a rapidly growing and quite large git repository that can retrieved via:

$ git clone http://fedorapeople.org/groups/armv8/rootfs.git

This was seeded with stage1 results and the current stage2 results that were built using the scripts found here:

$ git clone http://fedorapeople.org/groups/armv8/bootstrap.git

All of the source that was used can be found at http://fedorapeople.org/groups/armv8/SRPMs/f17. The directory unchanged contains the SRPMs that are unchanged from the source one can get from the Fedora 17 sources. The directory changed contains all the Fedora 17 (or in some cases, 18) SRPMs that had to change to build.

If you have access to cross-compilers for ARMv8 in your PATH, and those cross-compilers are named aarch64-redhat-linux-gnu-*, you should be able to recreate all of stage1 with the following:

$ git clone http://fedorapeople.org/groups/armv8/bootstrap.git
$ cd bootstrap
$ mkdir f17-local
$ scp username@fedorapeople.org:/project/armv8/SRPMs/f17/changed/* f17-local
$ ln -s f17-local SRPMlocal
$ mkdir f17-sources
$ scp username@fedorapeople.org:/project/armv8/SRPMs/f17/unchanged/* f17-sources
$ ln -s f17-sources SRPMs
$ ./stage1

If you do not have cross-compilers with names of the form given above, you will need to edit the stage1 script to point at the right places.

If you wish to contribute changes to the stage1 SRPMs or the bootstrap scripts, please submit either a git diff for the package needing to be changed, or a git pull request from your clone of the repository, to the fedora-arm mailing list for review and ultimately incorporation into the proper git repository by the maintainer (ahs3).

NB: better access to RPMs for the x86_64 cross-toolchain for aarch64 is being worked on. In the meantime, tools can be found at Linaro.

Assisting with Stage2

Stage2 builds upon the rootfs originally created in stage1 and found here:

$ git clone http://fedorapeople.org/groups/armv8/rootfs.git

Short term git problem
If for some reason the URI above does not work, please try:

$ git clone git://fedorapeople.org/~ahs3/rootfs.git

To use this rootfs as a native build environment, we have to use the platform simulator provided by ARM -- the ARM Foundation model. Instructions on using the model with the rootfs are found here: How To Use the ARM Foundation Model. Once you have booted the model with the rootfs, you'll be presented with a console running BusyBox. From there, running stage2 is very straightforward:

$ cd /stage2
$ ./stage2

The stage2 script uses makefile snippets defined /stage2/recipe.d and stage2/recipe.mk to order which packages get built in what order. You can modify that order by either (a) cheating and marking something as done by creating the proper file in /stage2/done (e.g., 'touch /stage2/done/gdb' lets stage2 continue past the build for gdb even though it currently fails), or (b) the more difficult route of modifying the makefile snippets directly.

If you wish to contribute changes to the stage2 SRPMs or the bootstrap scripts, please submit a git diff for the package needing to be changed, or a git pull request from your clone of the repository, to the fedora-arm mailing list for review and ultimately incorporation into the proper git repository by the maintainer (ahs3).

Subsequent Stages

Stage3 will look a lot like a larger stage2, and hopefully have more hands involved. And while it is okay at this point to disable sub-packages like "docs", or to turn off functionality like selinux, or audit requirements, do not disable features that will be required by other packages directly in order to get a minimally working mock buildroot established. If in doubt, ask on #fedora-arm for advice.

Starting in stage4, we will be able to drop the use of the rootfs from git and work from real images. Stages 4-6 will start to look more like a normal Fedora release, hopefully.