Architectures/ARM/AArch64

Have a question? Join the Fedora ARM team on IRC in #fedora-arm on Freenode

Introduction

ARM chips have been wildly successful in embedded applications, cell phones and in tablet devices, but until recently the ARM Architecture was able only to address 32-bits of physical memory. As part of a push into the nascent ARM server (Hyperscale) market (as well as for use in more complex traditional embedded applications), ARM have extended the 32-bit Architecture to support 40-bit physical addressing in the Cortex-A15 processor. This was a big step, and several vendors have shipped successful 32-bit ARM server platforms based upon the LPAE 40-bit technology within the Cortex-A15, the older Cortex-A9 processor, and also using custom designs conforming to the 32-bit architecture.

A broader push into the Enterprise space requires the introduction of a 64-bit ARM Architecture. This happens with the introduction of the new AArch64 execution state in version 8 of the ARM Architecture. AArch64 is not a pure extension of the 32-bit ARM Architecture, but is rather a new architecture state within ARMv8, complete with a new A64 instruction set. It is born out of many years of research into modern architecture design. Importantly, AArch64 as a separate execution state means that some future processors may not support the older AArch32 execution state. Although initial 64-bit ARM processors will be fully backwards compatible, we are taking the bold and forward-looking step of treating AArch64 as the only execution state on ARMv8 processors. We will not be supporting 32-bit execution on these systems. This will allow for many beneficial implementation trade-offs to be made, such as the use of a larger 64K page size by default, and will result in 64-bit clean ARM server systems free from legacy code. It is important to make this distinction immediately such that there is an opportunity for 64-bit only server systems to emerge within the next few years. Not having to implement a full 32-bit pipeline in addition to the newer 64-bit architecture will increase energy efficiency in future ARM server systems. Think of AArch64 as part of a natural progression of supported ARM Architectures in the Fedora ARM project: armv5tel, armv7hl, aarch64. The new architecture is named aarch64 consistent with ARM's own choice of upstream naming, and in respect of ARM's desires to separate the ARM trademark from the architecture name.

You can find out more about the | ARMv8 Architecture on ARM's website.

This page documents the initiative to bring Fedora support to the AArch64 execution state within the ARMv8 architecture, starting with the bootstrapping of Fedora 17. Future releases of Fedora will support both ARM-implemented 64-bit processor cores - such as the new Cortex-A57 - as well as custom 64-bit cores from third parties like Applied Micro. A demonstration of the initial bootstrap has already been given on the latter hardware, expected to be available in 2013. The goal is to support an initial Fedora 19 remix on real ARMv8 silicon, with a more complete Fedora 20 release. As soon as we are able to acquire production servers for PHX, we will also establish a formal ARMv8 build system in Phoenix. Until that time, Red Hat will internally host the initial ARMv8 build hardware, once it becomes available. In the initial bootstrap, the software Foundation Models are used, as described below. We will also collaborate with both Linaro and the Linaro Enterprise Group, as part of a broader Linux community effort to support the new 64-bit architecture.

Bootstrap Process

Essentially, AArch64 bootstrap is following the same process as that used for Fedora 15 on ARM: ARM HardFP Bootstrap. The older F15 process was intentionally designed with AArch64 bootstrap in mind, and so this similarity is also intended.

The purpose of the effort is to enable Fedora on ARMv8, using the standard 64-bit ABI as defined by ARM. The good news is that unlike ARMv7 and earlier, there is only one ABI (see AAPCS for 64-bit).

Everything Is Saved In One Place
Please note that all of the files and repositories for the ARMv8 bootstrap work are stored at http://fedorapeople.org/groups/armv8. Contact the maintainer Al Stone if you need to have write access.

Goals

Our goals are to complete the bootstrap "stages" (1,2,3,4,5,6) at which we anticipate certain levels of functionality from the root filesystem, beginning with very basic cross compilation, proceeding to native builds, and ultimately to a working mock and Koji configuration able to build production packages.

CURRENT STAGE: We have completed stage1, and are progressing through stage2.

Bootstrap Tools

Again capitalizing on the HardFP port work previously done, we re-used and modified as necessary several scripts written earlier by DJ Delorie. These scripts are stored in a git repository and can be retrieved as follows:

$ git clone http://fedorapeople.org/groups/armv8/bootstrap.git

Patches are of course welcome.

Short term git problem
If the URI above does not work, please try:

$ git clone git://fedorapeople.org/~ahs3/bootstrap.git

Stage 1 - make

Status: mostly complete

Stage 1 was an initial bootstrap using cross compilation from source (not from SRPMs) on x86_64 hosts, for a minimal set of aarch64 packages. The goal was to create a minimal chroot environment which was sufficient for building the more complex packages of stage2. This implies providing native versions of GCC, binutils, glibc, plus some basic tools such as gzip, patch and awk, along with a few others. Stage1 results include these tools and a BusyBox environment (in lieu of requiring the use of a chroot, which is another possibility, using the OpenEmbedded Linaro images and running the rootfs within those, for example) that provides most of the normal and expected Linux commands. It does not include packages beyond the very minimal set required to get to a native building stage (still without RPM).

If you have access to cross-compilers for ARMv8 in your PATH, and those cross-compilers are named aarch64-redhat-linux-gnu-*, you should be able to recreate all of stage1 with the following:

$ git clone http://fedorapeople.org/groups/armv8/bootstrap.git
$ cd bootstrap
$ mkdir f17-local
$ scp username@fedorapeople.org:/project/armv8/SRPMs/f17/changed/* f17-local
$ ln -s f17-local SRPMlocal
$ mkdir f17-sources
$ scp username@fedorapeople.org:/project/armv8/SRPMs/f17/unchanged/* f17-sources
$ ln -s f17-sources SRPMs
$ ./stage1

If you do not have cross-compilers with names of the form given above, you will need to edit the stage1 script to point at the right places.

Further information:

• Stage1 Notes
• Task list:
  • [ahs3] Package and make available the Linaro aarch64 cross-toolchain [INPROGRESS]
  • Kernel needs to be brought up to date -- either upstream 3.7 with patches, Linaro AArch64, or something similar [TODO]
  • Binutils has had several aarch64 patches added in the last few weeks [TODO]
  • GCC patches should be re-examined for possible updates [TODO]
  • Glibc patches probably need to be updated to latest upstream [TODO]
  • Investigate and resolve additional rpath info needed for cloog. Why? [TODO]
  • Investigate and resolve additional rpath info needed for libselinux. Why? [TODO]
  • Investigate and resolve additional rpath info needed for sed. Why? [TODO]
  • [ahs3] Make sure any changes made go back into upstream, where appropriate [INPROGRESS]
  • [jcm, ahs3] Resolve cross-distribution compatibility issues, switch fully to /lib64; now rebuilding with updated toolchain [DONE]
  • [jcm] Resolve issues with architecture features in binutils/toolchain

Stage 2 - rpmbuild

Status: in progress

Stage 2 has the goal of getting a working native build of RPM (rpmbuild) running within the rootfs. This includes enough packages built from source (not from SRPMs, but on the target) to get a working "rpmbuild" binary sufficient to rebuild RPM and other packages. The output from this is the ability to build native RPM packages.

Further information:

• Stage2 Notes
• How To Use the ARM Foundation Model -- instructions for using what is currently the only publicly available platform simulator for ARMv8 with the rootfs built in stage1 (which is how one builds stage2 results)
• Task list (these all need to be built in dependency order):
  • [ahs3] publish patchify scripts [DONE] -- see git://fedorapeople.org/~ahs3/patchify.git
  • [ahs3] Gdb FTBFS -- newer upstream patches available? NB: latest rebuild worked fine for no obvious reason [DONE]
  • [ahs3] Upstream patches? Perl, nspr, to date [INPROGRESS]
  • [ahs3] Gettext FTBFS [DONE]
  • [ahs3] Build chkconfig [DONE]
  • [ahs3] Build sqlite [DONE]
  • [ahs3] Build pth [DONE]
  • [ahs3] Build openssl [DONE]
  • [ahs3] Build popt [DONE]
  • [ahs3] Build libffi [DONE]
  • [ahs3] Build pkgconfig [DONE]
  • [ahs3] Build glib2 [DONE]
  • [ahs3] Build nss-util [DONE]
  • [ahs3] Build nss-softokn [DONE]
  • [ahs3] Build nss [INPROGRESS]
  • [msalter] Build elfutils [BUILT - not in git tree]
  • [msalter] Build rsync [BUILT - not in git tree]
  • [msalter] Build fakechroot [BUILT - not in git tree]
  • [msalter] Build file [BUILT - not in git tree]
  • [msalter] Build ustr [BUILT - not in git tree]
  • [msalter] Build libusb1 [BUILT - not in git tree]
  • [msalter] Build libusb [BUILT - not in git tree]
  • [msalter] Build tzdata [BUILT - not in git tree]
  • [msalter] Build redhat-rpm-config [INPROGRESS]
  • Build rpm [TODO]
  • Build cracklib [TODO]
  • Build libsepol [TODO]
  • Build attr [TODO]
  • Build acl [TODO]
  • Build libsemanage [TODO]
  • Build pam [TODO]
  • Build shadow-utils [TODO]
  • Build strace [TODO]
  • Build filesystem [TODO]
  • Build iso-codes [TODO]
  • Build audit [TODO]
  • Build gnupg [TODO]
  • Final re-build of rpm [TODO]

Stage 3 - mock

Status: TODO

Stage 3 is the effort to get a minimal set of useful buildroot RPMs sufficient to rebuild the buildroot and have a working yum, mock configuration. RPMs will be manually built with rpmbuild and uploaded to a shared central rootfs repository.

At stage3, it may be necessary to intentionally disable certain unnecessary features of RPMs, and GNU autotools (and so on) will automatically exclude certain features dependent on non-present functionality. Consequently, these RPMs will not be final and must be rebuilt in the following stages.

Stage 4 - distribution bootstrap

Status: TODO

At Stage 4 we will run a complete mock rebuild of the distribution, identifying and solving dependency and platform build issues. With those out of the way, we will switch over to koji while waiting for the koji and yum changes needed to correctly support the new architecture type. This stage will absolutely require the presence of real ARMv8 hardware for completion.

The result of stage4 is threefold:

1. The set of packages needed for running Koji
2. A set of SRPMs which actually can build some form of F15, or sufficiently large part of it.
3. A set of binary RPMs that can be used initially by koji while doing the first koji rebuild of the distribution

Stage 5

Status: TODO

Complete rebuild of the distribution using koji

Stage 6

Status: TODO

At this point we should have koji-shadow running automatically shadowing the main koji.

Getting Involved

There are various ways you can get involved. If in doubt, visit the Fedora ARM IRC (#fedora-arm on irc.freenode.net) and ask one of us for advice.

Who's involved?

• Al Stone
• Jon Masters
• Chris Tyler
• DJ Delorie
• David Marlin
• Dennis Gilmore
• Brendan Conoboy
• Peter Robinson
• Chen Baozi
• Mark Salter
• Your Name Could Go Here!

Task Lists

In the sections above are task lists for the various stages. These are the ones we currently know about. If you have further suggestions, let us know on IRC or the fedora-arm mailing list where we can discuss them and figure out how they fit into the overall scheme of things. As part of that discussion, we'll decide if and how it gets added to the lists. As you'll notice, there is a marker in front of each task (well, sometimes) and a marker at the end of each task (always). The marker at the end should be one of TODO (meaning the task has not been started by anyone), INPROGRESS (implying someone is actively working on it), or DONE (um, well, the task is done :). If the task is INPROGRESS or DONE, there should be a marker at the front indicating who is doing the work (e.g., ahs3 is Al Stone's nick on IRC).

If you're looking for something to do, find a task that is marked TODO, and edit this page to show that you're working on it. If someone has put their nick on the task, please ask them first and share nicely like grown-ups. For example, change the line:

   * grizzle the fingo [TODO]

to this:

   * [myNick] grizzle the fingo [INPROGRESS]

And when you've completed the task, change it to:

   * [myNick] grizzle the fingo [DONE]

Git Based Rootfs Workflow

For stages 1 through 3, we are coordinating work using the git repository at http://fedorapeople.org/groups/armv8/rootfs.git.

IMPORTANT: Please do work using only the ARM Foundation model. While cross-compilers are possible, we are trying to stay as close as possible to doing a target-based (i.e., real hardware) bring up at this time. We use cross-compilation in stage1, but we are trying to be very close to the "Fedora way" of operating, which is build on target. For the time being, the ARM Foundation model is as good as it gets. As soon as hardware becomes available, we will make this more stringent and insist all stage2 and later work being done on real hardware.

IMPORTANT: Build flags, and similar options are defined in the stage1 and stage2 scripts. In stage1/stage2, please do not enable or alter architecture features that are not required for minimal bootstrap, and especially DO NOT do not change build flags and such as those that are defined globally in the RPM packages (rpm, redhat-rpm-config, fedora-release...).

• http://fedorapeople.org/groups/armv8/rootfs.git -- just pull the master branch

Our workflow is centered around the above git repository -- a copy of the current state of the rootfs. You will need to clone this git repository and then build a disk image from it; instructions can be found in How To Use the ARM Foundation Model. If you can do so and make your git clone publicly visible, you can then send pull requests to the fedora-arm list for review and incorporation. If you cannot make your clone publicly visible, please send a git diff of the SRPM changes made to the fedora-arm list; after review and incorporation, we will build from the source changes and incorporate the binaries back into the rootfs also (sending the binaries to the list would just be painful).

Using the ARMv8 Models

ARM provides two different platform simulators that can be used to run AArch64 binaries; the FAST model is under strict license and not generally available. The Foundation model is binary only, but still accessible to anyone who wishes to use it (see http://www.arm.com/fvp). Instructions for using the models are here:

• Using the ARMv8 Model with an NFS Root
• Using the ARMv8 Model with a Disk Image

Getting one or the other of these working is a prerequisite for assisting in this effort.

Kernel command lines are hard-coded
In both of the cases above, the models use a '.axf' file. This file provides a basic boot environment for the model. However, the kernel parameters in the .axf file are hard-coded when it is created. If you need to change them, you can either ask on #fedora-arm and if someone has time, they can build you a custom one. Or, the more straightforward (and likely quicker) way is to create your own custom .axf using the patch tool found at git://fedorapeople.org/~ahs3/patch-axf.git. Do a git clone of that repository and follow the instructions in the README.

Assisting with Stage1

The results of the first three stages of the bootstrap are being kept in a rapidly growing and quite large git repository that can retrieved via:

$ git clone http://fedorapeople.org/groups/armv8/rootfs.git

This was seeded with stage1 results and the current stage2 results that were built using the scripts found here:

$ git clone http://fedorapeople.org/groups/armv8/bootstrap.git

All of the source that was used can be found at http://fedorapeople.org/groups/armv8/SRPMs/f17. The directory unchanged contains the SRPMs that are unchanged from the source one can get from the Fedora 17 sources. The directory changed contains all the Fedora 17 (or in some cases, 18) SRPMs that had to change to build.

If you have access to cross-compilers for ARMv8 in your PATH, and those cross-compilers are named aarch64-redhat-linux-gnu-*, you should be able to recreate all of stage1 with the following:

$ git clone http://fedorapeople.org/groups/armv8/bootstrap.git
$ cd bootstrap
$ mkdir f17-local
$ scp username@fedorapeople.org:/project/armv8/SRPMs/f17/changed/* f17-local
$ ln -s f17-local SRPMlocal
$ mkdir f17-sources
$ scp username@fedorapeople.org:/project/armv8/SRPMs/f17/unchanged/* f17-sources
$ ln -s f17-sources SRPMs
$ ./stage1

If you do not have cross-compilers with names of the form given above, you will need to edit the stage1 script to point at the right places.

If you wish to contribute changes to the stage1 SRPMs or the bootstrap scripts, please submit either a git diff for the package needing to be changed, or a git pull request from your clone of the repository, to the fedora-arm mailing list for review and ultimately incorporation into the proper git repository by the maintainer (ahs3).

NB: better access to RPMs for the x86_64 cross-toolchain for aarch64 is being worked on. In the meantime, tools can be found at Linaro.

Assisting with Stage2

Stage2 builds upon the rootfs originally created in stage1 and found here:

$ git clone http://fedorapeople.org/groups/armv8/rootfs.git

Short term git problem
If for some reason the URI above does not work, please try:

$ git clone git://fedorapeople.org/~ahs3/rootfs.git

To use this rootfs as a native build environment, we have to use the platform simulator provided by ARM -- the ARM Foundation model. Instructions on using the model with the rootfs are found here: How To Use the ARM Foundation Model. Once you have booted the model with the rootfs, you'll be presented with a console running BusyBox. From there, running stage2 is very straightforward:

$ cd /stage2
$ ./stage2

The stage2 script uses makefile snippets defined /stage2/recipe.d and stage2/recipe.mk to order which packages get built in what order. You can modify that order by either (a) cheating and marking something as done by creating the proper file in /stage2/done (e.g., 'touch /stage2/done/gdb' lets stage2 continue past the build for gdb even though it currently fails), or (b) the more difficult route of modifying the makefile snippets directly.

If you wish to contribute changes to the stage2 SRPMs or the bootstrap scripts, please submit a git diff for the package needing to be changed, or a git pull request from your clone of the repository, to the fedora-arm mailing list for review and ultimately incorporation into the proper git repository by the maintainer (ahs3).

Subsequent Stages

Stage3 will look a lot like a larger stage2, and hopefully have more hands involved. And while it is okay at this point to disable sub-packages like "docs", or to turn off functionality like selinux, or audit requirements, do not disable features that will be required by other packages directly in order to get a minimally working mock buildroot established. If in doubt, ask on #fedora-arm for advice.

Starting in stage4, we will be able to drop the use of the rootfs from git and work from real images. Stages 4-6 will start to look more like a normal Fedora release, hopefully.

Target flags

No special flags are required at this time.

Hardware floating point

Hardware floating point in ARMv8 processors is considered standard and not optional. Hence, no special options are required.