Architectures/ARM/Fedora15 HardFP Bootstrap

NOTE: This is a page in progress. Skip ahead to the section describing current stage4/5 bootstrap work if you are interested in helping out.

We are currently engaged in bootstrap of support for an ARMv7 (armv7hl) "hardfp" (hardware floating point) version of Fedora 15. In addition to adding support for hardware floating point, this port makes a transition to newer variant of the ARM ABI (sometimes referred to as a new variant of the "EABI" informally, though the name "EABI" does not exist any more in official terms for ARM systems), and in particular to a newer variant of the Procedure Calling Standard for ARM Architecture AAPCS. The newer ABI (as defined in section 6 of the AAPCS) uses floating point registers as part of the procedure calling interface. We will treat "armv7hl" (hardfp, little endian) as a new target architecture. As such, it is not supported to install both armv5tel (ARMv5) and armv7hl (ARMv7) packages on the same Fedora 15 system. multilib may be an opportunity later on as the kernel can handle both, but at this stage we aim for handling them as separate architectures.

Important Links

• Package status: Fedora15 HardFP Bootstrap package status
• Latest VFAD: Monday November 14th 2011 (completed)

Goals

We have various goals within this bootstrapping project. Briefly, these fall under "stages" (1,2,3,4,5,6) at which we anticipate certain levels of functionality from the root filesystem, beginning at a level of cross compilation and proceeding to native builds, and ultimately to a working mock and Koji configuration able to build production packages.

CURRENT STAGE: We are at stage4.

Stage 1 - make

Status: complete

Stage 1 was an initial bootstrap using cross compilation (on x86_64 hosts) of a minimal set of armv7hl packages, from source (not from SRPMs), sufficient to have a minimal chroot environment from which gcc, binutils, and so forth were available for native builds. Stage 1 included gcc, binutils, glibc, etc. It did not include util-linux, or packages beyond the very minimal set required to get to a native building stage (still without RPM).

Stage 2 - rpmbuild

Status: complete

Stage 2 was an effort to get a working native build of RPM (rpmbuild) running within the rootfs. This included enough packages built from source (not from SRPMs, but on the target) to get a working "rpmbuild" binary sufficient to rebuild RPM and other packages. The output from this was an ability to build native RPM packages.

• Stage 1/2 bootstrap scripts: bootstrap.git repo

Stage 3 - mock

Status: complete

Stage 3 was an effort to get a minimal set of useful buildroot RPMs sufficient to rebuild the buildroot and have a working yum, mock configuration. RPMs were manually built with rpmbuild and uploaded to a shared central rootfs repository.

At stage3, it was necessary to intentionally disable certain unnecessary features of RPMs, and it is known that GNU autotools (and so on) will automatically exclude certain features dependent on non-present functionality. Consequently, these RPMs are not final and must be rebuilt in the following stages.

Stage 4 - distribution bootstrap

Status: mostly completed. finishing up a couple of loose ends left from stage3, and waiting for armv5tel to catch up

At Stage 4 we run a complete mock rebuild of the distribution, identifying & solving dependency & platform build issues before switching over to koji and while waiting for some koji & yum changes needed to correctly support the new architecture type.

To participate actively building packages you need a native ARMv7 system running Fedora 15 (use the rootfs below) and some experience with Fedora packaging. Please see

• rootfs http://scotland.proximity.on.ca/fedora-arm/f15hardfp/bootstrap/rootfs_stage4_20110825/ (installation instructions)
• latest builder script: http://arm-temp.ausil.us/pub/fedora-arm/arm-rebuild.sh
• notes & updates http://lists.fedoraproject.org/pipermail/arm/2011-September/001937.html
• Fedora15 HardFP Bootstrap package status

And please keep yum and arm-rebuild.sh updated on your build node.

To summarize: Install the rootfs either as a bootable OS for your arm platform, or as a chroot (make sure that you bind mount the /proc, /sys, and /dev filesystems at least - if you don't know what this means, please do not attempt the chroot based building approach). Within that rootfs, become the builder user and run the arm-rebuild.sh script in builder's home directory. If you have to manually build a package (i.e. arm-specific patches or newer srpms), see the stage 4 notes.

Even if you are not actively building packages please help documenting the nature of the build issues at Fedora15 HardFP Bootstrap package status which is mostly a matter of reading build.log or root.log and condensing the information in a more digestable way.

The result of stage4 is threefold

a) The set of packages needed for running Koji

b) A set of SRPMs which actually can build some form of F15, or sufficiently large part of it.

c) A set of binary RPMs that can be used initially by koji while doing the first koji rebuild of the distribution

Stage 5

Complete rebuild of the distribution using koji

Stage 6

At this point we should have koji-shadow running automatically shadowing the main koji.

Getting Involved

There are various ways you can get involved. If in doubt, visit the Fedora ARM IRC (#fedora-arm on irc.freenode.net) and ask one of us for advice.

Who's involved?

• Jon Masters
• Chris Tyler
• DJ Delorie
• Anthony Boccia
• Jon Chiappetta
• Paul Whalen
• Dennis Gilmore
• Brendan Conoboy
• Peter Robinson
• Henrik Nordström

Target flags

We will be supporting the vfpv3-d16 architecture level as required of the optional AAPCS section 6 hardware floating point calling convention ABI. We will not be requiring vfpv3-d32 since not all ARMv7 parts implement all 32 double floating point registers and they are not required for base hardfp ABI compatibility. We will build for a Cortex-A8 optimized target, not requiring NEON, and with Thumb2 instructions disabled. Due to the way in which AAELF (ELF bindings for ARM Architecture) and AAPCS, as well as Thumb interworking are implemented, we can safely assume that Thumb2 can be enabled at the individual package level later without breaking any compatibility, or requiring other packages to be rebuilt.

Hardware floating point

Hardware floating point in ARMv7 processors takes the form of the VFPv3 (Vector Floating Point version 3) co-processor, or the optional VFPv4 processor (that we do not target and will not be targeting), which is a modern vector processing unit unrelated to the original ARM FPU present or emulated in earlier ARM systems.

---

Stage 1-3 Notes

Warning: The project is no longer in stages 1, 2, or 3. Do not follow these instructions any more.

Stages 1 through 3 used a central git repository to hold the rapidly growing root file system. The initial filesystem was seeded by the stage 1 bootstrap. To obtain a copy of the rootfs and scripts:

• ssh://YOUR_FAS_USERNAME@git.fedorahosted.org/git/arm.git - armv7hl rootfs work in progress. This is suitable for use with "chroot" (after setting up a bind mount for /proc) from an existing ARM v7 compatible board running F13. If you are not in the gitarm FAS group within the Fedora account system, you can clone using anonymous git instead.
• git://fedorapeople.org/~djdelorie/bootstrap.git - upstream for stage1/stage2 scripts (we ultimately will sync the bits in the rootfs stage1/stage2 scripts here for future use).

Stage 3 dependency resolution & output

While it is sufficient to disable sub-packages like "docs", or to turn off functionality like selinux, audit requirements, it is not sufficient to disable features that will be required by other packages directly in order to get a minimally working mock buildroot established. If in doubt, ask on #fedora-arm for advice.

Output: RPMs and SRPMs in an armv7hl-YOUR_FAS_USERNAME bit branch of the armv7hl rootfs within the Fedora ARM git repo, stored in /stage3/armv7hl and later installed into the rootfs as part of merging by the maintainer (jonmasters). Mirrors of these packages are stored in a yum repo on scotland.proximity.on.ca/fedora-arm/armv7hl/

Git based rootfs workflow

IMPORTANT: Please only perform these steps on real hardware. While emulators and cross-compilers are possible, we are doing a target-based bring up at this time. If you would like to make suggestions, please contact us. We have discussed the topic of emulation and cross-compilation many times before. We use cross-compilation to a certain extent in stage1, but we are trying to be very close to the "Fedora way" of operating, which is build on target. Therefore, please refrain from performing these steps from within a QEMU emulator, for example.

IMPORTANT: Build flags, and similar options are defined in the stage1 and stage2 scripts. In stage1/stage2, please do not enable features that are not required for minimal bootstrap, and especially DO NOT enable Thumb2, or NEON, and so forth in any contributed builds. In stage3 onward, do not change build flags, etc. as these are defined globally in the RPM packages (rpm, redhat-rpm-config, fedora-release...)

Warning: The project is past stage 3 and no longer uses this repository.

• http://git.fedorahosted.org/git/arm.git - pull the "armv7hl" branch

We have developed a workflow around a git based root filesystem. You will need to clone this git repository, either onto your board directory, or into a directory you export over NFS. In one possible situation, you may clone the git repository into a local "fedora_arm" repository, then clone this again into a root-owned "fedora_armv7hl_rootfs" that is NFS exported in such a fashion that it can be used as a chroot from an ARM v7 board running the existing Fedora 13 ARM release.

The following commands will clone the repository and setup the right branch(es):

$ git clone --branch armv7hl ssh://git.fedorahosted.org/git/arm.git fedora_armv7hl_rootfs
$ cd fedora_armv7hl_rootfs
$ git branch armv7hl-YOUR_FAS_USERNAME
$ git checkout armv7hl-YOUR_FAS_USERNAME

• NOTE: Make sure that you have requested access to the Fedora ARM git repository in FAS, or just replace "ssh" with "git", and send patches/pull requests rather than committing directly to your own named branch of the repository - the use of "YOUR_FAS_USERNAME" is a convenience, it is not a technical requirement, we just do not want commits on master directly please, as it helps us track things more easily.

This will create a (your) named working branch of "armv7hl", which is a root filesystem. You should export this repository directly to your ARMv7 target board using NFS, or clone it again a second time remotely onto the target. A second clone can be beneficial because you will be running various commands inside this root filesystem under the root user account on the target and probably want to avoid permission problems with running commits as a regular user (otherwise, you will need to use "chown -R username" periodically to avoid problems managing the repo as a regular user). In the case of a second clone:

$ git clone --branch armv7hl-YOUR_FAS_USERNAME fedora_armv7hl_rootfs_target

• You can do stuff as root within this and "git push" back to your main repo without permission problems (ignore the lock warning during commit unless you are sharing this repository locally, in which case you need a sudo solution or similar)

IMPORTANT: Regardless of how you provide the target with the root filesystem, it does not contain a populated /dev directory due to needing to avoid having git need to house special files. This is done whenever you run stage2, but if you need to do it manually, you will need to perform these steps:

mkdir dev
pushd dev
sudo mknod null         c   1 3
sudo mknod zero         c   1 5
sudo mknod tty          c   5 0
sudo mknod console      c   5 1
sudo mknod sda          b   8 0
sudo mknod sda1         b   8 1
sudo mknod sda2         b   8 2
sudo mknod sda3         b   8 3
sudo mknod sda4         b   8 4
sudo mknod mmcblk0      b 179 0
sudo mknod mmcblk0p1    b 179 1
sudo mknod mmcblk0p2    b 179 2
sudo mknod mmcblk0p3    b 179 3
sudo mknod mmcblk0p4    b 179 4
sudo mknod ttyO0        c 253 0
sudo mknod ttyO1        c 253 1
sudo mknod ttyO2        c 253 2
sudo mknod ttyO3        c 253 3
popd

Now that you have a working copy of the armv7hl root filesystem, and have exported it to your target either over NFS (probably from a second copy as in the above scenario) or by making a remote clone using git on/onto the target you should establish a correct /proc bind mount and "chroot" into this on your target (you may need to leave the chroot and re-enter if you use git commands that change the repository files remotely):

$ mount -o bind /proc /path/to/exported/fedora_armv7hl_rootfs_target/proc
$ mount -o bind -t none /dev /path/to/exported/fedora_armv7hl_rootfs_target/dev
$ mount -o bind -t none /dev/pts/ /path/to/exported/fedora_armv7hl_rootfs_target/dev/pts
$ chroot /path/to/exported/fedora_armv7hl_rootfs_target/ /sbin/init

NOTE: The reason to bind mount /proc is that RPM contain logic to determine armv7hl compatibility that involves examining the cpuflags. If you do not bind mount /proc, you will build armv5tel binaries that are NOT ARMV7HL COMPATIBLE. Please make sure you can see /proc from within the chroot environment. A later set of patches to RPM may use alternative logic based around ELF file sectional attributes on RPM itself, but not yet.

Stage 2 Notes

Stage 2 bootstrap is now complete. But in this stage, the following was performed.

On the host, edit the "stage2/stage2" script, which is a copy of the one in the bootstrap repo. Please read and follow any instructions you find at the top of that script. In there, you will find various target "modules" (packages). Choose a package (refer to and update the real time list of work items at Etherpad) and add entries for it both to the latter part of the file, and a "go" command at the top to call it at the appropriate point during bootstrap (unless ordering matters, just as the last item in the current list). For example, there is a set of commands for db4 already in the current stage2 script:

NOTE: Make sure you updated the Etherpad with what you are working on

    db4 )
	mcd $BUILDDIR/db4
	$SRC/db-*/dist/configure $TCONFIGARGS
	make $J
	make $J install
	;;

There is a need to add a "go db4" higher in the file, in the correct position relative to the other "go" commands. In most cases, adding to the end of the list will be fine, unless ordering matters due to build dependencies - in that case, try to add a comment to this effect so that it is easier to understand why you have needed certain ordering.

On the host, you will need to download the most recent successfully built SRPM from primary arch Fedora 15 (refer to Koji to find the F15 SRPM for the package in question) to use as a seed. Extract and apply the patches required for Fedora through the use of the following commands (we don't currently keep a copy of the SRPM in the rootfs repo, since it is not really required - but we do keep the extracted sources, and so forth):

$ HOME=stage2/ rpm -ivh /path/to/downloaded/srpm
$ HOME=stage2/ rpmbuild --target=armv7-unknown-linux-gnu --nodeps -bp stage2/rpmbuild/SPECS/package_name.spec

You should also add your package to the list of stage2 sources at the end of the stage1 script. If you have built your rootfs with stage1, you can then run "./stage1 stage2" to update the source tree in your stage2 area. If you have checked your stage2 area out from git, email your stage1 patch to DJ or Jon, and use the manual RPM instructions above to install your sources.

Ensure that, in any case (especially if you have overridden your RPM macros and the use of a HOME environment variable doesn't work for you) the extracted RPM sources, spec, and directory of fully applied bits are in place in stage2/rpmbuild (in SOURCES, SPECS, and BUILD, respectively). We will NOT change these from Fedora 15, nor make changes to them during build. All builds take place in a separate directory (stage2/builds), typically exploiting the fact that GNU Autoconf handles having sources in a separate directory from the build directory. If a particular build cannot operate in this fashion, use cp or rsync (see stage1 in the bootstrap repo for examples) to copy the bits into place in the builds/package_name directory.

After making changes to the stage2/stage2 script, you should be able to run the following from the target chroot:

$ /stage2/stage2 package_name

If you did things correctly, then the result should be a stage2/builds/package_name directory containing the build tree, and the bits should have been installed into the rootfs. If this works, please commit the changes onto your branch:

$ git commit -as new_paths_here

Then push these to any higher level repository you might have if you are working in a second copy (e.g. fedora_armv7hl_rootfs_target push to fedora_armv7hl_rootfs). Finally, with your changes on your armv7hl-YOUR_FAS_USERNAME branch, you can push this to an identically named branch on the Fedora ARM git repo, and request a merge by mailing the list (copy jcm@jonmasters.org with merge requests), or simply post a pull request from your externally hosted repo. You could try sending patches to the list directly, but they will likely be pretty large, so as a fallback, post a link to a patch file you are hosting elsewhere (such as fedorapeople) if you are unable to use git for this part.

Changes will be merged from various branches into the top-level armv7hl periodically. This ensures some ability to track who is making what changes, and when, and allows us to revert problematic builds. We will aim to keep this rootfs around only as long as we need it to get to the point where we have a working RPM.

Contribution checklist

The following things should be in your commit(s):

• Changes to stage2/stage2
• Additional exploded sources in stage2/rpmbuild
• The buildroot from building in stage/builds
• The installed bits in bin,etc,lib,usr,etc.

You should attach the exact version of the package you used (Fedora 15 NVR) in the commit message. If you require any clarification, refer to existing commits as examples for how to proceed and contact us on IRC.

Stage 3 Notes

Stage 3 bootstrap is now complete. But in this stage, the following was performed.

Within the working chroot environment, you can see a /stage3 directory. This contains a directory named "armv7hl" filled with RPMs and SRPMs. You should choose a new package to build (using the armv7hl bootstrap package status wiki page for hints, and by asking on #fedora-arm if in doubt about useful packages to work on (or to verify nobody has started work and has not updated the wiki to indicate so doing). We need dependencies for mock (e.g. "userspace", and its dependencies) as well as other packages known to be in the minimal mock buildroots for Fedora or to be required to rebuild the minimal buildroot (a larger set).

Having chosen a packages, download its SRPM from Fedora 15 primary arch Koji. Stash that in /stage3/armv7hl/SRPMS. Then, from within /stage3:

$ HOME=`pwd` rpm -ivh armv7hl/SRPMS/the-package-srpm
$ HOME=`pwd` rpmbuld -ba rpmbuild/SPECS/the-package.spec

For many packages, a "--nodeps" is required at the "rpmbuild" stage because its deps are available from stage1/stage2 binaries that are not yet rebuild in stage3. But verify that this is the case before forcing "--nodeps". Bonus points for rebuilding stage1/stage2 packages as real RPMs, and for noticing dependencies that are not explicitly provided. Do verify that dependencies have been built before building packages - if necessary, hold off on a package build and go build its dependencies recursively first.

The output from this exercise is an /stage3/armv7hl/RPMS and a /stage3/armv7hl/SRPMS update in a git repo branch named armv7hl-YOUR_FAS_USERNAME that can be pulled and merged into the rootfs. An alternative is to post a URL with your RPM/SRPM combination. In most cases the SRPM should be identical to the original F15 package. If you need to make changes, don't bump the release, but instead add "arm" to it (and "arm1", "arm2", etc. if needed), so creating foo-1.2-3arm rather than foo-1.2-3.

After requesting a merge, the armv7hl rootfs will be updated and your RPM(s) can be picked up by others for building dependent packages. The rootfs will explicitly contain an installed version which will be done as part of the merge (do not install your package directly before requesting a merge, since this may complicate merging).

How to build

Packages should be retrieved in SRPM form from F15 on primary architecture (from Fedora Koji), then their dependencies should be examined. In cases where minimal dependencies exist to build a sane package (albeit perhaps without a "docs" sub-package, etc.) then that package should be built and the status updated here. Packages requiring dependencies should first have those dependencies built. If a package recursively requires itself, it might be necessary to build a locally minimal version of the package to bootstrap itself, then a real RPM. We care about the real RPM/SRPM for armv7hl, and so do not need the interim version of a package if an RPM/SRPM combination is available.

Once an RPM/SRPM combination is available, it should be uploaded somewhere, and an email/IRC announcement made. Even more preferable is to populate the RPMs into /stage3/armv7hl within a branch named armv7hl-YOUR_FAS_USERNAME within the rootfs git repo. Ping jcm (jonmasters) on #fedora-arm IRC and he will pull this (or pull from a URL), then he will install these RPMs into the master armv7hl branch from which others can update their rootfs. Packages are also mirrored onto scotland.proximity.on.ca/fedora-arm/armv7hl and this contains yum repo data pre-installed into the rootfs as well (that can later be used by mock once a base set of packages is available to do so).

Results

http://arm-temp.ausil.us/pub/fedora-arm/stage3-full/ https://netst.org/pub/linux/fedora/arm/15/ ftp://ftp.delorie.com/pub/armv7hl/ http://scotland.proximity.on.ca/fedora-arm/armv7hl/

Stage 4 Notes

The arm-rebuild.sh script contains the name of the central server and the subdirectory where all the files are kept. The rootfs we provide has the correct SSH keys pre-installed to access them. We'll refer to that location as $SRV in these examples.

Most of stage 4 happens using an automated build in a small custom rootfs. SRPMs are fetched from a central server and built, with the resulting RPMs and logs sent back to the central server. For most of the RPMs, the build is automatic and just having lots of "builders" running the build script suffices.

Sometimes, however, an SRPM will not build. Such cases are noted at Fedora15 HardFP Bootstrap package status. The procedure for broken builds is as follows:

• Download and review the build logs to determine what broke ($SRV/stage4/package/*.log). Edit the package status page with a summary of your findings.
• If an updated package is available for F15, build that instead (see below).
• If the fix is in an F16 or rawhide package, but not F15, ask the package maintainer to push it as an update to F15 also.
• Failing that, fix the package yourself.

When asking the package maintainer to update F15, use a template like this:

Subject: <i>packagename</i> fails to build from source in F15, have been fixed in F16/rawhide

Please push the <i>package-version</i> update to F15 as well. The
current package in F15 fails to build and the armv7 secondary arch
needs a working F15 source package to build from.

If you need to provide a custom SRPM to fix the bug, please change the Release: tag in the spec file to have .0.armN appended (example: Release: 3%{?dist}.0.arm1).

The SRPMs are kept in $SRV/SRPMS/. To "claim" an SRPM, move it to $SRV/SRPMS/build/. When you're done building the fixed RPMs, create a $SRV/stage4/package/ subdirectory and upload your SRPM and RPMs there, as well as any logs. Note: the subdir name and SRPM/RPM locations must be correct or the createrepo command will fail. Finally, run createrepo --update -d $SRV/stage4/ on the central server.The arm-rebuild.sh script contains the name of the central server and the subdirectory where all the files are kept. The rootfs we provide has the correct SSH keys pre-installed. We'll refer to that location as $SRV in these examples.

Having fixed the package, you should now push the changes upstream. Contact the package maintainer and ask them to apply the fix. If you cannot determine who the package maintainer is, or the maintainer does not respond, contact one of the primary ARM maintainers (like Dennis Gilmore or Peter Robinson). The maintainers can apply the fixes to Fedora's primary repository. Also, locate the appropriate project-upstream (for example, X.org for X patches) contribution instructions and send your patch there too, if applicable.

Caution: Your patch will be used for all architectures, not just ARM. You should use %ifarch in the spec file, and suitable conditionals in your patches, to ensure that you do not break other builds! Also, if your patch is intended to be a temporary workaround just to get past boostrapping, use a suitable release number, like 0.0.armbootstrapN (example: Release: 0.0.armbootstrap2), to make it obvious that the RPM is temporary, and to ensure that "official" rpms are considered "newer".