Changes/FEX

Integrate FEX in Fedora Linux

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

FEX is a fast emulator that allows one to run x86 and x86-64 binaries on an AArch64 Linux host. FEX requires a number of supporting components, including a RootFS image, and integration with krun to support 16k page-size hosts. The purpose of this Change is to integrate FEX itself and its supporting components into Fedora Linux, to provide a delightful out-of-box experience for users that want to run x86 and x86-64 binaries on their aarch64 systems. This also includes integration into the AArch64 Fedora KDE spin as a non-blocking component of the spin.

Owner

• Name: Davide Cavalca, Neal Gompa, Alyssa Rosenzweig
• Responsible Teams: Asahi SIG, KDE SIG
• Email: davide(at)cavalca.name, ngompa13(at)gmail.com, alyssa(at)rosenzweig.io

Current status

• Targeted release: Fedora Linux 42
• Last updated: 2024-09-24
• Announced
• Discussion thread
• FESCo issue: #3273
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

When running Fedora Linux on an AArch64 host, one can normally only run AArch64 binaries. This can be a problem if the user needs to run preexisting software that was built for x86 or x86-64, which is still the predominant architecture. If the software is opensource it can sometimes be recompiled (or, even better, packaged in Fedora), but this isn't always an option and could require significant work on the user's part. For proprietary software, there is no viable option short of persuading the author to release a native AArch64 build.

FEX allows the user to sidestep this issue entirely by making it possible to run x86 and x86-64 binares on AArch64 Linux hosts, as if they were native. FEX achieves this via emulation, and it integrates with the binfmts infrastructure to provide a seamless experience. To ensure wide compatibility, FEX is also able to leverage a distribution-shipped root filesystem tree (RootFS) to provide core system libraries that the emulated binaries might need.

FEX only supports AArch64 host systems running a 4k page-size kernel. This is the default in Fedora Linux, but it presents a problem for Fedora Asahi Remix, as Apple Silicon Macs use a 16k page-size. To address this, we will leverage krun to run FEX inside a microVM with a 4k page-size kernel, thus providing a compatible environment with minimal overhead.

Feedback

What's the difference between FEX, QEMU and box64 ? Why should a user prefer one vs the other?

QEMU, FEX and box64 are all user-mode emulators that allow one to run foreign arch binaries. Additionally, QEMU also provides a system emulator, that allows one to run a whole OS in a virtual machine -- that's out of scope for the context of this Change, which is only concerned with user-mode emulation of binaries.

The main difference mostly boils down to architecture support and accuracy vs performance tradeoffs.

QEMU, box64 and FEX all support x86_64 on aarch64; FEX and QEMU also support x86 on aarch64, box64 does not (this matters because, for example, Steam requires both x86 and x86_64). QEMU and box64 also support additional architectures that are out of scope for this Change.

FEX only supports systems running 4k page-size kernels, and this is why we plan to leverage krun to support FEX on other page-size kernels. box64 implements hacks to support a limited subset of x86_64 apps on a 16k page-size kernel; this is not a general solution, and is specifically broken with e.g. wine. QEMU does not care about the page-size and works the same regardless.

Performance is similar between box64 and FEX, and both are steadily improving. QEMU is the best implementation for correctness, but it's extremely slow in comparison, making it unsuitable for a lot of practical usecases (such as gaming).

The Change owners are actively involved with FEX development upstream and are focusing on making it faster. Given this, we believe that FEX will provide the best out-of-the-box experience for most users.

How does FEX interact with binfmts when there's other things also setting up binfmts for the same stuff (say, QEMU); also how will this work with krun?

The implementation details are still being defined here, but in general we expect that we'll want krun to take precedence over bare FEX on non-4k page-size systems, and FEX to take precedence over QEMU. We are considering deploying a shim for the binfmt handler to allow for smarter handling of this (e.g. to dynamically choose whether to use krun).

Should we have a virtual Provides: binfmt($target) to make it easier to depend on an emulator instead of a specific implementation?

We believe this is out of scope for this Change.

Will this work with x86 and x86_64 flatpak too?

Not yet. This planned, but it requires non-trivial technical work. In the interest of keeping things focused, we're going to consider it out of scope for the initial implementation, and might revisit it down the road depending within the timeframe of this Change.

Will this work with just x86_64 or x86 as well?

Yes, as mentioned above we want to support both x86_64 (i.e. 64-bit) and x86 (i.e. 32-bit) binaries. This is one of the reasons why we're targeting FEX instead of box64.

Which platforms are we targeting for krun?

krun is primarily meant for Fedora Asahi Remix, which runs on Apple Silicon Macs using a 16k page-size kernel, and that's the target we're going to develop and test against. That said, it should work for other any non-4k page-size systems (say, aarch64 boxes running a 64k page-size kernel) as well.

Benefit to Fedora

Users will be able to run x86 and x86-64 binaries out of the box when running Fedora Linux on their Aarch64 systems. This particularly improves the usability of Fedora KDE for day-to-day usage on AArch64 systems.

Scope

• Proposal owners:
  • DONE: Package FEX in Fedora Linux - https://src.fedoraproject.org/rpms/fex-emu
  • WIP: Create a Kiwi definition to produce a FEX root filesystem - https://pagure.io/fedora-kiwi-descriptions/pull-request/83
  • TODO: Package the root filesystem in Fedora Linux as an RPM
  • DONE: Package sommelier -- https://src.fedoraproject.org/rpms/chromiumos-platform
  • DONE: Package krun - https://src.fedoraproject.org/rpms/rust-krun
  • TODO: Integrate FEX with krun to support 16k page-size systems
  • TODO: Create a comps group for FEX and supporting components
  • TODO: Attach the comps group to the kde-desktop-environment group
  • TODO: Create a Kiwi definition to produce a sysroot image to be used by FEX at build time to enable thunks
  • TODO: Write documentation

• Other developers: assist with package reviews and provide feedback

• Release engineering: #12299

• Policies and guidelines: N/A (not needed for this Change)

• Trademark approval: N/A (not needed for this Change)

• Alignment with Community Initiatives:

Upgrade/compatibility impact

No expected upgrade or compatibility impact, this is a purely additive Change.

How To Test

Once the relevant components have been packaged and the comps group has been created, this Change will be testable with:

1. dnf install @fex-x86-emulation
2. ...
3. profit!

User Experience

Users will be able to run x86 and x86-64 binaries out of the box when running Fedora Linux on their Aarch64 systems.

Dependencies

This is pretty much self contained.

Contingency Plan

• Contingency mechanism: (What to do? Who will do it?) Don't ship this
• Contingency deadline: N/A (not a System Wide Change)
• Blocks release? No

Documentation

The Change owners will write documentation on how to use FEX, both for general Fedora systems and in the context of Fedora Asahi Remix.