Changes/IntelSGX

Intel SGX Software Stack

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

The Intel SGX technology enables creation of execution enclaves, whose memory is encrypted and thus protected from all other code running on the CPU, including SMM, firmware, kernel and userspace. This proposal is to introduce the SGX host software stack and development packages to Fedora, to enable future introduction applications and features which have a dependency on SGX technology.

The primary feature that will leverage SGX in a subsequent Fedora release is expected to be Intel TDX, which provides confidential virtual machines, and is in the process of being integrated with QEMU and KVM.

Owner

• Name: Daniel Berrange
• Email: berrange@redhat.com

Current status

• Targeted release: Fedora Linux 42
• Last updated: 2024-11-11
• [Announced]
• [<will be assigned by the Wrangler> Discussion thread]
• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

The Intel SGX technology enables creation of execution enclaves, whose memory is encrypted and thus protected from all other code running on the machine, including SMM, firmware, kernel and userspace. While it has many potential use cases, this proposal is focused around the infrastructure needed to enable support for attestation of TDX confidential virtual machines.

The SGX software stack compromises a number of components

• Support for developing new enclaves
  • Header files for the enclave code (a minimalist C library, C++ library, crypto and some other misc libraries)
  • Static library archives for linking into the enclave binaries.
  • Build helper tools (for signing enclaves, generating code enclave API entrypoints)
• Support for developing applications that use enclaves, eg to be able to load and communicate with enclaves.
  • Header files for platform code
  • Dynamic libraries for platform code
• Support for deploying applications that use enclaves
  • Enclave service daemon - assists unprivileged applications in loading enclaves
  • Registration tools - assists platform administrator in acquiring certificates from Intel servers to identity the platform
  • Quote generation daemon - assists QEMU in acquiring signed attestation reports for TDX VMs.

The code and binaries related to the host OS platform components will be installed in the normal filesystem locations common to all Fedora packages. e.g. libraries in /usr/lib64, headers in /usr/include and binaries in /usr/bin.

For the purposes of packaging, the enclaves will be treated as cross-compilation target. While the compiler build architecture target is x86_64, the runtime has custom C / C++ libraries that must be used, and requires a separate code loader. Enclaves cannot be directly linked to applications, they are strictly independent libraries.

With this in mind, all enclave related headers and libraries are proposed to be installed at /usr/x86_64-intel-sgx, specifically under lib64 and include sub-directories. There are no binaries for enclaves, only libraries, so no bin dir is required.

The generated binary packages will generally all have an 'sgx-' prefix to their name, with enclave related packages having an more specialized 'sgx-enclave-' name prefix.

The source packages for the reproducible build toolchain will have 'sgx-compat-' as a prefix, to distinguish them from regular toolchain packages

For reasons of brevity, this change proposal does not go far into all technical details of SGX. For a deeper understanding of SGX enclaves beyond this proposal, consult this companion document.

Feedback

• Feedback: The SGX technology can be used as a way to implement DRM. Notable example has been BluRay playback.. Answer: As with many technologies, it is possible to use SGX in ways that are both positive and negative, wrt the owner / users of a machine. Use of SGX for DRM in BluRay playback is hostile to the owner/user of a machine. This change is NOT proposing to introduce / support any such usage/applications in Fedora. The fact that bad uses of SGX exist outside of Fedora, must not block the use of SGX in in Fedora for scenarios where it can offer features that benefit Fedora's users.

• Feedback: The SGX enclave code is not open source, because it requires a vendor signature on output binaries. Answer: 100% of the SGX code is made available under a variety open source licenses (Apache, BSD, MIT, GPL & more). One of the architectural enclaves, pce, requires an Intel signature because it is used to establish the root of trust with the hardware. In their role of bootstrapping use of SGX hardware, the architectural enclaves are a type of firmware, and it is normal for firmware to have a vendor signature. In contrast to almost all firmware which is proprietary, the architectural enclave code is all under Fedora approved OSS licenses. All of the architectural enclaves, can be rebuilt from source by a user, with customizations, signed with a user specified key and then loaded if desired. There are some subtle limitations if attempting this, documented in the companion document.

Benefit to Fedora

As a general purpose infrastructure technology, SGX can be applied to / used by a wide variety of scenarios / applications.

In the context of this change proposal, no application usage is intended to be introduced. This proposal is focused on infrastructure enablement.

A followup change proposal in a subsequent Fedora release will be made to introduce Intel TDX confidential virtual machines, which is anticipated to be the first end user facing usage of SGX technology. Attestation is the means by which a guest VM owner, can prove that their VM machine is running in confidential mode on genuine Intel hardware, as opposed to being in a "blue pill" environment. All currently shipping Intel CPUs which support TDX build attestation on top of SGX with OSS enclave code (the tdqe enclave), as opposed to embedding attestation in proprietary firmware.

NB, the tdqe enclave will be included in this proposal, but will remain unused until the followup TDX change proposal introduces the QEMU changes that depend on it.

Scope

Proposal owners

Add at least the following source packages to Fedora:

• CppMicroServices - a C++ runtime library for building microservices daemons
• sgx-srpm-macros - define some common macros for where SGX content will live in the filesystem tree
• linux-sgx - provide the SGX platform development headers & libraries, runtime libraries, supporting daemons, unsigned architectural enclaves.
• linux-sgx-enclaves-prebuilt - the pre-built, signed architectural enclaves, as provided by Intel.

Optional extra - reproducible builds

The upstream SGX project provides a mechanism to perform a reproducible build. This can be used to validate that the enclave payload in the signed pre-built enclaves matches the source it claimed to originate from.

While the pre-built, signed enclaves qualify to be shipped "as is" under the Fedora firmware packaging guidelines, going through the reproducible build process will guarantee provenance of the shipped binaries.

The upstream process is designed around a Dockerfile that defines a NixOS build environment. To reproduce the same binaries outside NixOS, would require Fedora to have a toolchain and supporting code pinned at a specific designated version, with specific build time features enabled in the case of gcc & binutils.

IOW, to achieve a reproducible buld would require the following additional package(s):

• sgx-compat-gccXXX - one (or more) specific GCC versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
• sgx-compat-binutilsXXX - one (or more) specific binutils versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
• sgx-compat-nasmXXX - one (or more) specific NASM versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
• sgx-compat-glibc-headersXXX - one (or more) specific GCC versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
• sgx-compat-kernel-headersXXX - one (or more) specific GCC versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.

With those present, then it would be possible to provide a further set of source package(s)

• linux-sgx-enclavesXXX - one (or more) packages that reproduces the pre-build, signed architectural enclaves

This package would be built in rawhide, for each SGX release, but it would not emit any binary RPMs. Its %check section would merely check that the enclaves it produces match those in the linux-sgx-enclaves-prebuilt package, which is what is shipped.

Note: not every architectural enclave binary is re-issued on every release. New signed builds are only made available when there is a CVE fix, or a feature enhancement. IOW the linux-sgx-enclaves-prebuilt package may contain enclave binaries from a mixture of releases. At time of writing, the SGX 2.25 release included new builds for every enclave, but a future 2.26 release may only update a subset. Thus at some points in time, Fedora may require more than one linux-sgx-enclavesXXX package present. Duplicates will be dropped as & when they are obsoleted by newer releases.

At time of writing, with the SGX 2.25 release, the versions required to perform a fully reproducible build will be

• gcc: 8.5.0, 9.5.0
• binutils: 2.38, 2.40
• nasm: 2.16.01
• glibc: 2.38
• kernel: 5.17

Note: the fact that two versions are needed for binutils and GCC, is an upstream an upstream bug

Note: the enclaves are supposed to be built exclusively against the SGX SDK which provides its own C runtime headers. Thus the fact that it has a requirement for header files from glibc & kernel is also likely another upstream bug.

The complexity of the reproducible build process makes this a non-trivial undertaking, for something that is not strictly required by the packaging guidelines for firmware. This option is included in recognition of Fedora's strong commitment to the support of OSS, despite its relaxed stance wrt firmware that was done for real world practical reasons.

Variant - reproducible builds

As a variant on the above process, it would be possible to avoid the linux-sgx-enclaves-prebuilt source package entirely and simply bundle the pre-built, signed enclaves in the linux-sgx-enclavesXXX packages used for the reproducible build process.

The compelling reason to do such a combination is that it would let Fedora guarantee that it would make the reproducible build process into a hard blocker for shipping the pre-built, signed enclaves.

The downside to such an approach, however, is massive pain in the event that updates need to be shipped in stable streams of Fedora to fix CVEs. As noted above, the reproducible build requires specific toolchain & library versions. As new release of SGX that can conceivably introduce a dependency on newer versions. Also each new release would also entail creating a new linux-sgx-enclavesXXX source package for proving reproducibility, given the need to validate multiple releases in parallel at any given point in time.

It is highly undesirable to have the "new package process" be a dependency for shipping CVE fixes, as it risks significantly delaying delivery to users.

This variant is included to illustrate the alternative strategy available, but with the recommendation that this NOT be followed for practicality of shipping updates.

Other developers

It is not anticipated that other package maintainers need do anything to support introduction of SGX. The kernel is already built with the SGX feature enabled.

systemd ships with udev rules to enable creation of some of the required device nodes. The SGX package contains addon udev rules for remaining device nodes. The latter may be contributed upstream to systemd in future, and would flow back into Fedora in a normal systemd update.

Release engineering

N/A - does not impact deliverables for releng

Policies and guidelines

Agreement is needed around the designation of the pre-built, signed SGX architectural enclaves as firmware. Note, "architectural enclaves" are being considered distinct from "application enclaves" - the latter should be expected to build from source in the normal manner & be signed by Fedora, if any such enclaves are ever added to Fedora.

Architectural enclaves as firmware

Normal Fedora practice requires building everything from source. There is a general exception to this for firmware blobs, which don't need to be built from source and don't need to be under an OSS compliant license as long as the binaries can be freely distributed. Hardware firmware binaries almost always include a digital signature from the vendor, to ensure that only trusted firmware is loaded onto the device. Thus even if source code is available for a given firmware binary, Fedora would still need to be shipping it as a pre-built vendor supplied signed binary, in order to pass the cryptographic verification checks typically performed by hardware.

Another example of shipping pre-built vendor binaries would be CPU microcode. This could be considered firmware for the core CPU, and again are vendor supplied, signed binaries, with no source code available.

The architectural enclaves are made available with 100% of their code under Fedora approved OSS licenses. They are always used, however, as pre-built signed binaries due to the need to establish a chain of trust rooted back to Intel as the CPU hardware vendor.

Overall, the architectural enclaves meet the definition of firmware documented in the Fedora licensing guidelines:

1. The files must be non-executable within the Fedora Linux context (note: this means that the files cannot run on their own, not that they are just chmod -x). ✅ enclave ELF binarjies must be loaded into memory using the specialized SGX loader.
2. The files must not be libraries, within the Fedora Linux context. ✅ enclave ELF binaries must be loaded into memory using as specialized SGX loader, the glibc ELF loader can not be used for this.
3. The files must be standalone, not embedded in executable or library code (within the Fedora Linux context). ✅ enclave binaries are always standalone, self-contained files. They are consumed by regular host software, but cannot be linked to Fedora platform binaries/libraries.
4. The files must be necessary for the functionality of open source code being included in Fedora Linux or to enable Fedora Linux to boot on a specific device, where no other reliable and supported mechanisms exist. ✅ the pre-built, signed architectural enclaves shipped by Intel, serve to bootstrap the use of SGX technology by applications. ❌ as higher level software services, other application enclaves would not qualify.

Thus it should be permissible to ship the pre-built, signed binaries with no policy changes.

Reproducible build validation policy

A slightly different reference point is in the handling of shim, where Fedora builds a binary from known good source, possibly with local patches, and sends it off for signing by Microsoft, packaging the binary that is sent back. Microsoft has various requirements before it will permit signing a vendor's shim.

Consider if the requirements on shim signing were tightened a little more to require a designated toolchain version, such that all shim builds were 100% reproducible on any platform. There would be no need to send the binary to Microsoft for signing, as any binary built in Fedora would inherently match a standard pre-published signed binary from that shim release + toolchain. It would be impossible to distinguish Microsoft signing & returning Fedora's reproducible build, from Microsoft sending back their original pre-signed build.

This is the situation SGX architectural enclaves are in. None the less, there are some differences between the shim & SGX scenarios. The first is that Fedora has the ability to add arbitrary local patches to shim & build with arbitrary toolchain versions of its choosing today. This would not be possible if following the hypothetical stricter reproducible build process above. The second is that on most (but not all) hardware, users can enroll personal certificates in the firmware to allow it to trust a shim binary signed by the user.

In the case of SGX enclaves, it is strictly possible from a technical POV for a user to build & load & run their own self-signed SGX architectural enclaves. From a practical POV, however, it is inescapable for the pce enclave to have an Intel signature, as without that no certificate proving authenticity of the hardware can be obtained from Intel. The other enclaves do have ways to be used with user supplied signatures, provided the consuming use cases allow sufficient flexibility in their verification mechanisms.

The shim model gives further impetus to the idea of it being acceptable to ship pre-built binaries, if a reproducible build process can prove the binaries match the source under Fedora approved licenses. Thus a reproducible build process has been illustrated as an viable option for this change, should there be dissatisfaction with treating pre-built architectural enclaves exclusively under the firmware guidelines.

Trademark approval

N/A

Alignment with the Fedora Strategy

This aligns with

• "Reaching the world". Including SGX will make the Fedora support for hosting Intel TDX confidential virtual machines feature complete, by enabling attestation by the guest owner
• "Innovation & leadership in technology". SGX is a general purpose infrastructure technology which enables application developers to build systems to securely run sensitive workloads. Confidential virtual machines are likely to become a standard part of the public cloud in the coming years, as well as make inroads into private clouds. As noted earlier, SGX unlocks the ability to ship TDX confidetial VM technology in future Fedora.

Upgrade/compatibility impact

This is a new package set which should not have any upgrade impact, as it will not initially be a dependency of other software. In future it may be pulled in automatically as a dependency in certain KVM deployment scenarios. Even when installed, using anything related to SGX first requires host firmware changes to enable use of the technology. The systemd services provided have their unit files conditionalized on the existence of /dev/sgx_enclave device nodes.

Early Testing (Optional)

Do you require 'QA Blueprint' support? N

The proposed new packages are available for testing via Copr, until such time as they are reviewed & built in Fedora koji:

• https://copr.fedorainfracloud.org/coprs/berrange/sgx-ng/

These should work on any Intel Xeon class platform which has a suitable HW configuration. NB there may be specific DIMM population requirements.

How To Test

• https://fedoraproject.org/wiki/Virt/SGX Documentation on host setup] is available but that's a fairly minimalist test. It does not do much that's interesting to an end user, but is at least proving that the pce and ide enclaves are usable.

User Experience

Initially minimal user experience impact, since on its own it doesn't deliver noticeable end user features, as it is not believed that any existing applications in Fedora are able to leverage SGX.

The initial user benefit will be that users can bootstrap trust in SGX on their Fedora host. This will facilitate users in deploying 3rd party applications of their choosing that utilize SGX.

At a later time, when support for Intel TDX is integrated into KVM and QEMU, the immediate Fedora user benefit will significantly expand.

Dependencies

The primary functional dependency for use of SGX is kernel support, which has existed in Fedora for some time. See "CONFIG_X86_SGX=y" in the kconfig files.

The packages include some new systemd unit files, two of which should be configured to be started by default. This will require changes to the systemd presets in the 'fedora-release' package.

• mpa_registration.service - this is conditionalized on SGX being enabled, as witnessed by existence of /dev/sgx_eclave. Thus enabling it by default will be a no-op on any existing machines which have not had SGX turned on in the firmware. It is expected to be installed on all SGX installations
• qgs.socket (as a trigger for qgs.service) - this is likewise conditionalized on SGX being enabled. This will may be pulled in as a dependency of either libvirt or QEMU RPMs, TBD in the future TDX change proposal.

Contingency Plan

• Contingency mechanism: The new packages have no ill effects on existing Fedora usage. Any outstanding work can be postponed to a later release if required.
• Contingency deadline: Beta free
• Blocks release? No

Documentation

Documentation will be provided to describe:

• Configuring hardware UEFI to enable use of SGX
• Obtaining PCK certificates for a host from the Intel trusted services API

A subsequent change proposal will cover usage of SGX with TDX confidential virtual machines.