Changes/IntelSGX

Intel SGX Software Stack

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.

Summary

The Intel SGX technology enables creation of execution enclaves, whose memory is encrypted and thus protected from all other code running on the machine, including SMM, firmware, kernel and userspace. This proposal is to introduce the SGX host software stack to Fedora, to enable applications and features which have a dependency on SGX technology.

Owner

• Name: Daniel Berrange
• Email: berrange@redhat.com

Current status

• Targeted release: Fedora Linux 42
• Last updated: 2024-10-24
• [Announced]
• [<will be assigned by the Wrangler> Discussion thread]
• FESCo issue: <will be assigned by the Wrangler>
• Tracker bug: <will be assigned by the Wrangler>
• Release notes tracker: <will be assigned by the Wrangler>

Detailed Description

The Intel SGX technology enables creation of execution enclaves, whose memory is encrypted and thus protected from all other code running on the machine, including SMM, firmware, kernel and userspace. While it has many potential use cases, this proposal is focused around the infrastructure needed to enable support for attestation of TDX confidential virtual machines.

The SGX software stack compromises a number of components

• Support for developing new enclaves
  • Header files for the enclave code (a minimalist C library, C++ library, crypto and some other misc libraries)
  • Static library archives for the enclave code.
  • Build helper tools (for signing enclaves, generating code enclave API entrypoints)
• Support for developing applications that use enclaves
  • Header files for platform code
  • Dynamic libraries for platform code
• Support for deploying applications that use enclaves
  • Enclave service daemon - assists unprivileged applications in loading enclaves
  • Registration tools - assists platform administrator in acquiring certificates to identity the platform
  • Quote generation daemon - assists QEMU in acquiring signed attestation reports for TDX VMs.

Enclave use cases

At a high level, SGX enclaves provide a way to run code in a secure, encrypted & isolated execution context, free from manipulation by any other software component on the machine, aside from DoS attacks.

There are manways

Compatibility with Fedora mission for software & content freedom

When dealing with cryptographically attested software, it is important to consider whom is benefitting from verifying the execution context. There are generally three parties involved in such software

• Providing party - the organization or individual that is signing the software being used. This could be the original software author/vendor, or an intermediary such as a Linux distributor.
• Deploying party - the organization or individual that is running the software in question. This could be a so called 'end user', or it could be a service provider running something on their behalf.
• Verifying party - the organization or individual that is relying on verification of the executed software. This could be the so called 'end user', but it could be a 3rd party attempting to place constraints on usage (e.g. DRM)

The Fedora missions states

 We are dedicated to free software and content. 
 Advancing software and content freedom is a 
 central community goal, which we accomplish 
 through the software and content we promote.

In all cases, Intel acts as a "verifying party" when a platform is deployed or provisioned, when the platform owner requests a certificate to prove their hardware is authentic. This is a low level a bootstrapping task, to enable higher level use of SGX. Thus while Intel is the verifying party at a low level, the ultimate beneficiary is determined by the higher level usage of SGX who is also a verifying party.

Thus, while this low level usage has some impact on software freedom, it is limited to the minimum required to bootstrap use of the SGX technology. This is on a par with how firmware, or CPU microcode, binaries must have valid signatures to bootstrap use of the platform

Use of SGX as a DRM mechanism in BluRay playback software is not benefit for the user of the player, as it constrains the scenarios in which they can consume the content. The beneficiary is a 3rd party organization seeking to control the end user. Such use of SGX enclaves which compromise either software or contet freedom should be considered incompatible with Fedora's mission for end user freedom and thus not permitted.

Use of SGX as a mechanism for proving the integrity of a virtual machine has multiple beneficiaries. The Fedora user who owns the hypervisor OS install benefits in being able to prove integrity of the service they provide. Where the virtual machine is running Fedora as its OS, the Fedora user of the VM further benefits in being able to prove integrity of their guest OS. Including enclaves needed to attest TDX is generally compatible with Fedora's mission. The owners of the host & guest OS, gets to decide the usage policy and thus are not constraining their own software and content freedom. This usage of SGX is compatible with Fedora's mission for end user freeom and thus should be permitted.

Enclaves in depth vs packaging & licensing guidelines

Generally Fedora requires all software to be fully built from source, with the complete & corresponding source made available under one of the permitted code licenses.

There is, however, an exception for firmware binaries which do not have to provide source, and merely required to permit free redistribution.

The question is how to treat enclaves, and deciding this requires understanding what SGX enclaves are and how they are used.

Enclaves as a new platform build target

SGX enclaves are compiled to x86 machine code from C/C++ source, using custom C/C++ runtime libraries, instead of the host OS glibc / libstdc++. IOW, while the enclaves can build with the existing x86 C/C++ toolchain, they are more like a cross-compilation target in the way that a separate non-Linux runtime is used.

The resulting SGX enclave binary uses the ELF format, creating ELF executables, albeit with a .so file suffix. While the executable binary does have a .interp ELF section, the referenced ld-linux.so cannot load an enclave. A specialized SGX host OS library must be called to create an enclave and load code into it. Given that, the .interp ELF section is in fact redundant and misleading, and has been proposed for removal.

To load an SGX enclave, the ELF binary must have an embedded signature, that is created by the sgx_sign tool, using signing keys provided by whomever is going to attest to trustworthiness of the enclave binary. The signer can be whomever authors the enclave, but in most cases, can also be an arbitrary third party.

The critical point is that whomever is going to later verify the SGX enclave must be willing to trust whomever signed it. Such a trust relationship decision is dependent on the usage context of the enclave in question.

Enclave usage classes

Broadly speaking there are considered to be two classes of enclave:

1. Architectural enclaves - these are a small set of common enclaves, created and signed by Intel, which ship as standard in any SGX deployment. They provide a hardware enablement role, bootstrapping the cryptographic root of trust on a platform.
2. Application enclaves - this is an arbitrary set of custom enclaves, written by independent software vendors / providers to perform application specific tasks. They are signed by whichever party is to be relied upon to validate trustworthiness of the enclave code.

Both architectural and application enclaves are built and deployed in the same way, using the same runtime libraries. Their difference is largely where they sit in the chain of trust, with architectural enclaves being at the base, below all applications.

Given their use in bootstrapping the SGX platform chain of trust, the architectural enclaves would normally be considered to be serving the same role as platform firmware would. They establish an association with the physical platform hardware, which all other SGX uses relies upon.

Application enclaves, meanwhile are clearly just userspace code, with a special protected execution context. They are created by ordinary application developers as needed to handle some specialized application tasks and have no relationship to platform firmware.

Enclave execution context

All enclaves run in ring 3, which is traditionally the realm of userspace. The execution context, while using a region of memory allocated by & accounted to the loading application, is securely isolated from everything else on the system including the loading application. The enclaves' roles can look like either firmware, or kernel or userspace roles depending on the context on what the enclave is doing.

Running in ring 3 limits their ability to run privileged instructions, but the normal rules of access for rings do not apply. Nothing in ring 0 (firmware / kernel), ring 3 (userspace), or even psuedo-ring -1 (SMM) can interact with the enclaves, except via their published API entrypoints which are called via the EENTER instructions using a separate stack. IOW, the traditional view of hardware execution contexts being a simple stack of rings has been rather distorted by the capabilities of SGX, which exist as a distinct context off to the side of the rings model.

The architectural enclaves

There are 8 so called "architectural enclaves" built, signed and distributed by Intel.

• libsgx_le.signed.so - "Launch Enclave (le)"
  • This was historically used to control what enclaves are able to be launched on a platform. Modern systems allow loading on any enclave no matter who signed it.
  • This enclave requires an Intel signature
  • This architectural enclave is considered obsolete & is NOT intended to be shipped in Fedora.

• libsgx_qe.signed.so - "Quoting Enclave (qe)"
  • This was historically used in attestation of enclaves, using the Enhanced Privacy ID (EPID) scheme, which is replaced by RSA in SGX DCAP (Data Center Attestation Primitives) software.
  • This enclave requires an Intel signature
  • This architectural enclave is considered obsolete & is NOT intended to be shipped in Fedora.

• libsgx_pve.signed.so - "ProVisioning Enclave (pve)"
  • This was also historically used in attestation of enclaves, using the Enhanced Privacy ID (EPID) scheme, which is replaced by RSA in SGX DCAP (Data Center Attestation Primitives) software.
  • This enclave requires an Intel signature
  • This architectural enclave is considered obsolete & is NOT intended to be shipped in Fedora.

• libsgx_pce.signed.so - "Platform Certification Enclave (pce)"
  • This certifies all other enclaves running on the system. The identity of the Intel's signing key, together with secrets inside the CPU set in fuses at time of manufacture, to derive a key that is known only to the enclave and Intel. This is needed in order to obtain a certificate from Intel that proves the hardware is genuine.
  • This enclave requires an Intel signature

• libsgx_id_enclave.signed.so - "Identification Enclave (ide)"
  • This is used to extract some identification information about the processor that is needed to acquire a platform /processor certificate after deploying or re-provisioning a platform.
  • This enclave recommends an Intel signature

• libsgx_qe3.signed.so - "Quoting Enclave (qe3)"
  • This is used in attestation of enclaves, using the newer RSA scheme in SGX DCAP
  • This enclave recommends an Intel signature

• libsgx_tdqe3.signed.so - "Trust Domain Quoting Enclave (tdqe)"
  • This is used in attestation of TDX virtual machines, using the newer RSA scheme in SGX DCAP
  • This enclave recommends an Intel signature

• libsgx_qve.signed.so - "Quote Verification Enclave (qve)"
  • This is used to provide a way to verify attestation reports on an otherwise untrusted machine using SGX DCAP.
  • This enclave recommends an Intel signature
  • The pre-built binary for this enclave includes a build of openssl that potentially contains forbidden crypto & is NOT intended to be shipped in Fedora unless the crypto can be approved or eliminated

Out of the four architectural enclaves that require an Intel signature, only one is intended to be shipped in Fedora.

While the remaining four architectural enclaves can theoretically have a non-Intel signature, if this is done it will increase friction of usage. The identity of the signer of enclaves becomes embedded in quotes, and is checked by the user (or software) validating the quote which is unlikely to be expecting a non-Intel signer for these architectural enclaves.

Thus it is considered that all shipped architectural enclaves need to have the Intel signature.

Users who wish to change & rebuild architectural enclaves from source can sign the result with their own keys, for all except the libsgx_pce.signed.so which unavoidably requires an Intel signature in order to derive the correct shared keys. If doing this, they will have to ensure that applications they use for verification are able to support use of non-Intel signers.

The firmware exception

The Fedora licensing guidelines describe the requirements for a pre-built binary to quality for the firmware exception allowing source to be omitted.

• The files must be non-executable within the Fedora Linux context (note: this means that the files cannot run on their own, not that they are just chmod -x).
  • ✅ Enclaves are distributed as ELF executable files, which must be loaded using specialized SGX supporting code that sets up the enclave execution context on the processor, not the regular platform ld-linux.so.

• The files must not be libraries, within the Fedora Linux context.
  • ✅ While the enclaves .so file extension sugggests they are libraries, this is just a naming convention. The platform ld tool cannot link a binary to an enclave .so file, in the way a platform library is consumed. They must be loaded using specialized SGX supporting code that sets up the enclave execution context on the processor.

• The files must be standalone, not embedded in executable or library code (within the Fedora Linux context)
  • ✅ Every enclave is distributed as a standalone file, fully separated from any executable or library which may leverage its functionality.

• The files must be necessary for the functionality of open source code being included in Fedora Linux or to enable Fedora Linux to boot on a specific device, where no other reliable and supported mechanisms exist
  • ✅/❌ Depends on context of the enclave

Of the 4 pre-built, Intel signed, architectural proposed to be shipped, pce should trivially qualify. Without that there is no practical way to use SGX technology in Fedora, as no host certificate can be acquired to prove the hardware is genuine.

Of the remaining enclaves 'ide', 'qe3', 'tdqe', it is reasonable to say they are necessary, as verification of quotes from application enclaves will be expected to chain back to signer identities published by Intel. The purpose of the quoting enclaves is to prove use of a genuine Intel platform, and so the trust relationship for these enclaves is expected between the user and Intel to bootstrap use of the platform.

With application enclaves, there is goal is for the party who verifies the enclave output, to be able to demonstrate that known good code is executing in a secure context on Intel SGX. The verifying party needs to trust whomever built and signed the enclaves.

Looking back at the description of "benefitting parties", if the signer of application enclaves can not be Fedora, then it is highly unlikely that SGX is being used in a way that benefits the Fedora user. With this in mind application enclaves should not ordinarily qualify under the 4th clause for the firmware exception.

See the Policies & guidelines section for the concrete recommendation of how to apply Fedora policies.

Comparison with other technology

There is not a directly analogous technology for SGX enclaves on PC class hardware from other vendors, but we can compare how tasks using SGX would be accomplished elsewhere

The Intel TDX technology for virtual machines relies on SGX to provide signing of attestation reports. The guest VM will talk to the SEAM module to obtain an attestation report, which is then passed to the QEMU in the host OS. QEMU will pass it on to the 'qgs' daemon and which will use SGX enclaves to validate the attestation report and then create a signed quote for it. The guest owner can validate this quote and the chain of trust for the certificate used to sign it, linking back to Intel as the original hardware vendor

The AMD SEV SNP technology by comparison relies on a physically separate processor known as the PSP (Platform Secure Processor). The guest VM will talk tot he PSP to obtain a signed attestation report. The geust owner can immediately validate this report and the chain of trust for the certificate used to sign it, linking back to AMD as the original hardware vendor.

In the SGX/TDX approach, the SGX architectural enclaves provided by Intel establish the root of trust for the processor, allowing derivation of a key for signing the attestation reports, that has an associated certificate provided by Intel.

In the AMD SEV SNP approach, the PSP firmware is shipped with the platform and has the required signing keys are derived without needing any additional userspace software.

The SEV-SNP is the more classical hardware vendor approach, where the low level pieces are entirely closed source in the system firmware. The SGX/TDX approach is unusual / surprising, with code published as fully open source software. In both cases, the low level code has to be signed by keys controlled by the hardware vendor to establish the root of trust for the system. Thus while the SGX/TDX code is fully open source, there is a constraint in usage of certain output binaries which require the vendor's signature and can't be replaced by a user signature.

Benefit to Fedora

As a general purpose infrastructure technology, SGX can be applied to / used by a wide variety of scenarios / applications.

The primary goal in introducing SGX into Fedora, however, is to support the Fedora KVM virtualization stack when it introduces confidential virtual machines running with Intel TDX. The TDX attestation implementation in currently integrated with Intel CPUs is built on the SGX technology. Attestation is the means by which a guest VM owner, can prove that their VM machine is running in confidential mode on genuine Intel hardware, as opposed to being in "blue pill" environment.

Scope

Proposal owners

Add the following packages to Fedora

• CppMicroServices - a C++ runtime library for building microservices daemons
• sgx-srpm-macros - define some common macros for where SGX content will live in the filesystem tree
• sgx-compat-gccXXX - one (or more) specific GCC versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
• sgx-compat-binutilsXXX - one (or more) specific binutils versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
• sgx-compat-nasmXXX - one (or more) specific NASM versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
• sgx-compat-glibc-headersXXX - one (or more) specific GCC versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
• sgx-compat-kernel-headersXXX - one (or more) specific GCC versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
• linux-sgx-enclavesXXX - one (or more) packages for performing a reproducible build of architectural enclaves
• linux-sgx - provide the SGX platform development headers & libraries, runtime libraries, and supporting daemons

The pre-built, signed architectural enclaves are not always re-created on each SGX release. New binaries are only issued if there was a required feature change, or a security fix required. Thus the package for the reproducible enclave builds is separated out from the general SGX package. Furthermore, to perform a reproducible build of the full set of enclaves, requires potentially more than one linux-sgx-enclavesXXX package. For example SGX 2.23 and 2.24 releases only updated the 'qve' enclave, all other enclaves remained on version 2.22. This in turn creates the requirement for multiple GCC/binutils/nasm versions, at some points. The current newest 2.25 release has updated all enclaves, so initially only a single 'linux-sgx-enclaves-2_25' is expected to be required.

NB, an upstream bug has resulted in different parts of the enclave code being built with different GCC/binutils versions. This should be resolved in future releases:

NB, The requirement for glibc and kernel header packages is likely another upstream bug. The enclave code should be build exclusively against the SGX runtime, which is a completely custom C library. This is being investigated upstream.

Other developers

The kernel functionality required for SGX is already present in Fedora kernel packages, so no work by other maintainers is anticipated.

Release engineering

N/A - does not impact deliverables for releng

Policies and guidelines

• Architectural enclaves.
  • Prebuilt binaries signed by Intel to be permitted under the existing firmware exception
  • Given the availability of complete corresponding source code & a supported reproducible build process, all pre-built binaries MUST be verified through a reproducible build performed in Fedora, using a fully open source toolchain.

• Application enclaves
  • No shipping of pre-built binaries permitted. Everything must follow normal Fedora policies requiring build from source. Any signature must use Fedora controlled persistent keys, or single use keys.

The distinction between architectural enclaves (signed by Intel to bootstrap trust) and application enclaves (signed by Fedora, with trust chained from the AEs), is considered conceptually similar to the distinction between shim (signed by Microsoft to bootstrap trust), vs kernel (signed by Fedora, with trust chained from shim).

Trademark approval

N/A

Alignment with the Fedora Strategy

This aligns with

• "Reaching the world". Including SGX will make the Fedora support for hosting Intel TDX confidential virtual machines feature complete, by enabling attestation by the guest owner
• "Innovation & leadership in technology". SGX is a general purpose infrastructure technology which enables application developers to build systems to securely run sensitive workloads.

Upgrade/compatibility impact

This is a new package set which should not have any upgrade impact, as it will not initially be a dependency of other software. In future it may be pulled in automatically as a dependency in certain KVM deployment scenarios. Even when installed, using anything related to SGX first requires host firmware changes to enable use of the technology. The systemd services provided have their unit files conditionalized on the existence of /dev/sgx_enclave device nodes.

Early Testing (Optional)

Do you require 'QA Blueprint' support? N

The proposed new packages are available for testing via Copr:

 https://copr.fedorainfracloud.org/coprs/berrange/sgx-ng/

How To Test

• Document how to validate a single-socket system by obtaining PCK certificates automatically
• Document how to configure a multi-socket system to enable its registration with Intel services, and request a PCK certificate

User Experience

Initially minimal user experience impact, since on its own it doesn't deliver noticeable end user features, as it is not believed that any existing applications in Fedora are able to leverage SGX.

The initial user benefit will be that users can bootstrap trust in SGX on their Fedora host. This will facilitate users in deploying 3rd party applications of their choosing that utilize SGX.

At a later time, when support for Intel TDX is integrated into KVM and QEMU, the immediate Fedora user benefit will significantly expand.

Limitations

As noted earlier, due to the need for Intel signatures on 4 pre-built architectural enclaves there is a small constraint on user software freedom, if they choose to make use of SGX technology.

For the ide, qe3 and tdqe architectural enclaves, it is possible for users to rebuild the source with arbitrary local modifications, and sign the result with their own local key. The resulting enclaves can be deployed and used at runtime, however, it is not guaranteed that applications will be able to successfully verify quotes produced by such enclaves. It depends on whether applications have been written to expect non-Intel signatures.

For the 'pce architectural enclaves, it is possible for users to rebuild the source with arbitrary local modifications, and sign the result with their key. The resulting enclave can be deployed and used at runtime, however, it will not be possible to obtain a certificate from Intel proving authenticity of the hardware. Thus when verifying quotes, it will not be able to successfully verify the complete chain of trust.

There are no limitations on customization, rebuilds & re-signing of permitted application enclaves, as any usage with signing constraints is proposed to make such enclaves ineligible for inclusion in Fedora.

Dependencies

No existing packages will have a dependency on this change initially. In future, some deployments of QEMU may change to pull in certain SGX packages, to support Intel TDX.

Kernel support for SGX already exists in Fedora:

 $ git grep CONFIG_X86_SGX=y
 kernel-x86_64-debug-fedora.config:CONFIG_X86_SGX=y
 kernel-x86_64-debug-rhel.config:CONFIG_X86_SGX=y
 kernel-x86_64-fedora.config:CONFIG_X86_SGX=y
 kernel-x86_64-rhel.config:CONFIG_X86_SGX=y
 kernel-x86_64-rt-debug-rhel.config:CONFIG_X86_SGX=y
 kernel-x86_64-rt-rhel.config:CONFIG_X86_SGX=y

Contingency Plan

• Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
• Contingency deadline: N/A (not a System Wide Change)
• Blocks release? N/A (not a System Wide Change), Yes/No

Documentation

N/A (not a System Wide Change)