Intel SGX Software Stack
This document represents a proposed Change. As part of the Changes process, proposals are publicly announced in order to receive community feedback. This proposal will only be implemented if approved by the Fedora Engineering Steering Committee.
Summary
The Intel SGX technology enables creation of execution enclaves, whose memory is encrypted and thus protected from all other code running on the machine, including SMM, firmware, kernel and userspace. This proposal is to introduce the SGX host software stack to Fedora, to enable applications and features which have a dependency on SGX technology.
Owner
- Name: Daniel Berrange
- Email: berrange@redhat.com
Current status
- Targeted release: Fedora Linux 42
- Last updated: 2024-10-23
- [Announced]
- [<will be assigned by the Wrangler> Discussion thread]
- FESCo issue: <will be assigned by the Wrangler>
- Tracker bug: <will be assigned by the Wrangler>
- Release notes tracker: <will be assigned by the Wrangler>
Detailed Description
The Intel SGX technology enables creation of execution enclaves, whose memory is encrypted and thus protected from all other code running on the machine, including SMM, firmware, kernel and userspace. While it has many potential use cases, this proposal is focused around the infrastructure needed to enable support for attestation of TDX confidential virtual machines.
The SGX software stack compromises a number of components
- Support for developing new enclaves
- Header files for the enclave code (a minimalist C library, C++ library, crypto and some other misc libraries)
- Static library archives for the enclave code.
- Build helper tools (for signing enclaves, generating code enclave API entrypoints)
- Support for developing applications that use enclaves
- Header files for platform code
- Dynamic libraries for platform code
- Support for deploying applications that use enclaves
- Enclave service daemon - assists unprivileged applications in loading enclaves
- Registration tools - assists platform administrator in acquiring certificates to identity the platform
- Quote generation daemon - assists QEMU in acquiring signed attestation reports for TDX VMs.
Enclaves vs packaging & licensing guidelines
Generally Fedora requires all software to be fully built from source, with the complete & corresponding source made available under one of the permitted code licenses.
There is, however, an exception for firmware binaries which do not have to provide source, and merely required to permit free redistribution.
The question is how to treat enclaves, and deciding this requires understanding what SGX enclaves are and how they are used.
Enclaves as a new platform build target
SGX enclaves are compiled to x86 machine code from C/C++ source, using custom C/C++ runtime libraries, instead of the host OS glibc / libstdc++. IOW, while the enclaves can build with the existing x86 C/C++ toolchain, they are more like a cross-compilation target in the way that a separate non-Linux runtime is used.
The resulting SGX enclave binary uses the ELF format, creating ELF executables, albeit with a .so' file suffix. While the executable binary does have a .interp ELF section, the referenced ld-linux.so cannot load an enclave. A specialized SGX host OS library must be called to create an enclave and load code into it. Given that, the .interp ELF section is in fact redundant and misleading, and has been proposed for removal.
To load an SGX enclave, the ELF binary must have an embedded signature, that is created by the sgx_sign tool, using signing keys provided by whomever is going to attest to trustworthiness of the enclave binary. The signer can be whomever authors the enclave, but in most cases, can also be an arbitrary third party.
The critical point is that whomever is going to later verify the SGX enclave must be willing to trust whomever signed it. Such a trust relationship decision is dependent on the usage context of the enclave in question.
Enclave usage classes
Broadly speaking there are considered to be two classes of enclave:
- Architectural enclaves - these are a small set of common enclaves, created and signed by Intel, which ship as standard in any SGX deployment. They provide a hardware enablement role, bootstrapping the cryptographic root of trust on a platform.
- Application enclaves - this is an arbitrary set of custom enclaves, written by independent software vendors / providers to perform application specific tasks. They are signed by whichever party is to be relied upon to validate trustworthiness of the enclave code.
Both architectural and application enclaves are built and deployed in the same way, using the same runtime libraries. Their difference is largely where they sit in the chain of trust, with architectural enclaves being at the base, below all applications.
Given their use in bootstrapping the SGX platform chain of trust, the architectural enclaves would normally be considered to be serving the same role as platform firmware would. They establish an association with the physical platform hardware, which all other SGX uses relies upon.
Application enclaves, meanwhile are clearly just userspace code, with a special protected execution context. They are created by ordinary application developers as needed to handle some specialized application tasks and have no relationship to platform firmware.
Enclave execution context
What often confuses people is that all enclaves run in ring 3, which is traditionally the realm of userspace, while the enclaves' jobs look like either firmware or userspace jobs depending on context.
Their execution context, while using a region of memory aallocated by & accounted to the loading application, is securely isolated from everything else on the system. The memory assigned for use by an enclave is encrypted and unable to be accessed directly by the application thereafter. Nothing in ring 0 (firmware/kernel), ring 3 (userspace), or even psuedo-ring -1 (SMM) can interact with the enclaves, except via their published API entrypoints which are called via the EENTER instructions using a separate stack. IOW, the traditional view of hardware execution contexts being a simple stack of rings has been rather distorted by the capabilities of SGX.
Enclave licensing
All the code provided by the
The Fedora licensing guidelines describe the requirements for a pre-built binary to quality for the firmware exception allowing source to be omitted.
- The files must be non-executable within the Fedora Linux context (note: this means that the files cannot run on their own, not that they are just chmod -x).
- ✅ Enclaves are distributed as ELF executable files, but this is just a convenient binary format. The ld-linux.so dynamic loader is unable to execute enclave binaries, and indeed a RFE upstream is proposing to eliminate the '.interp' ELF section contents to make this clear. They must be loaded using specialized SGX supporting code that sets up the enclave execution context on the processor.
- The files must not be libraries, within the Fedora Linux context.
- ✅ While the enclave code is distributed as an ELF executable file, with a .so file extension, these are not shared libraries in the sense that Fedora knows them. The ld tool cannot link a binary to an enclave .so file. They must be loaded using specialized SGX supporting code that sets up the enclave execution context on the processor.
- The files must be standalone, not embedded in executable or library code (within the Fedora Linux context)
- ✅ Every enclave is distributed as a standalone ELF executable file
- The files must be necessary for the functionality of open source code being included in Fedora Linux or to enable Fedora Linux to boot on a specific device, where no other reliable and supported mechanisms exist
- ✅ Architectural enclaves. Users of SGX need to verify the chain of trust ultimately links back to a certificate issued by Intel, that attesting that the physical CPU was manufactured by them. The need for Intel to be the root signer is a physical hardware constraint, since only they are able to validate keys that are derived from secrets embedded in the CPU at time of manufacture.
- ❌ Application enclaves. It is valid for enclaves that provide application specific code to be signed by an arbitrary party. Whether a user trusts the signing party is an arbitrary decision they make. Not all uses of SGX will be viable when Fedora acts as a signer, and where not, such enclaves should be omitted from Fedora.
Feedback
Benefit to Fedora
As a general purpose infrastructure technology, SGX can be applied to / used by a wide variety of scenarios / applications.
The primary goal in introducing SGX into Fedora, however, is to support the Fedora KVM virtualization stack when it introduces confidential virtual machines running with Intel TDX. The TDX attestation implementation in currently integrated with Intel CPUs is built on the SGX technology. Attestation is the means by which a guest VM owner, can prove that their VM machine is running in confidential mode on genuine Intel hardware, as opposed to being in "blue pill" environment.
Scope
Proposal owners
Add the following packages to Fedora
- CppMicroServices - a C++ runtime library for building microservices daemons
- sgx-srpm-macros - define some common macros for where SGX content will live in the filesystem tree
- sgx-compat-gccXXX - one (or more) specific GCC versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
- sgx-compat-binutilsXXX - one (or more) specific binutils versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
- sgx-compat-nasmXXX - one (or more) specific NASM versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
- sgx-compat-glibc-headersXXX - one (or more) specific GCC versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
- sgx-compat-kernel-headersXXX - one (or more) specific GCC versions, built with targetted configure arguments, to match the GCC configuration required for enclave reproducible build.
- linux-sgx-enclavesXXX - one (or more) packages for performing a reproducible build of architectural enclaves
- linux-sgx - provide the SGX platform development headers & libraries, runtime libraries, and supporting daemons
The pre-built, signed architectural enclaves are not always re-created on each SGX release. New binaries are only issued if there was a required feature change, or a security fix required. Thus the package for the reproducible enclave builds is separated out from the general SGX package. Furthermore, to perform a reproducible build of the full set of enclaves, requires potentially more than one linux-sgx-enclavesXXX package. For example SGX 2.23 and 2.24 releases only updated the 'qve' enclave, all other enclaves remained on version 2.22. This in turn creates the requirement for multiple GCC/binutils/nasm versions, at some points. The current newest 2.25 release has updated all enclaves, so initially only a single 'linux-sgx-enclaves-2_25' is expected to be required.
NB, an upstream bug has resulted in different parts of the enclave code being built with different GCC/binutils versions. This should be resolved in future releases:
NB, The requirement for glibc and kernel header packages is likely another upstream bug. The enclave code should be build exclusively against the SGX runtime, which is a completely custom C library. This is being investigated upstream.
Other developers
The kernel functionality required for SGX is already present in Fedora kernel packages, so no work by other maintainers is anticipated.
Release engineering
N/A - does not impact deliverables for releng
Policies and guidelines
Define policy requirements wrt shipping of pre-built SGX architectural enclaves (AEs) with cryptographic signatures from Intel.
Two options
- Declare SGX architectural enclaves can be considered firmware, and thus exempt from build-from-source policy requirements.
- Define a policy explicitly for SGX enclaves, that permits shipping pre-built signed binaries for architectural enclaves only, if-and-only-if, a reproducible build can validate the binaries.
NB, in all options, the proposed policy is ONLY intended to application to architectural enclave shipped by Intel. Any application that creates its own SGX enclaves would NOT be covered by this. ie this is NOT proposing to allow arbitrary 3rd party binary enclaves providing application code in Fedora, only the low level hardware enablement/bootstrapping enclave.
Application enclaves are considered out of scope for this feature proposal, but if required in the future, it is suggested that a Fedora signing key would be required for any application enclaves.
The distinction between architectural enclaves (signed by Intel to bootstrap trust) and application enclaves (signed by Fedora, with trust chained from the AEs), is considered conceptually similar to the distinction between shim (signed by Microsoft to bootstrap trust), vs kernel (signed by Fedora, with trust chained from shim).
Trademark approval
N/A
Alignment with the Fedora Strategy
This aligns with
- "Reaching the world". Including SGX will make the Fedora support for hosting Intel TDX confidential virtual machines feature complete, by enabling attestation by the guest owner
- "Innovation & leadership in technology". SGX is a general purpose infrastructure technology which enables application developers to build systems to securely run sensitive workloads.
Upgrade/compatibility impact
This is a new package set which should not have any upgrade impact, as it will not initially be a dependency of other software. In future it may be pulled in automatically as a dependency in certain KVM deployment scenarios. Even when installed, using anything related to SGX first requires host firmware changes to enable use of the technology. The systemd services provided have their unit files conditionalized on the existence of /dev/sgx_enclave device nodes.
Early Testing (Optional)
Do you require 'QA Blueprint' support? N
The proposed new packages are available for testing via Copr:
https://copr.fedorainfracloud.org/coprs/berrange/sgx-ng/
How To Test
- Document how to validate a single-socket system by obtaining PCK certificates automatically
- Document how to configure a multi-socket system to enable its registration with Intel services, and request a PCK certificate
User Experience
Initially minimal user experience impact, since on its own it doesn't deliver noticable end user features. Only once a followup proposal for integrating Intel TDX into KVM is done will the user experience changes.
Dependencies
No existing packages will have a dependency on this change initially. In future, some deployments of QEMU may change to pull in certain SGX packages, to support Intel TDX.
Kernel support for SGX already exists in Fedora:
$ git grep CONFIG_X86_SGX=y kernel-x86_64-debug-fedora.config:CONFIG_X86_SGX=y kernel-x86_64-debug-rhel.config:CONFIG_X86_SGX=y kernel-x86_64-fedora.config:CONFIG_X86_SGX=y kernel-x86_64-rhel.config:CONFIG_X86_SGX=y kernel-x86_64-rt-debug-rhel.config:CONFIG_X86_SGX=y kernel-x86_64-rt-rhel.config:CONFIG_X86_SGX=y
Contingency Plan
- Contingency mechanism: (What to do? Who will do it?) N/A (not a System Wide Change)
- Contingency deadline: N/A (not a System Wide Change)
- Blocks release? N/A (not a System Wide Change), Yes/No
Documentation
N/A (not a System Wide Change)