An introduction to SGX enclaves
Enclaves as a platform build target
SGX enclave binaries contain regular x86 machine code, and are built using the regular x86 compiler toolchain. While supporting both 32-bit and 64-bit, in the context of Fedora, only 64-bit support will be provided.
While they use the regular platform toolchain, enclaves are compiled against a custom C/C++ runtime, not the host OS glibc/libstdc++. They are not able to interact with the host OS kernel to make syscalls, the only permitted interaction with the host OS is through the designated public API entrypoints declared for an enclave. IOW, once executing, code inside the enclave is fully self-contained.
This self-contained execution environment and differing runtime libraries, directs us towards considering enclaves to be a separate platform build target. A precedent for packaging this can be seen in the handling of MinGW packages in Fedora, which is treated as a form of cross-compilation, with all content living under a directory tree rooted at /usr/x86_64-w64-mingw32
With this in mind, the SGX enclave code is intended to live under a directory tree rooted at /usr/x86_64-intel-sgx. In the very unlikely event that 32-bit support is later desired, it would use a tree rooted at /usr/i686-intel-sgx.
Beneath these trees, only the lib64 and include directories will be required. A bin directory is not required, as there is no concept of SGX executables, everything is a library.
Enclave binary format and loader
SGX enclave binaries use the ELF, somewhat surprisingly being represented as ELF executables, rather than ELF libraries, despite using a .so file extension. An enclave is very much a library with no concept of a main function, all execution is triggered via public API entrypoints.
As executuables, the ELF files for SGX enclaves will include an .interp section whose content happens to reflect whatever host OS loader was the default in the linker used. For the SGX architectural enclaves, the loader is the NixOS ld-linux.so from the reproducable build environment. Given it serves no purpose and is highly misleading, this ld-linux.so path is proposed for removal.
All SGX enclaves will include a digital signature calculated and embedded by the sgx_sign at build time. The key used for the signature is determined by whoever is building the enclave, and for architectural enclaves is expected to be an Intel owned key.
The signatures are validated at the time the enclaves are loaded, by the SGX specific loader. Historically usage of SGX would be locked down, such that enclaves could not be loaded until the signer's identity was on an allow list. As implemented in Linux today, however, there are no restrictions on who can sign enclaves that are loaded. Linux uses the "Flexible Launch Control" feature of newer SGX firmware to dynamically change the allow list to match that needed by whatever enclave is being loaded. Ability to load enclaves is controlled simply by permissions on the /dev/sgx_provision device node.
The identity of the signer of the enclave is associated with the loaded enclave and can be used as an input to a key derivation algorithm, such that if the enclave code is rebuilt & re-signed by a 3rd party, it is guaranteed that any derived keys will change, preventing impersonation of the original enclave.
Enclave execution context
Enclaves run in ring 3, which is traditionally the realm of userspace. The execution context, while using a region of memory allocated by (& accounted to) the loading application, is securely isolated from everything else on the system including the loading application.
Running in ring 3 prevents enclaves running privileged instructions, but the normal hierarchical rules of access protection between rings do not apply. Nothing in ring 0 (firmware / kernel), ring 3 (userspace), or even psuedo-ring -1 (SMM) can interact with the enclaves memory. Nor can the enclave code access anything outside their own enclave.
The only interaction beween enclaves and the outside world is via calls to their published API entrypoints, using the EENTER instructions and a separate stack. IOW, the traditional view of hardware execution contexts being a simple stack of rings has been somewhat distorted by the capabilities of SGX.
They are probably best thought of as living in a set of additiaonl execution contexts, completely distinct from (or off to the side of) the traditional protection rings.
Enclave usage roles
Broadly speaking enclaves can be considered to be serving in one of two conceptual roles.
- Architectural enclaves. A small finite set of common enclaves, created and signed by Intel, which ship as standard in any SGX deployment. They provide a hardware enablement role, bootstrapping the cryptographic root of trust on a platform and providing certain core services for other enclaves.
- Application enclaves. An arbitrary set of custom enclaves, written by independent software vendors / providers to perform application specific tasks. They are signed by whichever party is to be relied upon to validate trustworthiness of the enclave code. IOW, they should be considered part of the user application stack.
Both architectural and application enclaves are built and deployed in the same way, using the same runtime libraries. Their difference is largely where they sit in the chain of trust, with architectural enclaves being at the base, below all application enclaves.
Given their use in bootstrapping the SGX platform chain of trust, and providing trusted services to other enclaves, the architectural enclaves should normally be considered to be serving the same role as platform firmware would. IOW, they are firmware for SGX.
Application enclaves, meanwhile are clearly just userspace code, with a special protected execution context. They are created by ordinary application developers as needed to handle some specialized application tasks and have no relationship to platform firmware.
The architectural enclaves
There are 8 so called "architectural enclaves" built, signed and distributed by Intel.
- libsgx_le.signed.so - "Launch Enclave (le)"
- This was historically used to control what enclaves are able to be launched on a platform. Modern systems allow loading on any enclave no matter who signed it.
- This enclave requires an Intel signature
- This architectural enclave is considered obsolete & is NOT intended to be shipped in Fedora.
- libsgx_qe.signed.so - "Quoting Enclave (qe)"
- This was historically used in attestation of enclaves, using the Enhanced Privacy ID (EPID) scheme, which is replaced by RSA in SGX DCAP (Data Center Attestation Primitives) software.
- This enclave requires an Intel signature
- This architectural enclave is considered obsolete & is NOT intended to be shipped in Fedora.
- libsgx_pve.signed.so - "ProVisioning Enclave (pve)"
- This was also historically used in attestation of enclaves, using the Enhanced Privacy ID (EPID) scheme, which is replaced by RSA in SGX DCAP (Data Center Attestation Primitives) software.
- This enclave requires an Intel signature
- This architectural enclave is considered obsolete & is NOT intended to be shipped in Fedora.
- libsgx_pce.signed.so - "Platform Certification Enclave (pce)"
- This certifies all other enclaves running on the system. The identity of the Intel's signing key, together with secrets inside the CPU set in fuses at time of manufacture, to derive a key that is known only to the enclave and Intel. This is needed in order to obtain a certificate from Intel that proves the hardware is genuine.
- This enclave requires an Intel signature
- libsgx_id_enclave.signed.so - "Identification Enclave (ide)"
- This is used to extract some identification information about the processor that is needed to acquire a platform /processor certificate after deploying or re-provisioning a platform.
- This enclave recommends an Intel signature
- libsgx_qe3.signed.so - "Quoting Enclave (qe3)"
- This is used in attestation of enclaves, using the newer RSA scheme in SGX DCAP
- This enclave recommends an Intel signature
- libsgx_tdqe3.signed.so - "Trust Domain Quoting Enclave (tdqe)"
- This is used in attestation of TDX virtual machines, using the newer RSA scheme in SGX DCAP
- This enclave recommends an Intel signature
- libsgx_qve.signed.so - "Quote Verification Enclave (qve)"
- This is used to provide a way to verify attestation reports on an otherwise untrusted machine using SGX DCAP.
- This enclave recommends an Intel signature
- The pre-built binary for this enclave includes a build of openssl that potentially contains forbidden crypto & is NOT intended to be shipped in Fedora unless the crypto can be approved or eliminated
Out of the four architectural enclaves that require an Intel signature, only one is intended to be shipped in Fedora.
While the remaining four architectural enclaves can theoretically have a non-Intel signature, if this is done it will increase friction of usage. The identity of the signer of enclaves becomes embedded in quotes, and is checked by the user (or software) validating the quote which is unlikely to be expecting a non-Intel signer for these architectural enclaves.
Thus it is considered that all shipped architectural enclaves need to have the Intel signature.
Users who wish to change & rebuild architectural enclaves from source can sign the result with their own keys, for all except the libsgx_pce.signed.so which unavoidably requires an Intel signature in order to derive the correct shared keys. If doing this, they will have to ensure that applications they use for verification are able to support use of non-Intel signers.