Virt/SGXEnclaves

An introduction to SGX enclaves

Enclaves as a platform build target

SGX enclave binaries contain regular x86 machine code, and are built using the regular x86 compiler toolchain. While supporting both 32-bit and 64-bit, in the context of Fedora, only 64-bit support will be provided.

While they use the regular platform toolchain, enclaves are compiled against a custom C/C++ runtime, not the host OS glibc/libstdc++. They are not able to interact with the host OS kernel to make syscalls, the only permitted interaction with the host OS is through the designated public API entrypoints declared for an enclave. IOW, once executing, code inside the enclave is fully self-contained.

This self-contained execution environment and differing runtime libraries, directs us towards considering enclaves to be a separate platform build target. A precedent for packaging this can be seen in the handling of MinGW packages in Fedora, which is treated as a form of cross-compilation, with all content living under a directory tree rooted at /usr/x86_64-w64-mingw32

With this in mind, the SGX enclave code is intended to live under a directory tree rooted at /usr/x86_64-intel-sgx. In the very unlikely event that 32-bit support is later desired, it would use a tree rooted at /usr/i686-intel-sgx.

Beneath these trees, only the lib64 and include directories will be required. A bin directory is not required, as there is no concept of SGX executables, everything is a library.

Enclave binary format and loader

SGX enclave binaries use the ELF, somewhat surprisingly being represented as ELF executables, rather than ELF libraries, despite using a .so file extension. An enclave is very much a library with no concept of a main function, all execution is triggered via public API entrypoints.

As executuables, the ELF files for SGX enclaves will include an .interp section whose content happens to reflect whatever host OS loader was the default in the linker used. For the SGX architectural enclaves, the loader is the NixOS ld-linux.so from the reproducable build environment. Given it serves no purpose and is highly misleading, this ld-linux.so path is proposed for removal.

All SGX enclaves will include a digital signature calculated and embedded by the sgx_sign at build time. The key used for the signature is determined by whoever is building the enclave, and for architectural enclaves is expected to be an Intel owned key.

The signatures are validated at the time the enclaves are loaded, by the SGX specific loader. Historically usage of SGX would be locked down, such that enclaves could not be loaded until the signer's identity was on an allow list. As implemented in Linux today, however, there are no restrictions on who can sign enclaves that are loaded. Linux uses the "Flexible Launch Control" feature of newer SGX firmware to dynamically change the allow list to match that needed by whatever enclave is being loaded. Ability to load enclaves is controlled simply by permissions on the /dev/sgx_provision device node.

The identity of the signer of the enclave is associated with the loaded enclave and can be used as an input to a key derivation algorithm, such that if the enclave code is rebuilt & re-signed by a 3rd party, it is guaranteed that any derived keys will change, preventing impersonation of the original enclave.

Enclave execution context

Enclaves run in ring 3, which is traditionally the realm of userspace. The execution context, while using a region of memory allocated by (& accounted to) the loading application, is securely isolated from everything else on the system including the loading application.

Running in ring 3 prevents enclaves running privileged instructions, but the normal hierarchical rules of access protection between rings do not apply. Nothing in ring 0 (firmware / kernel), ring 3 (userspace), or even psuedo-ring -1 (SMM) can interact with the enclaves memory. Nor can the enclave code access anything outside their own enclave.

The only interaction beween enclaves and the outside world is via calls to their published API entrypoints, using the EENTER instructions and a separate stack. IOW, the traditional view of hardware execution contexts being a simple stack of rings has been somewhat distorted by the capabilities of SGX.

They are probably best thought of as living in a set of additiaonl execution contexts, completely distinct from (or off to the side of) the traditional protection rings.

Enclave usage roles

Broadly speaking enclaves can be considered to be serving in one of two conceptual roles.

1. Architectural enclaves. A small finite set of common enclaves, created and signed by Intel, which ship as standard in any SGX deployment. They provide a hardware enablement role, bootstrapping the cryptographic root of trust on a platform and providing certain core services for other enclaves.

1. Application enclaves. An arbitrary set of custom enclaves, written by independent software vendors / providers to perform application specific tasks. They are signed by whichever party is to be relied upon to validate trustworthiness of the enclave code. IOW, they should be considered part of the user application stack.

Both architectural and application enclaves are built and deployed in the same way, using the same runtime libraries. Their difference is largely where they sit in the chain of trust, with architectural enclaves being at the base, below all application enclaves.

Given their use in bootstrapping the SGX platform chain of trust, and providing trusted services to other enclaves, the architectural enclaves should normally be considered to be serving the same role as platform firmware would. IOW, they are firmware for SGX.

Application enclaves, meanwhile are clearly just userspace code, with a special protected execution context. They are created by ordinary application developers as needed to handle some specialized application tasks and have no relationship to platform firmware.

The architectural enclaves

There are 8 so called "architectural enclaves" built, signed and distributed by Intel.

• libsgx_le.signed.so - "Launch Enclave (le)"
  • This was historically used to control what enclaves are able to be launched on a platform. Modern systems allow loading on any enclave no matter who signed it.
  • This enclave requires an Intel signature
  • This architectural enclave is considered obsolete & is NOT intended to be shipped in Fedora.

• libsgx_qe.signed.so - "Quoting Enclave (qe)"
  • This was historically used in attestation of enclaves, using the Enhanced Privacy ID (EPID) scheme, which is replaced by RSA in SGX DCAP (Data Center Attestation Primitives) software.
  • This enclave requires an Intel signature
  • This architectural enclave is considered obsolete & is NOT intended to be shipped in Fedora.

• libsgx_pve.signed.so - "ProVisioning Enclave (pve)"
  • This was also historically used in attestation of enclaves, using the Enhanced Privacy ID (EPID) scheme, which is replaced by RSA in SGX DCAP (Data Center Attestation Primitives) software.
  • This enclave requires an Intel signature
  • This architectural enclave is considered obsolete & is NOT intended to be shipped in Fedora.

• libsgx_pce.signed.so - "Platform Certification Enclave (pce)"
  • This certifies all other enclaves running on the system. The identity of the Intel's signing key, together with secrets inside the CPU set in fuses at time of manufacture, to derive a key that is known only to the enclave and Intel. This is needed in order to obtain a certificate from Intel that proves the hardware is genuine.
  • This enclave requires an Intel signature

• libsgx_id_enclave.signed.so - "Identification Enclave (ide)"
  • This is used to extract some identification information about the processor that is needed to acquire a platform /processor certificate after deploying or re-provisioning a platform.
  • This enclave recommends an Intel signature

• libsgx_qe3.signed.so - "Quoting Enclave (qe3)"
  • This is used in attestation of enclaves, using the newer RSA scheme in SGX DCAP
  • This enclave recommends an Intel signature

• libsgx_tdqe3.signed.so - "Trust Domain Quoting Enclave (tdqe)"
  • This is used in attestation of TDX virtual machines, using the newer RSA scheme in SGX DCAP
  • This enclave recommends an Intel signature

• libsgx_qve.signed.so - "Quote Verification Enclave (qve)"
  • This is used to provide a way to verify attestation reports on an otherwise untrusted machine using SGX DCAP.
  • This enclave recommends an Intel signature
  • The pre-built binary for this enclave includes a build of openssl that potentially contains forbidden crypto & is NOT intended to be shipped in Fedora unless the crypto can be approved or eliminated

Rebuilding & self-signing architectural enclaves

As noted earlier the SGX enclave loader will validate signatures on enclaves, but at the same time it will allow any signer, even for the architectural enclaves. With this in mind, any user (or 3rd party) can build and customize any enclave for which the source is made available, signing with their own key, instead of the original vendor's key.

Rebuilding & changing the signing key of an enclave is not, however, guaranteed to be free of consequences. The key factor is that the identity of the signer, might be used as an input to key generation algorithms such that changing the signer will change the keys in use. This may be fine for some enclaves, while putting limitations on other enclaves.

In terms of the architectural enclaves, the libsgx_pce.signed.so enclave is the most fundamental. It is used to establish trust in ("certify") in other enclaves running on the system, providing the means for a user to prove the enclaves executed on genuine Intel CPUs inside an SGX execution context. pce derives a PCK (Processor Certification Key) based on the identity of its own signer, combined with secrets embedded in the CPU at time of manufacture. Upon provisioning a new machine with SGX, this PCK is extracted and sent to an public API service run by Intel, whom validates is and in return provides a PCK Certificate. This certificate is signed by a chain of CAs controlled by Intel, back to a well known root CA. If the signer of pce is a 3rd party, Intel's public API service will decline to issue any PCK Certificate for the host.

When an application is verifying a SGX quote, there are a number of steps involve, the last of which is validating a signature produced by the PCK and checking that the PCK certificate chain ends with Intel's well known root CA, thus proving that the enclave ran on genuine hardware. IOW, if not PCK certificate can be obtained, then it will not be possible to fully validate SGX quotes.