Virt/SGXEnclaves

An introduction to SGX enclaves

This document should be read as a companion to the Changes/IntelSGX SGX change proposal, providing greater details to aid in understanding the decisions & tradeoffs made.

Enclaves as a platform build target

SGX enclave binaries contain regular x86 machine code, and are built using the regular x86 compiler toolchain. While supporting both 32-bit and 64-bit, in the context of Fedora, only 64-bit support will be provided.

While they use the regular platform toolchain, enclaves are compiled against a custom C/C++ runtime, not the host OS glibc/libstdc++. They are not able to interact with the host OS kernel to make syscalls, the only permitted interaction with the host OS is through the designated public API entrypoints declared for an enclave. IOW, once executing, code inside the enclave is fully self-contained.

This self-contained execution environment and differing runtime libraries, directs us towards considering enclaves to be a separate platform build target. A precedent for packaging this can be seen in the handling of MinGW packages in Fedora, which is treated as a form of cross-compilation, with all content living under a directory tree rooted at /usr/x86_64-w64-mingw32

With this in mind, the SGX enclave code is intended to live under a directory tree rooted at /usr/x86_64-intel-sgx. In the very unlikely event that 32-bit support is later desired, it would use a tree rooted at /usr/i686-intel-sgx.

Beneath these trees, only the lib64 and include directories will be required. A bin directory is not required, as there is no concept of SGX executables, everything is a library.

Enclave binary format and loader

SGX enclave binaries use the ELF, somewhat surprisingly being represented as ELF executables, rather than ELF libraries, despite using a .so file extension. An enclave is very much a library with no concept of a main function, all execution is triggered via public API entrypoints.

As executuables, the ELF files for SGX enclaves will include an .interp section whose content happens to reflect whatever host OS loader was the default in the linker used. For the SGX architectural enclaves, the loader is the NixOS ld-linux.so from the reproducable build environment. Given it serves no purpose and is highly misleading, this ld-linux.so path is proposed for removal.

All SGX enclaves will include a digital signature calculated and embedded by the sgx_sign at build time. The key used for the signature is determined by whoever is building the enclave, and for architectural enclaves is expected to be an Intel owned key.

The signatures are validated at the time the enclaves are loaded, by the SGX specific loader. Historically usage of SGX would be locked down, such that enclaves could not be loaded until the signer's identity was on an allow list. As implemented in Linux today, however, there are no restrictions on who can sign enclaves that are loaded. Linux uses the "Flexible Launch Control" feature of newer SGX firmware to dynamically change the allow list to match that needed by whatever enclave is being loaded. Ability to load enclaves is controlled simply by permissions on the /dev/sgx_provision device node.

The identity of the signer of the enclave is associated with the loaded enclave and can be used as an input to a key derivation algorithm, such that if the enclave code is rebuilt & re-signed by a 3rd party, it is guaranteed that any derived keys will change, preventing impersonation of the original enclave.

Enclave execution context

Enclaves run in ring 3, which is traditionally the realm of userspace. The execution context, while using a region of memory allocated by (& accounted to) the loading application, is securely isolated from everything else on the system including the loading application.

Running in ring 3 prevents enclaves running privileged instructions, but the normal hierarchical rules of access protection between rings do not apply. Nothing in ring 0 (firmware / kernel), ring 3 (userspace), or even psuedo-ring -1 (SMM) can interact with the enclaves memory. Nor can the enclave code access anything outside their own enclave.

The only interaction beween enclaves and the outside world is via calls to their published API entrypoints, using the EENTER instructions and a separate stack. IOW, the traditional view of hardware execution contexts being a simple stack of rings has been somewhat distorted by the capabilities of SGX.

They are probably best thought of as living in a set of additiaonl execution contexts, completely distinct from (or off to the side of) the traditional protection rings.

Enclave usage roles

Broadly speaking enclaves can be considered to be serving in one of two conceptual roles.

1. Architectural enclaves. A small finite set of common enclaves, created and signed by Intel, which ship as standard in any SGX deployment. They provide a hardware enablement role, bootstrapping the cryptographic root of trust on a platform and providing certain core services for other enclaves.

1. Application enclaves. An arbitrary set of custom enclaves, written by independent software vendors / providers to perform application specific tasks. They are signed by whichever party is to be relied upon to validate trustworthiness of the enclave code. IOW, they should be considered part of the user application stack.

Both architectural and application enclaves are built and deployed in the same way, using the same runtime libraries. Their difference is largely where they sit in the chain of trust, with architectural enclaves being at the base, below all application enclaves.

Given their use in bootstrapping the SGX platform chain of trust, and providing trusted services to other enclaves, the architectural enclaves should normally be considered to be serving the same role as platform firmware would. IOW, they are firmware for SGX.

Application enclaves, meanwhile are clearly just userspace code, with a special protected execution context. They are created by ordinary application developers as needed to handle some specialized application tasks and have no relationship to platform firmware.

The architectural enclaves

There are 8 so called "architectural enclaves" built, signed and distributed by Intel.

• libsgx_le.signed.so - "Launch Enclave (le)"
  • This was historically used to control what enclaves are able to be launched on a platform. Modern systems allow loading on any enclave no matter who signed it.
  • This enclave requires an Intel signature
  • This architectural enclave is considered obsolete & is NOT intended to be shipped in Fedora.

• libsgx_qe.signed.so - "Quoting Enclave (qe)"
  • This was historically used in attestation of enclaves, using the Enhanced Privacy ID (EPID) scheme, which is replaced by RSA in SGX DCAP (Data Center Attestation Primitives) software.
  • This enclave requires an Intel signature
  • This architectural enclave is considered obsolete & is NOT intended to be shipped in Fedora.

• libsgx_pve.signed.so - "ProVisioning Enclave (pve)"
  • This was also historically used in attestation of enclaves, using the Enhanced Privacy ID (EPID) scheme, which is replaced by RSA in SGX DCAP (Data Center Attestation Primitives) software.
  • This enclave requires an Intel signature
  • This architectural enclave is considered obsolete & is NOT intended to be shipped in Fedora.

• libsgx_pce.signed.so - "Platform Certification Enclave (pce)"
  • This certifies all other enclaves running on the system. The identity of the Intel's signing key, together with secrets inside the CPU set in fuses at time of manufacture, to derive a key that is known only to the enclave and Intel. This is needed in order to obtain a certificate from Intel that proves the hardware is genuine.
  • This enclave requires an Intel signature

• libsgx_id_enclave.signed.so - "Identification Enclave (ide)"
  • This is used to extract some identification information about the processor that is needed to acquire a platform /processor certificate after deploying or re-provisioning a platform.
  • This enclave recommends an Intel signature

• libsgx_qe3.signed.so - "Quoting Enclave (qe3)"
  • This is used in attestation of enclaves, using the newer RSA scheme in SGX DCAP
  • This enclave recommends an Intel signature

• libsgx_tdqe3.signed.so - "Trust Domain Quoting Enclave (tdqe)"
  • This is used in attestation of TDX virtual machines, using the newer RSA scheme in SGX DCAP
  • This enclave recommends an Intel signature

• libsgx_qve.signed.so - "Quote Verification Enclave (qve)"
  • This is used to provide a way to verify attestation reports on an otherwise untrusted machine using SGX DCAP.
  • This enclave recommends an Intel signature
  • The pre-built binary for this enclave includes a build of openssl that potentially contains forbidden crypto & is NOT intended to be shipped in Fedora unless the crypto can be approved or eliminated

Rebuilding & self-signing architectural enclaves

As noted earlier the SGX enclave loader will validate signatures on enclaves, but at the same time it will allow any signer, even for the architectural enclaves. With this in mind, any user (or 3rd party) can build and customize any enclave for which the source is made available, signing with their own key, instead of the original vendor's key.

Rebuilding & changing the signing key of an enclave is not, however, guaranteed to be free of consequences. The key factor is that the identity of the signer, might be used as an input to key generation algorithms such that changing the signer will change the keys in use. This may be fine for some enclaves, while putting limitations on other enclaves.

In terms of the architectural enclaves, the libsgx_pce.signed.so enclave is the most fundamental. It is used to establish trust in ("certify") in other enclaves running on the system, providing the means for a user to prove the enclaves executed on genuine Intel CPUs inside an SGX execution context. pce derives a PCK (Processor Certification Key) based on the identity of its own signer, combined with secrets embedded in the CPU at time of manufacture. Upon provisioning a new machine with SGX, this PCK is extracted and sent to an public API service run by Intel, whom validates is and in return provides a PCK Certificate. This certificate is signed by a chain of CAs controlled by Intel, back to a well known root CA. If the signer of pce is a 3rd party, Intel's public API service will decline to issue any PCK Certificate for the host.

When an application is verifying a SGX quote, there are a number of steps involve, the last of which is validating a signature produced by the PCK and checking that the PCK certificate chain ends with Intel's well known root CA, thus proving that the enclave ran on genuine hardware. If not PCK certificate can be obtained, then it will not be possible to fully validate SGX quotes.

The other architectural enclaves, libsgx_qe.signed.so and libsgx_tdqe.signed.so also create keys of their own and use them for signing data. There is more potential flexibility for users and applications in this case. An application verifying a SGX quote must be told the identity of the signer of qe and tdqe and while by default applications will expect an Intel identity, it is possible for applications to be written to accept alternative identities.

In summary

• All architectural enclaves can be rebuilt, with arbitrary code customizations, and signed by third parties
• If the pce enclave is signed by a non-Intel signature, SGX quotes can only be partially verified, due to inability to acquire a PCK Certificate matching the custom pce implementation
• If the qe or tdqe enclaves are signed by a non-Intel signature, it might be possible to supply this non-Intel identity to applications verifying SGX quotes

Who benefits from use of enclaves

When dealing with cryptographically attested software, it is important to consider who is benefiting from verifying the execution context. There are generally three parties involved in such software

• Providing party - the organization or individual that is signing the software being used. This could be the original software author/vendor, or an intermediary such as a Linux distributor.
• Deploying party - the organization or individual that is running the software in question. This could be a so called 'end user', or it could be a service provider running something on their behalf.
• Verifying party - the organization or individual that is relying on verification of the executed software. This could be the so called 'end user', but it could be a 3rd party attempting to place constraints on usage (e.g. DRM)

In all cases, Intel acts as a "verifying party" when a platform is deployed or provisioned, when the platform owner requests a PCK Certificate to prove their hardware is authentic. This is a low level a bootstrapping task, to enable higher level use of SGX. Thus while Intel is the verifying party at a low level, it should not be considered a beneficiary of SGX in this context.

Use of SGX as a mechanism for proving the integrity of a virtual machine has multiple beneficiaries, several directly being Fedora users.

The Fedora user who owns the hypervisor OS install benefits in being able to prove integrity of the service they provide to guest owners. By using SGX to prove that a VM runs with TDX, host owner has shown they do not have access to guest memory. This reduces risk and liability to the hosting provider, whether the danger comes from an external hacker compromising the hypervisor, or from one of their own sysadmins going rogue.

The Fedora user of the VM similarly benefits in being able to prove integrity of their guest OS, giving them confidence that their sensitive information is significantly less likely to be at risk of compromise by the hosting provider's infrastructure, as compared to a traditional non-confidential VM. Non-Fedora OS can also be run as confidential VMs and benefit from TDX, and thus from SGX.

It is possible to extend the trust relationship out to a remote 3rd party who is interacting with services provided by the VM. Such a remote person can prove that the service they are interacting with is running in a trusted virtual machine, and in turn increase their willingness to provide sensitive information to the remote service. The remote 3rd party, may or may not, be another Fedora user. In addition the service running inside the VM, may or may not be acting in the best interests on the remote 3rd party. The latter question is outside the scope of Fedora's responsibility, as a platform technology provider.

Conversely a well known example of where SGX is used in an end user hostile manner is as a form of DRM in BluRay players. In this case the verifying party is the commercial organization aiming to control the manner in which end users consume content they have purchased. Using SGX for DRM in Fedora applications would not be beneficial to the freedom of Fedora users.

IOW, as with all technologies, SGX can be used in a variety of ways, some of which are beneficial to Fedora users and don't restrict their freedom, while other scenarios are directly hostile to Fedora users and their freedom. The goal of Fedora is to permit the application use cases which are compatible with user freedom, while excluding the harmful usage.