This commit is a result of debugging a rare race condition during build
of some of our examples, which resulted in "enclave EINIT failed -
Invalid measurement" error.
It turns out that Make in versions that doesn't support the `&:`
operator ("Rules with Grouped Targets") silently ignores it and calls
the recipe for each target separately, without even a warning.
Signed-off-by: Michał Kowalczyk <mkow@invisiblethingslab.com>
Change log (most important only):
- unify CPU context structures - now we have only one version -
`PAL_CONTEXT` - which is shared between LibOS and PALs and it should
depend only on the host architecture (not OS),
- syscalls emulation changed:
- dedicated LibOS stack is now used for syscalls emulation,
- removed one indirection level in syscalls table - now it stores
`shim_do_*` functions directly,
- signal handling - completely rewritten:
- all signal queues use proper locking schemes now,
- signals are handled *only* when returning to the user app from LibOS
or PAL,
- nested signals are now possible,
- the app is allowed to jump out of signal handler with the same
sematics as on normal Linux,
- signal altstack is now fully supported,
- syscall restarting is now supported,
- doing a backtrace from the signal handler works properly,
- disallow injecting host-level signals, with one exception, see
`sys.enable_sigterm_injection` manifest option for more details.
Instead of 'loader.debug_type', introduce 'loader.log_level'
and 'loader.log_file', along with a set of definitions for
logging at a chosen level.
For now, the call sites keep using the legacy macros (SGX_DBG and
debug()), because converting them all will conflict with other
big changes in the code base. The existing LibOS calls are
assumed to be at 'info' level.
This is the next part of the great loader rework, with a lot of breaking changes:
- Complete removal of the "trusted children" thing - now children
processes can be spawned arbitrarily and from arbitrary mountpoint
types, without any additional configuration needed.
- There's a new, required option in the manifest: `libos.entrypoint` - it
specifies the URI to the entry binary in the first process. There's no
need anymore to name the manifest and the first binary identically.
- On SGX, the main binary is not measured in MRENCLAVE anymore - only
PAL, LibOS and the manifest are measured. This is enough to bind
MRENCLAVE to a specific entrypoint user executable if wanted - it
just has to be mounted as a trusted file.
- All Graphene SGX enclaves have now exactly the same MRENCLAVE. This is
a hash of a "Graphene stub", which can "fork" into one of two states
in runtime: initial process or child. The initial process creates a
new "Graphene namespace" with a clean state, it can also be attested
remotely (contrary to child processes). The initial process can spawn
children processes by spawning a Graphene stub and directing it to
start in the child mode. It then attests it locally, and if
successful, establishes an encrypted pipe, "connects" to its own
namespace and treats as trusted (including sending protected files
key).
- Now, there's only one, central manifest describing the initial state
of a Graphene instance which can be spawned from it (previously, each
process required a separate manifest which could have different
configuration - which wasn't actually supported and didn't make sense
design-wise). One downside of central manifests is that all processes
require the same enclave configuration (e.g. size), but that was
already the case so far because of broken checkpointing code. Also,
this is only a temporary problem, which will cease to exist after the
introduction of EDMM.
- `sgx.static_address` was renamed to `sgx.nonpie_binary` and now has to
be inserted manually by users (`sgx_sign` tools doesn't know about the
binaries run inside, which can be even provided or generated in
runtime by the user's workload).
- Caveat: the memory gap for non-PIE executables was removed because it
requires adding a new option to the manifest to be cleanly
implemented. This is left for some future loader rework PR.
This is a major refactor of the way manifests are loaded and handled,
which will be followed by a complete rework of the loader code (which
will include e.g. centralized config).
Changes/fixes:
- Huge part of manifest handling was refactored and untangled.
- Starting without a manifest is now disallowed. This was actually
accidentally broken for some time and no one complained. It also makes
little sense in practice and in Graphene's overall design, e.g. it
conflicts with protected argv.
- Now we only allow starting by giving the executable, not manifest (the
magic resolution logic was removed).
- Now manifests are sent over pipes between parent and children, instead
of children finding and loading them on their own. This is a
preparation for the upcoming centralized manifests change.
- Previously manifests were parsed 2 times on Linux and 3 times on
Linux-SGX (by untrusted PAL, trusted PAL and LibOS). This is now
fixed.
- The common `pal_main()` now requires that the backend-specific PAL
loader loads the manifest before calling it. SGX code already has to
do it (for proper initialization), so let's unify this interface for
all PALs.
- Fix for a PAL crash when manifest size was divisible by page size
(sic!). NULL termination was missing, but most of the time the padding
to page size saved Graphene from crashing.
Pylint output was filtered so that many files with existing pylint
violations were allowed to stay broken.
I made sure all files pass pylint, but whitelisted some rules that
we commonly disable:
* missing docstrings: most of the code is tests/internal anyway
* invalid-name: too many violations, and we commonly use one- or
two-character names (like "a, b" or "t1, t2") which is
disallowed by this rule; we could tweak it and then fix
remaining violations such as camel-case or lowercase constants
* fixme: we leave TODOs as a matter of practice, same as in C
* high-level style rules like too-few-* and too-many-*,
no-self-use
Hopefully that will make using pylint less annoying, while also
catching serious issues (such as unused variables or imports).
The manifest syntax stays exactly the same, including 0 and 1
integers to denote boolean values (this is done for ease of porting
and can be fixed in future commits). The only visible change is
surrounding strings in the manifest with quotes (requirement of
TOML). All manifests and Makefiles of our tests and example apps are
ported to the new TOML syntax. Documentation is updated.
On debug builds, the standard output is littered with debug log from
graphene, so this likely never worked properly. This wasn't catched by
jenkins either.
Previously, Graphene assumed that if it was built with the DCAP
SGX driver or in-kernel SGX driver, then it should use DCAP/ECDSA
based attestation. In fact, the SGX driver has nothing to do with
the attestation scheme. This commit allows to use EPID based
attestation even when Graphene is built with the DCAP SGX driver.
Supporting these options complicates the design of Graphene and loading
logic significantly, providing little useful functionality:
- loader.exec:
- the main user of it were our tests
- worked only for the first process spawned inside Graphene, as it
was a unidirectional manifest->binary mapping, so the child
process didn't know about the corresponding manifest.
- sgx.sigfile:
- probably all existing usages of it were completely redundant
- was resolved relatively to CWD instead of the executable location,
which made it mostly useless
From now on, the correct location of the files is:
- either place the manifest and sigfile next to the binary, with a
matching name, or
- create a symlink to the binary in the folder where manifests are
stored and launch it through this symlink
The purpose of this example was to show how to break a cyclic dependency
when spawning processes, using an ugly hack. This hack will stop working
after manifest resolution cleanup, which will be followed by a loader
rework, which finally will make this use-case work out-of-the-box,
without requiring any hacks.
The manifest option "sgx.allow_file_creation" is useless (most
real-world apps will set it to "1" anyway). So this commit simply
removes this option and always allows to create files.
This commit forces OpenVINO to use OpenMP library instead of TBB.
This is due to the currently missing functionality in Graphene to
correctly initialize TBB (missing /sys/devices/ support).
This example shows how to write manifests for multi-process applications
and highlights the trick of breaking an endless loop of SGX trusted
children by using symlinks. This example also stresses the FS
checkpointing system of Graphene.
Our current TensorFlow example is actually TensorFlow Lite (less
functional version of TensorFlow, used only for inference). Rename
it to avoid confusion of new users.
Trusted child enclaves must inherit the provisioned master (wrap) key
for protected files from the parent enclave on fork/clone. This commit
adds this functionality and modifies the RA-TLS secret provisioning
PF client to test this (including in Jenkins).
Kudos to Li Xun for reporting this bug and proposing a solution.
Previously, we introduced `sgx.zero_heap_on_demand` in Linux-SGX as a
knob to trade off runtime degradation on memory allocations for faster
enclave start-up times. This was an incorrect fix because Linux-SGX's
`_DkVirtualMemoryAlloc()` always zeroess the requested memory region,
so there was a double-zero of the heap at runtime. Note that LibOS
layer silently assumes that `_DkVirtualMemoryAlloc()` zeroes out the
memory, and many applications rely on this (Apache, Blender in my
experiments). Thus, this commit keeps the zero-out in
`_DkVirtualMemoryAlloc()` and removes zero-outs on enclave init and in
`get_enclave_pages()`. This renders `sgx.zero_heap_on_demand`
useless, so this manifest option is also removed. Also note that this
commit doesn't introduce any performance degradation (in fact, now
Graphene behaves as if `sgx.zero_heap_on_demand = 1` always).
The commit: [Examples] tensorflow: Fix default `make` target
replaced the target 'default' with 'all'.
However, the targets 'run-graphene', 'run-native' and 'check'
still depend on 'default', so it is not possible to use them.
To fix this, modify these targets to depend on 'all' instead.
This commit adds the ability to provision the wrap (master) key for
protected files at runtime (in contrast to previous approach of
hard-coding `protected_files_key` in the manifest as a temporary
solution). This is achieved as follows:
- New PAL API `DkSetProtectedFilesKey()` is added.
- New writable pseudo-file `/dev/attestation/protected_files_key` is
added. It calls `DkSetProtectedFilesKey()` after it was written to.
- New `SECRET_PROVISION_SET_PF_KEY` option is added to the Secret
Provisioning library. If it is set, the library assumes that the
first provisioned secret is the wrap key for PF and writes it into
the new pseudo-file.
The Secret Provisioning example `ra-tls-secret-prov` is updated to
include the new protected-files client. This client receives the wrap
key for PF via secret provisioning and reads & outputs the protected
file `files/input.txt`.
*NOTE*: The current implementation of provisioning the wrap key does
not work for `loader.argv_src_file` and `loader.env_src_file` if they
point to protected files (because provisioning happens after setting
up arguments and environment variables).
The client part of RA-TLS demo only ran outside SGX. This patch optionally
allows the client to run in Graphene SGX as well. This is useful for those
who need to create a secure channel between Graphene SGX processes running
on different machines.
Using Redis 6.0.5 on more recent distros (Fedora 32) avoids linker
issues like the following ones:
/usr/bin/ld: server.o:sds.h:37: multiple definition of `SDS_NOINIT';
quicklist.o:sds.h:37: first defined here
Using the latest busybox 1.32.0 on more recent distros (Fedora 32)
avoids linker issues like the following ones:
/usr/bin/ld: util-linux/lib.a(rdate.o): in function `rdate_main':
rdate.c:(.text.rdate_main+0x150): undefined reference to `stime'
/usr/bin/ld: coreutils/lib.a(date.o): in function `date_main':
date.c:(.text.date_main+0x2a4): undefined reference to `stime'
collect2: error: ld returned 1 exit status
Previously, RA-TLS was recommended to be used with LD_PRELOAD trick.
However, LD_PRELOAD is too hacky and shouldn't be used to preload
libraries to an executable (in contrast to its normal use of
replacing functions from one library with another). This commit
removes any mentions of LD_PRELOAD trick from RA-TLS, and
replaces LD_PRELOAD with `dlopen()` in ra-tls-mbedtls example.
Secret Provisioning provides a simple interface for enclavized
applications to connect to a trusted secret-provisioning service and
securely pass the secrets inside the SGX enclave. The application acts
as a client and the secret-provisioning service as a server. They
establish a TLS session via mutual authentication, where the service
sends a classical X.509 certificate and the application sends an
RA-TLS X.509 certificate. After the service verifies the RA-TLS
certificate and its SGX-related information, it provisions secrets
to the application. The TLS session may continue to be used by the
application and the server after the initial provisioning. Secret
Provisioning feature relies heavily on RA-TLS.
Secret Provisioning is shipped as three libraries:
- secret_prov_attest.so: creates RA-TLS X.509 certificate with SGX
quote embedded and sends it to the secret-provisioning service as
part of mutual attestation of TLS session. Linked into enclavized
applications. Not thread-safe.
- secret_prov_verify_epid.so: establishes a TLS session with mutual
attestation, verifies received RA-TLS certificate, and sends secrets
back to the enclavized application. Linked into secret-provisioning
service. Verifies RA-TLS certificate using EPID-based flows.
Multi-threaded and thread-safe (new thread for each client).
- secret_prov_verify_dcap.so: same as secret_prov_verify_epid.so but
verifies RA-TLS certificate using ECDSA/DCAP-based flows.
Multi-threaded and thread-safe (new thread for each client).
This commit also adds two Secret Provisioning examples, with a minimal
enclavized app, a more complicated enclavized app, and a single
secret-provisioning service. These examples are added to Jenkins.
LMbench was causing a lot of problems in the past and we just noticed
another one (the last one from the list below), which broke the
camel's back :)
Some of the issues:
- Original mirrors are unstable.
- Its source is terrible.
- It's not portable (e.g. needs custom patches to run on Fedora).
- Runs for quite long in CI (a few minutes).
- The last update to the project was 15 years ago (13 on dev branch).
- It uses some features not implemented in Graphene
(getrusage(RUSAGE_SELF, ...)) without checking for errors, which leads
to reporting bogus times.
- Actually, it silently skips failures, because the command is piped to
`tee` and then `tee`'s exit code is checked instead of the original
command's.
Michał Kowalczyk