Change log (most important only):
- unify CPU context structures - now we have only one version -
`PAL_CONTEXT` - which is shared between LibOS and PALs and it should
depend only on the host architecture (not OS),
- syscalls emulation changed:
- dedicated LibOS stack is now used for syscalls emulation,
- removed one indirection level in syscalls table - now it stores
`shim_do_*` functions directly,
- signal handling - completely rewritten:
- all signal queues use proper locking schemes now,
- signals are handled *only* when returning to the user app from LibOS
or PAL,
- nested signals are now possible,
- the app is allowed to jump out of signal handler with the same
sematics as on normal Linux,
- signal altstack is now fully supported,
- syscall restarting is now supported,
- doing a backtrace from the signal handler works properly,
- disallow injecting host-level signals, with one exception, see
`sys.enable_sigterm_injection` manifest option for more details.
This commit also fixes `pseudo_name_ops::list_name()` function pointer
type: `size_t len` argument instead of `int len`. It also adds a
regression test to exercise the newly supported /sys pseudo filesystem.
Instead of 'loader.debug_type', introduce 'loader.log_level'
and 'loader.log_file', along with a set of definitions for
logging at a chosen level.
For now, the call sites keep using the legacy macros (SGX_DBG and
debug()), because converting them all will conflict with other
big changes in the code base. The existing LibOS calls are
assumed to be at 'info' level.
- Use the same mechanism (debug_map) in Pal/Linux and Pal/Linux-SGX.
Previously, Pal/Linux emulated the _r_debug structure, normally
maintained by ld.so, but that cannot be done in SGX outer PAL,
because it's loaded by ld.so already.
- Maintain the debug maps outside of SGX enclave. This allows
initializing them before enclave start, and potentially makes
them easier to use.
- Initialize PAL debug map before enclave start. Previously, this
was done from inside the enclave, so you couldn't set a
breakpoint too early (e.g. in pal_linux_main).
- Store only load address, without list of sections. This is to
avoid parsing the list of sections just to report them to the
debugger. Unfortunately, the GDB version that we support still
needs these sections, but we can retrieve them in GDB plugin.
- Move Python GDB code related to debug maps to a common file.
This commit additionally replaces all `while (true) {}` inf loops with
`die_or_inf_loop` which either crashes the process or loops infinitely
and is not an undefined behavior like the original one (C disallows inf
loops without side effects).
This is the next part of the great loader rework, with a lot of breaking changes:
- Complete removal of the "trusted children" thing - now children
processes can be spawned arbitrarily and from arbitrary mountpoint
types, without any additional configuration needed.
- There's a new, required option in the manifest: `libos.entrypoint` - it
specifies the URI to the entry binary in the first process. There's no
need anymore to name the manifest and the first binary identically.
- On SGX, the main binary is not measured in MRENCLAVE anymore - only
PAL, LibOS and the manifest are measured. This is enough to bind
MRENCLAVE to a specific entrypoint user executable if wanted - it
just has to be mounted as a trusted file.
- All Graphene SGX enclaves have now exactly the same MRENCLAVE. This is
a hash of a "Graphene stub", which can "fork" into one of two states
in runtime: initial process or child. The initial process creates a
new "Graphene namespace" with a clean state, it can also be attested
remotely (contrary to child processes). The initial process can spawn
children processes by spawning a Graphene stub and directing it to
start in the child mode. It then attests it locally, and if
successful, establishes an encrypted pipe, "connects" to its own
namespace and treats as trusted (including sending protected files
key).
- Now, there's only one, central manifest describing the initial state
of a Graphene instance which can be spawned from it (previously, each
process required a separate manifest which could have different
configuration - which wasn't actually supported and didn't make sense
design-wise). One downside of central manifests is that all processes
require the same enclave configuration (e.g. size), but that was
already the case so far because of broken checkpointing code. Also,
this is only a temporary problem, which will cease to exist after the
introduction of EDMM.
- `sgx.static_address` was renamed to `sgx.nonpie_binary` and now has to
be inserted manually by users (`sgx_sign` tools doesn't know about the
binaries run inside, which can be even provided or generated in
runtime by the user's workload).
- Caveat: the memory gap for non-PIE executables was removed because it
requires adding a new option to the manifest to be cleanly
implemented. This is left for some future loader rework PR.
GCC (and other compilers, e.g. Clang) provide a stack protector
feature to detect stack corruptions. This is achieved by storing
a 64-bit canary value on the stack frame on function entry and
verifying this value on function exit. Previously, Graphene disabled
stack protector completely. This commit enables it in LibOS and PAL
code (only if `-mstack-protector` feature is supported by compiler).
The stack protector uses a random per-thread canary stored in the
TLS/TCB of each thread. Each PAL implementation must follow the
rule that TLS/TCB is accessed via the GS register and that the offset
of canary in TLS/TCB is 0x8. Since LibOS re-uses TLS/TCB of the PAL,
there is no need for additional enabling at the LibOS layer.
Since `-mstack-protector` feature is architecture-specific, it is
currently enabled only for x86-64 (and above rules on using gs:[0x8]
to access the canary apply only to x86-64).
Co-authored-by: Isaku Yamahata <isaku.yamahata@gmail.com>
Previously, if futex wait returned -EINTR it was returned from
`_DkEventWaitTimeout` too. Now if the waiting condition no longer holds
we ignore EINTR and treat that as a successful wait.
This commit also removes redundant, copy-pasted code (_DkEventWait) and
unused Pal API function (DkEventWait).
This function was empty and it is responsibility of the caller (Pal
level) of specific exception handling function (LibOS level) to return
from the exception.
Untrusted Linux-SGX PAL handles host-level asynchronous signals by
emulating the interrupt (-EINTR) of the pending OCALL. Unfortunately,
there was a type cast issue such that int32_t -EINTR (`-4`) was
casted to a positive uint64_t and then OCALL consumed this positive
number instead of erroring out on -EINTR. This commit adds explicit
type casting to fix this bug.
Previously, pal_linux_defs.h was located in a common include dir,
though it was only used by Linux PAL (Linux-SGX PAL has its own
and very different version of this header). This commit moves this
header under Pal/Linux and performs a cleanup of its macros without
logic changes. A couple other tiny cleanups is done as part of this
commit: moving x86-64 non-SGX instruction wrappers into "cpu.h",
renaming USER_ADDRESS_LOWEST to DEFAULT_HEAP_MIN for consistency with
Linux-SGX, removing redundant _DkSystemTimeQueryEarly(), increasing
THREAD_STACK_SIZE from 8KB to 64KB (to be on the safe side).
The profiling system instruments the asynchronous enclave exit
(AEX). Depending on configuration, we either snapshot the IP
value, or dump registers and (portion of) stack. The 'perf report'
tool can be used to generate a report from the samples.
These functions (despite documentation comment saying exactly the
opposite) accepted only positive error values, but most of the callsites
passed negative error codes instead. This resulted in Graphene printing
"Unknown error" instead of a proper error details.
This is a major refactor of the way manifests are loaded and handled,
which will be followed by a complete rework of the loader code (which
will include e.g. centralized config).
Changes/fixes:
- Huge part of manifest handling was refactored and untangled.
- Starting without a manifest is now disallowed. This was actually
accidentally broken for some time and no one complained. It also makes
little sense in practice and in Graphene's overall design, e.g. it
conflicts with protected argv.
- Now we only allow starting by giving the executable, not manifest (the
magic resolution logic was removed).
- Now manifests are sent over pipes between parent and children, instead
of children finding and loading them on their own. This is a
preparation for the upcoming centralized manifests change.
- Previously manifests were parsed 2 times on Linux and 3 times on
Linux-SGX (by untrusted PAL, trusted PAL and LibOS). This is now
fixed.
- The common `pal_main()` now requires that the backend-specific PAL
loader loads the manifest before calling it. SGX code already has to
do it (for proper initialization), so let's unify this interface for
all PALs.
- Fix for a PAL crash when manifest size was divisible by page size
(sic!). NULL termination was missing, but most of the time the padding
to page size saved Graphene from crashing.
* Make sure "stat.h" and "perm.h" are directly included where
necessary.
* Don't include "perm.h" inside "stat.h" but require it to be
included separately.
* Remove workarounds with __KERNEL__, __GLIBC__, defining pid_t
directly, and reversed include order (system headers before local
ones).
Instead of using S_I* flags, or hardcoded octal literals, use
helpers such as PERM_rwxrwxr_x. These are proposed in a Linux patch
by Ingo Molnar: https://lwn.net/Articles/696231/
Logging to file was broken, because the PAL file write operation
required the user to provide an absolute offset, and LibOS always
provided an offset of 0. This worked when logging to stdout, but
in case of a regular file, it kept overwriting the beginning of
file.
To fix that, we introduce a a special DkDebugLog call. This is a
better solution than tracking the file offset manually, because
the offset would need to be synchronized across different threads
and processes, and debug logs should be as simple as possible. At
the same time, we don't want PAL to provide a generic "append to
a file" mechanism, because it makes I/O less deterministic.
The manifest syntax stays exactly the same, including 0 and 1
integers to denote boolean values (this is done for ease of porting
and can be fixed in future commits). The only visible change is
surrounding strings in the manifest with quotes (requirement of
TOML). All manifests and Makefiles of our tests and example apps are
ported to the new TOML syntax. Documentation is updated.
Applications tend to use `/proc/cpuinfo` to get the `cpu cores`
and `physical id` for computing number of physical cores in a
socket. Currently `cpu cores` field is incorrectly implemented as
it is set to number of logical processors online and `physical id`
isn't implemented. This patch addresses both of these issues.
Previously, process communication (channel between parent and newly
created child) was protected via TLS only during send/receive of the
checkpoint; after that the channel was downgraded from TLS to
plaintext. The reason for this downgrade is historical (IPC was
complicated, and we wanted to have at least some TLS at the time).
This commit fixes this issue: Graphene now always uses TLS on IPC.
Slab memory manager logic uses enlarge_slab_mgr() (renamed to more
accurate maybe_enlarge_slab_mgr()) to allocate a new chunk of memory
for its slab objects. For allocation itself, slab manager must release
the lock before system malloc and then re-acquire it. At this point
other threads may allocate memory for the same slab level. Also,
system malloc may fail to allocate large memory region, and slab logic
will fall back to allocating smaller memory region. These two issues
were not properly handled; this commit fixes these bugs.
This library will be used to parse Graphene manifest files written
in TOML syntax. We patch the library slightly to remove unsupported
toml_parse_file() and toml_rtod(), as well as errno() handling.
Also move parse_size_str() to a more appropriate atoi.c file.
This is in preparation for replacing the old ad-hoc manifest
syntax with the TOML syntax.
I don't know any reason why would stating the file name we're in be
helpful for anything. Moreover, this information was incorrect in a few
cases (copy-paste bugs, probably).
Additionally, a few minor type/formatting fixes included.
Previously, Graphene preallocated 64MB for PAL internal metadata
like trusted/protected files metadata, handles metadata, etc.
If this limit was depleted, Graphene loudly failed, and the user
had no option but to change constant in source code and rebuild
Graphene. This commit adds the manifest option
`loader.pal_internal_mem_size` to allow increasing this limit.
The (((void *) &(arg)) == ((void *) (arg))) trick does not work
with clang, because clang does not accept it as a constant
expression and doesn't allow using it in static_assert.
I was not able to write a generic (any type) IS_ARRAY macro
compatible with static_assert, but here we only need to check for
arrays of chars and uint8_t-s.
In particular, this commit:
- Removes SLAB_DEBUG macros and corresponding code.
- Fixes memory leak in memmgr's enlarge_mem_mgr() by removing
__set_free_mem_area() call.
- Fixes bug of double-free of the very first memmgr area in
destroy_mem_mgr().
- De-duplicates "get new memory object" code by changing
get_mem_obj_from_mgr() to call get_mem_obj_from_mgr_enlarge().
- Simplifies and improves performance of free_mem_obj_to_mgr() since
there is no need to double-check that the object belongs to one of
the memmgr's areas because we already check memory_migrated().
- Fixes bug of free of wrong object in slabmgr's destroy_slab_mgr().
Previously, we introduced `sgx.zero_heap_on_demand` in Linux-SGX as a
knob to trade off runtime degradation on memory allocations for faster
enclave start-up times. This was an incorrect fix because Linux-SGX's
`_DkVirtualMemoryAlloc()` always zeroess the requested memory region,
so there was a double-zero of the heap at runtime. Note that LibOS
layer silently assumes that `_DkVirtualMemoryAlloc()` zeroes out the
memory, and many applications rely on this (Apache, Blender in my
experiments). Thus, this commit keeps the zero-out in
`_DkVirtualMemoryAlloc()` and removes zero-outs on enclave init and in
`get_enclave_pages()`. This renders `sgx.zero_heap_on_demand`
useless, so this manifest option is also removed. Also note that this
commit doesn't introduce any performance degradation (in fact, now
Graphene behaves as if `sgx.zero_heap_on_demand = 1` always).
Borys Popławski