After this change, LibOS will no longer perform dynamic linking.
The ELF loading code executes load commands and passes control to
interpreter (ld.so), which handles necessary relocations and
loading additional libraries. In this way, the code resembles what
Linux kernel does when executing a new program.
Before, dynamic linking was necessary for making LibOS entry point
(syscalldb function) available for applications. However, that meant
duplicating the work already done by ld.so, and introduced a lot of
unnecessary complexity. After changing LibOS entry API to use the
GS register, it's possible to omit dynamic linking entirely.
The main function (__load_elf_object()) still needs cleanup and
possibly rewriting from scratch. However, this change prepares
ground for that rewrite.
Summary of changes:
- Remove dynamic relocation step (DO_DYNAMIC_RELOCATE()).
- Don't call __load_elf_object() again for ELFs reported
via register_library(). We only need to notify GDB about these.
- Remove fields related to dynamic linking from link_map (dynamic
section address, hashes, etc.), and setup for these fields.
- need_interp(): To check if we need an interpreter, check only if
the binary requests one (PT_INTERP), instead of traversing the
dynamic section (before, we ignored dynamic dependencies on LibOS
itself, but now there shouldn't be any).
- RELOCATE(): always adjust addresses, instead of checking if
they're already inside the mapped range. I think the previous
behaviour was a workaround to make repeated relocations work.
- Get rid of load modes that are no longer used (OBJECT_REMAP,
OBJECT_USER).
- Remove the workaround for repeated relocation in glibc patches
(R_X86_64_NONE).
Signed-off-by: Paweł Marczewski <pawel@invisiblethingslab.com>
Instead of depending on dynamic linking for LibOS entry point
(syscalldb), we pass a pointer in the shim_tcb structure, so that
the patched code can enter syscall using 'jmp *%gs:<offset>'.
The same applies to the vDSO syscall code that previously needed
an up-to-date pointer to syscalldb function. Now, there is no
need to adjust the values inside the vDSO page.
In addition, this change removes the other two instances where we
import a symbol directly from LibOS: register_library (can be also
done through GS register) and glibc_version (not important because
we build Graphene and glibc together).
This simplifies things because the dynamic linking necessary to
make the syscalldb function available had to be performed by LibOS
itself (in many cases, effectively doing a second pass of dynamic
linking after ld.so). After this change, there will be no need for
LibOS to perform dynamic linking, and the ELF loading code can be
simplified.
Signed-off-by: Paweł Marczewski <pawel@invisiblethingslab.com>
Change log (most important only):
- unify CPU context structures - now we have only one version -
`PAL_CONTEXT` - which is shared between LibOS and PALs and it should
depend only on the host architecture (not OS),
- syscalls emulation changed:
- dedicated LibOS stack is now used for syscalls emulation,
- removed one indirection level in syscalls table - now it stores
`shim_do_*` functions directly,
- signal handling - completely rewritten:
- all signal queues use proper locking schemes now,
- signals are handled *only* when returning to the user app from LibOS
or PAL,
- nested signals are now possible,
- the app is allowed to jump out of signal handler with the same
sematics as on normal Linux,
- signal altstack is now fully supported,
- syscall restarting is now supported,
- doing a backtrace from the signal handler works properly,
- disallow injecting host-level signals, with one exception, see
`sys.enable_sigterm_injection` manifest option for more details.
This commit also fixes `pseudo_name_ops::list_name()` function pointer
type: `size_t len` argument instead of `int len`. It also adds a
regression test to exercise the newly supported /sys pseudo filesystem.
The msg field of 'struct shim_ipc_msg' will for example be cast to
'struct shim_ipc_sysv_tellkey*' (in ipc_sysv_tellkey_callback()) and
needs to be properly aligned. There are also casts to other IPC-related
structures that also require alignment.
Instead of 'loader.debug_type', introduce 'loader.log_level'
and 'loader.log_file', along with a set of definitions for
logging at a chosen level.
For now, the call sites keep using the legacy macros (SGX_DBG and
debug()), because converting them all will conflict with other
big changes in the code base. The existing LibOS calls are
assumed to be at 'info' level.
This commit additionally replaces all `while (true) {}` inf loops with
`die_or_inf_loop` which either crashes the process or loops infinitely
and is not an undefined behavior like the original one (C disallows inf
loops without side effects).
This commit removes types from syscall argument printing functions as
a preparation for next changes. Now the table with syscall parsers have
all parsing functions embeded directly.
Also some minor cleanups in the same area.
Commit "Introduce one, central manifest, zero-config children and
constant MRENCLAVE" removed execve-in-a-new-process, so execve specific
checkpointing functions are not needed anymore (pending_signals,
arguments, environ).
This is the next part of the great loader rework, with a lot of breaking changes:
- Complete removal of the "trusted children" thing - now children
processes can be spawned arbitrarily and from arbitrary mountpoint
types, without any additional configuration needed.
- There's a new, required option in the manifest: `libos.entrypoint` - it
specifies the URI to the entry binary in the first process. There's no
need anymore to name the manifest and the first binary identically.
- On SGX, the main binary is not measured in MRENCLAVE anymore - only
PAL, LibOS and the manifest are measured. This is enough to bind
MRENCLAVE to a specific entrypoint user executable if wanted - it
just has to be mounted as a trusted file.
- All Graphene SGX enclaves have now exactly the same MRENCLAVE. This is
a hash of a "Graphene stub", which can "fork" into one of two states
in runtime: initial process or child. The initial process creates a
new "Graphene namespace" with a clean state, it can also be attested
remotely (contrary to child processes). The initial process can spawn
children processes by spawning a Graphene stub and directing it to
start in the child mode. It then attests it locally, and if
successful, establishes an encrypted pipe, "connects" to its own
namespace and treats as trusted (including sending protected files
key).
- Now, there's only one, central manifest describing the initial state
of a Graphene instance which can be spawned from it (previously, each
process required a separate manifest which could have different
configuration - which wasn't actually supported and didn't make sense
design-wise). One downside of central manifests is that all processes
require the same enclave configuration (e.g. size), but that was
already the case so far because of broken checkpointing code. Also,
this is only a temporary problem, which will cease to exist after the
introduction of EDMM.
- `sgx.static_address` was renamed to `sgx.nonpie_binary` and now has to
be inserted manually by users (`sgx_sign` tools doesn't know about the
binaries run inside, which can be even provided or generated in
runtime by the user's workload).
- Caveat: the memory gap for non-PIE executables was removed because it
requires adding a new option to the manifest to be cleanly
implemented. This is left for some future loader rework PR.
Previously, Graphene saved and restored only the GP registers of the
CPU context on entering and leaving syscall emulation in syscalldb().
In reality, the CPU context must also contain FP control word (fpcw)
and the SSE/AVX/... control word (mxcsr). This commit preserves these
control words across app-to-Graphene context switches.
During syscall emulation, Graphene may clobber the FP/SSE/AVX/...
state (except the control words). We rely on the fact that apps do
*not* assume that this state is preserved across syscalls (except
the control words). Thus, it is enough to save/restore only the
control words on each syscall. This commit also removes previous
hack of performing expensive xsave/xrstor instructions on clone().
This is a major refactor of the way manifests are loaded and handled,
which will be followed by a complete rework of the loader code (which
will include e.g. centralized config).
Changes/fixes:
- Huge part of manifest handling was refactored and untangled.
- Starting without a manifest is now disallowed. This was actually
accidentally broken for some time and no one complained. It also makes
little sense in practice and in Graphene's overall design, e.g. it
conflicts with protected argv.
- Now we only allow starting by giving the executable, not manifest (the
magic resolution logic was removed).
- Now manifests are sent over pipes between parent and children, instead
of children finding and loading them on their own. This is a
preparation for the upcoming centralized manifests change.
- Previously manifests were parsed 2 times on Linux and 3 times on
Linux-SGX (by untrusted PAL, trusted PAL and LibOS). This is now
fixed.
- The common `pal_main()` now requires that the backend-specific PAL
loader loads the manifest before calling it. SGX code already has to
do it (for proper initialization), so let's unify this interface for
all PALs.
- Fix for a PAL crash when manifest size was divisible by page size
(sic!). NULL termination was missing, but most of the time the padding
to page size saved Graphene from crashing.
Previously Graphene never deleted dentries. This commit adds possible
dentry deletion on `unlink` and handle close. It also fixes some
dentry reference counting bugs.
Most important differences from the old version:
- strip global process information from the thread struct into a
dedicated one,
- a parent is informed about the child death when the whole process
(the last thread) dies (not on each thread exit),
- all threads have the same parent (spawning thread is NOT the parent of
the spawned thread),
- a thread is able to wait on children created by another thread,
- a process is able to wait for exited children after execve,
- rewritten `waitid` implementation (no more gotos, supports __WCLONE
and friends flags),
- added option for syscall restarting, for now used only in `waitid`.
Additionally various bugfixes, cleanups and missing locks added.
Sometimes we need to temporarily stop IPC helper thread from receiving
more messages, e.g. when doing execve just before migrating exited (but
not yet waited for) children list.
* Make sure "stat.h" and "perm.h" are directly included where
necessary.
* Don't include "perm.h" inside "stat.h" but require it to be
included separately.
* Remove workarounds with __KERNEL__, __GLIBC__, defining pid_t
directly, and reversed include order (system headers before local
ones).
Logging to file was broken, because the PAL file write operation
required the user to provide an absolute offset, and LibOS always
provided an offset of 0. This worked when logging to stdout, but
in case of a regular file, it kept overwriting the beginning of
file.
To fix that, we introduce a a special DkDebugLog call. This is a
better solution than tracking the file offset manually, because
the offset would need to be synchronized across different threads
and processes, and debug logs should be as simple as possible. At
the same time, we don't want PAL to provide a generic "append to
a file" mechanism, because it makes I/O less deterministic.
The manifest syntax stays exactly the same, including 0 and 1
integers to denote boolean values (this is done for ease of porting
and can be fixed in future commits). The only visible change is
surrounding strings in the manifest with quotes (requirement of
TOML). All manifests and Makefiles of our tests and example apps are
ported to the new TOML syntax. Documentation is updated.
The operation loops indefinitely on error. Instead, it should find
the first free FD, and then try allocating it.
In addition, the right error after exceeding the limit is EMFILE
(however, dup2() is still supposed to return EBADF if asking for
an out-of-range value, as checked by the dup201 LTP test).
They all lacked error checking and `wait_event` was completely broken:
it was reading from non-blocking pipe and treating EAGAIN as
successfully waited-for event.
Previously, IPC_PORT_SERVER meant "listening port", and the actual
communication ports had several types. Only two of these types were
used for differentiation during IPC broadcast (direct-child and
direct-parent types). All other types denoted who is the remote party
this port connects to, but this info is superfluous. So this commit
replaces all these types with a generic IPC_PORT_CONNECTION, and
renames IPC_PORT_SERVER to a more familiar IPC_PORT_LISTENING.
Previously, LibOS (shim) layer of Graphene didn't support XSAVE area.
The XSAVE area stores FP, XMM, YMM, ZMM, etc. registers and control
states and is handled via FXSAVE/XSAVE and FXRSTOR/XRSTOR x86-64
instructions. This commit is the first step towards adding full-
fledged XSAVE support to LibOS. It adds XSAVE related structs and
functions to LibOS code, and propagates XSAVE regs/states from
parent to child on thread creation via clone() (though not really
correctly). New test `fp_multithread` is added to LibOS regression.
Future commits will add correct XSAVE handling on syscall transitions
and arriving signals.
If a remote ipc port gets disconnected we assume that remote process
died unexpectedly and mark it as killed with SIGKILL, so it makes no
sense to also keep the exit code.
On Linux fork and vfork are just specific cases of clone. This commit
does small cleanup of clone code and deduplicates proces copying code by
always using clone.
Removed:
- `message_confirm` - not used anywhere (and probably won't ever be),
- `SYS_PRINTF` - this was just synonym of `debug` or `warn` (depending
on the context),
- all other functions that were used only by the two above.
Paweł Marczewski