This release fixes server to only create deltas between two regular files,
removing errors from attempting to make deltas to/from/between symlinks.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
In the event that a symlink target changes between two versions, and the
files the symlink points to also change and would create a delta, do not
create a delta between the symlinks. The real delta is between the
underlying files the symlinks point to, which is created separately.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
For the F->L and L->F type changes, make sure that there are no delta
files created between the dereferenced symlink on one end and the
regular file on the other.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
For three different Clear Linux OS builds in the last few months, deltas
were created between files with type change L->F (symlink to file).
This was allowed to occur because there is no check if
file->peer->is_link in __create_delta().
Instead, remove the file->is_link check and simply ensure that the
from/to file types are both F (i.e. "regular file").
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To simplify the discovery of C code style issues and enforce the rules
specified in .clang-format, I've added a new 'compliant' target for
running the appropriate clang-format command.
In case code style issues are found, source files are modified in place,
and the resulting diff can be viewed. The exit code in this case will be
1, so make will exit with an error. This helps to automate testing for
code style issues.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This release includes changes across various areas of server, specifically:
- Fixing fallthrough detection logic to detect files and directories correctly
when checking if they are "state" files.
- Removing stale and unused signature creation code
- Adding functional tests and enabling travis-ci integration
- Create alternative input layout to save IO for some cases(backward compatible)
- Update parallelism code to make it more versatile and editable
- Fix log call and add logging to stdout instead of just logfiles
- Honor proxy and cert checking settings
- Fix extracting files with bsdtar
- Enable locales in all programs
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
In Ostro OS, we already have a "full" directory with all files.
Splitting it up into bundles just so that swupd-create-update can
reconstruct the "full" directory is a waste of IO, and noticably slow
when run under pseudo.
To streamline the required work, a new layout for the "image" input
directory gets introduced:
- The "full" directory gets created by the caller before invoking
swupd-create-update.
- For each bundle, instead of a <bundle> directory, there is a
<bundle>.content.txt file, listing all entries (including directories)
of the bundle.
The traditional mode of operation still works as before because each operation
which normally works with a bundle directory checks whether there is such a
directory and if not, switches to the new mode.
That way it is even possible to mix the two modes, i.e. replacing only
some bundles with a content list, although that's probably not all
that useful.
This revised commit fixes the use of an uninitialized newversiondircontent
pointer in populate_dirs().
Fixes: swupd-server/#54
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
This is a pre-condition for using libarchive directly: libarchive
needs to know what the encoding of filenames is, and it uses the
current locale for that. Without setlocale(), the locale is "C", which
only supports ASCII filenames, leading to warnings about "Can't
encode..." from libarchive when it is forced to fall back to copying
strings verbatim when writing archives that require UTF-8 encoding.
As a side effect, error messages from libc will get translated
according to the user's environment.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
TAR_XATTR_ARGS is no longer used as part of a plain string. Embedding
the empty "" value for bsdtar inside an argv argument list passes an
empty parameter to bsdtar, leading to:
bsdtar: Must specify one of -c, -r, -t, -u, -x
To allow the the "no parameter" case, it has to be argument list: that
can be empty. If not empty, it has to end with a comma.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
In case a proxy is used for downloading the manifest/pack, make sure to
honor those settings. And also, make sure certificate verification
occurs for the downloads.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This reverts commit f01d9ca6c8.
Upon further testing, this patch causes a double free/corruption with the
current master branch and crashes two of the tests. We need to investigate
more before fully enabling it to ensure we don't regress.
When a CI system (like the one from Ostro) captures the output of
commands, but not necessarily intermediate log files, then it is
useful to also log to stdout. Another use case is calling the tools
interactively during development.
The new --log-stdout option in all three commands enables logging to
stdout in addition to the traditional log files.
The implementation recycles the existing init_log_stdout() (not used
before) and gives it the slightly different meaning of "also log to
stdout".
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The SWUPD_NUM_THREADS env variable is now understood by all three
commands and overrides the default number of threads. Setting it to 1
is useful while debugging the code that runs inside threads (only one
thread hits breakpoints there). If SWUPD_NUM_THREADS is invalid, a
warning is printed and the variable gets ignored, i.e. the default
parallelism is used.
The hard-coded parallelism of 12 threads when analysing the file system
gets replaced with n, where n is the number of available CPUs. The default
is the same as before elsewhere (n for packing, 3 * n for fullfiles).
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
In Ostro OS, we already have a "full" directory with all files.
Splitting it up into bundles just so that swupd-create-update can
reconstruct the "full" directory is a waste of IO, and noticably slow
when run under pseudo.
To streamline the required work, a new layout for the "image" input
directory gets introduced:
- The "full" directory gets created by the caller before invoking
swupd-create-update.
- For each bundle, instead of a <bundle> directory, there is a
<bundle>.content.txt file, listing all entries (including directories)
of the bundle.
The traditional mode of operation still works as before because each operation
which normally works with a bundle directory checks whether there is such a
directory and if not, switches to the new mode.
That way it is even possible to mix the two modes, i.e. replacing only
some bundles with a content list, although that's probably not all
that useful.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Right now, the travis-ci config simply installs required build
dependencies for swupd-server and works around the umask discrepancy to
proper run the functional test suite.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Three functional tests depend on the effective UID being non-zero
(non-root), so skip the tests if running as root.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This conditional checks for state *directories* that are generally
installed by default, and the conditional immediately below this one
checks for state files within these directories. So, if we do strncmp()
instead of strcmp(), the fallthrough logic doesn't occur, and state
files are not marked as such.
This reverts commit 63fb5fb61b.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Since the enablement of signature verification in swupd-client, the
signature creation step has been decoupled from swupd-server, and is
instead performed as a separate step in a DevOps flow. As a result of
this decoupling, the signature code in swupd-server has remained unused.
This commit removes all the signature creation code with the assumption
that the separate DevOps step is going to work better long-term. Also,
the existing signature creation support does not accord with
swupd-client's verification support.
An example of how Manifest.MoM files can be signed is found in the
https://github.com/clearlinux/mixer-tools repo.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This release contains various changes to fix segfaults and memory misuse,
simplifying logic when iterating directories and populating file structs
from manifests, and changing the secondary sort to be lexographically sorted
filenames for version-sorted manifests.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release contains various changes to fix segfaults and memory misuse,
simplifying logic when iterating directories and populating file structs
from manifests, and changing the secondary sort to be lexographically sorted
filenames for version-sorted manifests.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
To ease human readability of manifests, but without impacting manifest
delta efficiency, use a lexicographic filename secondary sort order when
sorting manifests by version.
Below is an example of how this commit changes the sorted order (the
first column is the version, and the second column is the filename).
# Before
10 zyxw
10 abcd
20 test2
20 test1
20 abba
20 aaaa
30 zzyy
# After
10 abcd
10 zyxw
20 aaaa
20 abba
20 test1
20 test2
30 zzyy
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The file struct must be populated to contain the proper stats, so just
check if it is a directory and iterate, ignoring the case where d_type may
not be defined on the system.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
According to POSIX.1 only d_name and d_ino fields of struct
dirent are standardized. d_type isn't always correctly set on file
systems like XFS. In such cases it makes sense to resort to
lstat(). Otherwise a user has hard time figuring out what's
wrong with her setup.
Also remove redundant populate_file_struct() as it's called
again in parallel threads.
Signed-off-by: Dmitry Rozhkov <dmitry.rozhkov@linux.intel.com>
If the os-core bundle is not listed in groups.ini we will run into
problems later on, including segfaults when processing bundle includes.
Check the os-core bundle is listed in the groups.ini during initialisation
and error gracefully when it is not.
Signed-off-by: Joshua Lock <joshua.g.lock@intel.com>
In the (unlikely) case that nothing changed between two builds,
get_deduplicated_fullfile_list() segfaults because it uses
manifest->files without checking for NULL, aka the empty list.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
At the moment, swupd_create_pack fails when some files have xattrs and
get patched because the xattrs of the test file do not match the
original, unpatched file.
That's because xattrs_copy() was applied to the wrong target file.
Fixes: swupd-server/#35
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The bundle-chroot-builder changed the location where it stores bundle
includes metadata, so swupd-server needs to read from the new location.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This patch extracts the [STATUS] field from our bundle information
metadata, and stores it in groups.ini. From there we put the contents
of this string verbatim into the manifest.
We don't interpret, encode or convert the contents of the [STATUS]
field in the manifest. Instead, we just strip non-alphanumeric
characters and pass the contents on. This leaves it entirely to the
client to parse and interpret the value of this field in the manifest.
If the bundle file, or the groups.ini file omits any status, nothing
is output to the Manifest file.
This release introduces a change to the hashing scheme regarding delta names,
and directory hashes, as well as minor fixes to support the new hashing.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
There is an issue when two different files in a newer release have same hash:
This is, when swupd updates a file by applying a delta, it takes the HASH2 to
know the file where delta must be applyed. If files are different in current
release (before updating), there must be 2 deltas, one for each file but as
the two files have same hash in new release delta's name are the same:
FROM-TO-HASH2 and it is when issue arises due to just the last delta file is
kept when swupd server creates files and packs. So when swupd client tries to
apply the delta to one of the file that does not corresponds it will fail and
generates an error. At the first look it will seem like delta file is corrupted
however the issue is that delta file was created for another file.
To solve this issue we include the hash for the original file in the delta's
name so that swupd client can take the correct delta and apply it:
FROM-TO-HASH1-HASH2.
Signed-off-by: Jose R Guzman <jose.r.guzman.mosqueda@intel.com>
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Calculate the hash for directories is desiderable to be independent
on the dirname due to the subsequent calculation on the staged/HASH
file. Here is used const "DIRECTORY" string for input name for
all folders.
Signed-off-by: Jose R Guzman <jose.r.guzman.mosqueda@intel.com>
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release includes a fix to replace system calls by local system_argv
functions, cleanup patches for error output and code style, adding delta
manifests to delta packs, consistency changes for tar and string allocation,
and removes creation of delta manifests out of create_update.
This is a re-roll of release 3.2.4 to fix a time skew issue that caused some
build failures.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
This release includes a fix to replace system calls by local system_argv
functions, cleanup patches for error output and code style, adding delta
manifests to delta packs, consistency changes for tar and string allocation,
and removes creation of delta manifests out of create_update.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Tudor Marcu