Perform a final link on all renames in the manifest in order to track
renames from older versions. Do not remove orphaned renames but instead
keep them around as deleted files as well.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Ghost files that will be deleted by third-party software on the
client-side instead of marking them as deleted in the manifest. This
allows the client to treat these as deleted when doing rename detection.
Remove those ghosted files from the manifest in the next update.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When running many instances of swupd_make_fullfiles in parallel, the
read end of the pipe between the tar's for creating the full file of a
directory becomes reused. The observed behavior of this is
swupd_make_fullfiles hangs indefinitely with the expected tar reader
missing. The corresponding tar writer is not killed with SIGPIPE because
there is at least one reader still for the pipe, swupd_make_fullfiles.
First forking from swupd_make_fullfiles, creating the pipe, and then
fork-and-exec'ing for each tar ensures that pipe and file descriptor
management is contained for the directory rename in question and cannot
be reused by other directory renames.
Signed-off-by: George T Kramer <george.t.kramer@intel.com>
Fixes#71
Prune debuginfo from manifests when the [Debuginfo][pruned]
configuration is set to "true" in server.ini.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When a format bump occurs and the new format is greater than the old
format, an actions field is written to the Manifest.MoM containing the
string "update". This "update" action tells the client that it is
necessary to re-execute swupd update to bring the client to the latest
version within the new format.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
TAR_XATTR_ARGS is no longer used as part of a plain string. Embedding
the empty "" value for bsdtar inside an argv argument list passes an
empty parameter to bsdtar, leading to:
bsdtar: Must specify one of -c, -r, -t, -u, -x
To allow the the "no parameter" case, it has to be argument list: that
can be empty. If not empty, it has to end with a comma.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The SWUPD_NUM_THREADS env variable is now understood by all three
commands and overrides the default number of threads. Setting it to 1
is useful while debugging the code that runs inside threads (only one
thread hits breakpoints there). If SWUPD_NUM_THREADS is invalid, a
warning is printed and the variable gets ignored, i.e. the default
parallelism is used.
The hard-coded parallelism of 12 threads when analysing the file system
gets replaced with n, where n is the number of available CPUs. The default
is the same as before elsewhere (n for packing, 3 * n for fullfiles).
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Since the enablement of signature verification in swupd-client, the
signature creation step has been decoupled from swupd-server, and is
instead performed as a separate step in a DevOps flow. As a result of
this decoupling, the signature code in swupd-server has remained unused.
This commit removes all the signature creation code with the assumption
that the separate DevOps step is going to work better long-term. Also,
the existing signature creation support does not accord with
swupd-client's verification support.
An example of how Manifest.MoM files can be signed is found in the
https://github.com/clearlinux/mixer-tools repo.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This patch extracts the [STATUS] field from our bundle information
metadata, and stores it in groups.ini. From there we put the contents
of this string verbatim into the manifest.
We don't interpret, encode or convert the contents of the [STATUS]
field in the manifest. Instead, we just strip non-alphanumeric
characters and pass the contents on. This leaves it entirely to the
client to parse and interpret the value of this field in the manifest.
If the bundle file, or the groups.ini file omits any status, nothing
is output to the Manifest file.
There is an issue when two different files in a newer release have same hash:
This is, when swupd updates a file by applying a delta, it takes the HASH2 to
know the file where delta must be applyed. If files are different in current
release (before updating), there must be 2 deltas, one for each file but as
the two files have same hash in new release delta's name are the same:
FROM-TO-HASH2 and it is when issue arises due to just the last delta file is
kept when swupd server creates files and packs. So when swupd client tries to
apply the delta to one of the file that does not corresponds it will fail and
generates an error. At the first look it will seem like delta file is corrupted
however the issue is that delta file was created for another file.
To solve this issue we include the hash for the original file in the delta's
name so that swupd client can take the correct delta and apply it:
FROM-TO-HASH1-HASH2.
Signed-off-by: Jose R Guzman <jose.r.guzman.mosqueda@intel.com>
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Calculate the hash for directories is desiderable to be independent
on the dirname due to the subsequent calculation on the staged/HASH
file. Here is used const "DIRECTORY" string for input name for
all folders.
Signed-off-by: Jose R Guzman <jose.r.guzman.mosqueda@intel.com>
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
There are new functions that replace system() calls: system_argv(), system_argv_fd and system_argv_pipe().
They are based on execvp() function and they can manage any argument including filenames with characters
that could be interpreted as meta-characters in a regular system() function.
So no escape for the arguments needed and is less prone to errors.
Signed-off-by: Jose R Guzman <jose.r.guzman.mosqueda@intel.com>
Since removing of all of the matching deleted files between a build with
an older manifest format and current needs to happen *after*
match_manifests() finishes accounting for new and deleted entries in the
manifests, simply move the logic into a separate function to be called
after match_manifests().
Because one or more file deletions should trigger a fresh bundle manifest
creation, account for the return value of this new function in addition
to the result of manifest pruning and whether any bundle includes were
changed. Since "os-core" is treated specially, and it's expected to
contain changed files for every build, only the non-"os-core" bundle
manifests need the trigger.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To decrease chances of specifying an incorrect format number, since the
default is a hardcoded value "3", always require the user to pass the -F
option to swupd_create_update.
In the future, it would be nice to read the default value from a config
file, with the -F option overriding the value.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To better support a workflow for incrementing the format on the command
line, it's useful to have the format be an integer instead of a string.
When reading a manifest (ie: old manifest) we can read that into struct
manifest. Then when building a new manifest we can compare against the
prior manifest's format. If the format incremented, we can prune files
previously marked as deleted in the old manifest so they no longer
appear in the new manifest.
This is beneficial because a file that has a version number in its name
will repeatedly be deleted and replaced by a new similar file. Over
time this leads to manifests getting bigger and bigger because they
retain the entire history of all file names which ever existed, even if
a huge number of them are and have been for a long time marked as
deleted.
Signed-off-by: Tim Pepper <timothy.c.pepper@linux.intel.com>
When deciding if a new manifest should be generated for a new version,
the included manifests were not being used to detect a manifest
change. This caused manifests that had new includes: lines in the new
version to not be generated resulting in files from the included
manifest to be missed when adding/updating that bundle.
The fix was to compare includes: lines from the current and previous
manifest versions for differences and generate new manifests when that
was the case. In order to compare manifests duplicate includes: lines
are not allowed and previously os-core could be added multiple times to
a manifest because it was always added. This change also added detection
for os-core before automatically adding it to a bundle.
When subtracting a manifest from another, all the included manifests
also will need to be subtracted from the original. To do this build up a
unique list of all manifests included (directly or indirectly) from the
manifest being subtracted from.
With that in place, this gives an opportunity to provide the
functionality of subtracting all included manifests from the main
manifest by calling subtract_manifests(A, A);
To capture all information that is logged, change init_log() to make
every log file name unique, and add a call to swupd_make_fullfiles to
enable logging.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Since GNU tar fails to extract files with xattrs preserved when
Integrity Measurement Architecture (IMA) is enabled some vendors
may choose to install libarchive-based tar (bsdtar) on their embedded
devices, so the swupd server needs to be able to create archives
in its format.
This patch adds one compile-time options --enable-bsdtar that is used
to enable/disable GNU tar specific options. Also it harmonizes
the command strings to be compatible with both GNU tar and bsdtar.
Particularly it
- changes --exclude pattern from '%s'/* to more explicit '%s/?*' because
bsdtar's pattern matching is greedier than in tar: it uses tcsh's
globbing where '*' can be anything including the null string and the
original pattern would include the directory itself;
- OS file names are escaped with leading ./ to avoid collisions with
file names starting with @ which has special meaning in bsdtar.
Signed-off-by: Dmitry Rozhkov <dmitry.rozhkov@linux.intel.com>
Matthew Johnson