The previous release (v3.6.1) was tagged at the wrong commit. This
release includes the changes described in that release note.
This release fixes a bug where swupd-server was creating the staged and
delta directories within the packs with 0750 permissions instead of the
expected 0700 permissions. It also adds a configuration option to
server.ini to ban debuginfo from the manifests and configure where
debuginfo libs and src are stored. Also prunes mistakenly added
debuginfo from the manifests when added accidentally and configured to
do so via server.ini.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Fixes#71
Prune debuginfo from manifests when the [Debuginfo][pruned]
configuration is set to "true" in server.ini.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release fixes a bug where swupd-server was creating the staged and
delta directories within the packs with 0750 permissions instead of the
expected 0700 permissions. It also adds a configuration option to
server.ini to ban debuginfo from the manifests and configure where
debuginfo libs and src are stored.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
These directories were previously created with 750 permissions, which
caused a new check in swupd-client to remove them in order to correct
the permissions to 700. Create them the right way in the first place.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Instead of only calculating the contentsize for files that were updated
in the current version (an inaccurate number for download size since
this is not the compressed size), calculate for all files in the
manifest. Additionally, do not add submanifest contentsizes to the
current manifest contentsize, as this will result in overcount on client
systems when multiple bundles include the same bundle.
With this change the contentsize field of the manifests will only report
the size of the files unique to that bundle. It is then the client's
responsibility to calculate total bundle size including included
bundles. This is reasonably easy to accomplish with the upcoming
swupd-client bundle-list --deps feature.
Although the above use case is a bit more work for the client, it
additionally allows a user to calculate the installation size of
multiple bundles much more easily, since it only has to count bundle
dependencies once to ensure files are not over counted.
If two bundles not in the same include chain have overlapping content,
summing the include chain of each bundle in the client will result in an
over-estimation of the total size on the system. The more content is
shared, the higher the over-estimation. In reality this overlap will not
be large, but it is currently impossible to calculate the exact
installed size using just the contentsize.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This release removes a stale, unused script, adds a check to ensure format
number validity to ensure format numbers do not decrease, and enables adding
an actions: field in the Manifest.MoM to tell swupd-client to perform certain
actions post update.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Adds a functional test to check for the
actions: update
line in the Manifest.MoM for the new format.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
When a format bump occurs and the new format is greater than the old
format, an actions field is written to the Manifest.MoM containing the
string "update". This "update" action tells the client that it is
necessary to re-execute swupd update to bring the client to the latest
version within the new format.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Because the intention is for swupd format numbers to either remain the
same between LAST_VER and current, or to increase as part of a format
bump, this leaves the remaining undesirable case.
Add a basic check to make sure the format never decreases, and add a
functional test.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This release adds rename detection support which can be enabled via the
--enable-rename-detection option during configure time.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
If the environment variable RUN_JUST_ONE is set then skip any test
which is not numerically equal to it.
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
There is no need for duplicate includes to exist in manifest headers, so
search the includes lists first before adding a new entry.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This release fixes subtracting files from manifests when both versions are marked deleted. Server subtracting of files when both are deleted
proves problematic for client updates, because the version at which the
files were deleted will differ, and the client uses the versions to
determine when it should delete files for an update.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Subtracting files from manifests when both versions are marked deleted
proves problematic for client updates, because the version at which the
files were deleted will differ, and the client uses the versions to
determine when it should delete files for an update.
As an example, consider a distro with 2 bundles, "A" and "B". Bundle B
includes bundle A, which results in files in A being subtracted from B.
In this situation the following sequence of four changes result in a
subtraction that prevents a client update from deleting a file.
1) In version 10, file /usr/foo is added to bundles A and B.
2) In version 20, /usr/foo is deleted from bundle A.
3) In version 30, /usr/foo is deleted from bundle B.
4) In version 40, bundle B is modified.
Due to arbitrary modifications to bundle B in step 4, /usr/foo is
subtracted from bundle B, because it's also deleted in bundle A.
However, the file versions mismatch. So, an update from version 20 to 40
will result in /usr/foo not being deleted because the deleted entry from
bundle A was not deleted in a version newer than 20.
This patch fixes the issue by ensuring that the deleted entry in bundle
B remains intact for version 30. And the client update then works
correctly: an update from 20 to 40 will properly delete /usr/foo,
because 20 < 30 <= 40.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This release fixes server to only create deltas between two regular files,
removing errors from attempting to make deltas to/from/between symlinks.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
In the event that a symlink target changes between two versions, and the
files the symlink points to also change and would create a delta, do not
create a delta between the symlinks. The real delta is between the
underlying files the symlinks point to, which is created separately.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
For the F->L and L->F type changes, make sure that there are no delta
files created between the dereferenced symlink on one end and the
regular file on the other.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
For three different Clear Linux OS builds in the last few months, deltas
were created between files with type change L->F (symlink to file).
This was allowed to occur because there is no check if
file->peer->is_link in __create_delta().
Instead, remove the file->is_link check and simply ensure that the
from/to file types are both F (i.e. "regular file").
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To simplify the discovery of C code style issues and enforce the rules
specified in .clang-format, I've added a new 'compliant' target for
running the appropriate clang-format command.
In case code style issues are found, source files are modified in place,
and the resulting diff can be viewed. The exit code in this case will be
1, so make will exit with an error. This helps to automate testing for
code style issues.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This release includes changes across various areas of server, specifically:
- Fixing fallthrough detection logic to detect files and directories correctly
when checking if they are "state" files.
- Removing stale and unused signature creation code
- Adding functional tests and enabling travis-ci integration
- Create alternative input layout to save IO for some cases(backward compatible)
- Update parallelism code to make it more versatile and editable
- Fix log call and add logging to stdout instead of just logfiles
- Honor proxy and cert checking settings
- Fix extracting files with bsdtar
- Enable locales in all programs
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
In Ostro OS, we already have a "full" directory with all files.
Splitting it up into bundles just so that swupd-create-update can
reconstruct the "full" directory is a waste of IO, and noticably slow
when run under pseudo.
To streamline the required work, a new layout for the "image" input
directory gets introduced:
- The "full" directory gets created by the caller before invoking
swupd-create-update.
- For each bundle, instead of a <bundle> directory, there is a
<bundle>.content.txt file, listing all entries (including directories)
of the bundle.
The traditional mode of operation still works as before because each operation
which normally works with a bundle directory checks whether there is such a
directory and if not, switches to the new mode.
That way it is even possible to mix the two modes, i.e. replacing only
some bundles with a content list, although that's probably not all
that useful.
This revised commit fixes the use of an uninitialized newversiondircontent
pointer in populate_dirs().
Fixes: swupd-server/#54
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
This is a pre-condition for using libarchive directly: libarchive
needs to know what the encoding of filenames is, and it uses the
current locale for that. Without setlocale(), the locale is "C", which
only supports ASCII filenames, leading to warnings about "Can't
encode..." from libarchive when it is forced to fall back to copying
strings verbatim when writing archives that require UTF-8 encoding.
As a side effect, error messages from libc will get translated
according to the user's environment.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
TAR_XATTR_ARGS is no longer used as part of a plain string. Embedding
the empty "" value for bsdtar inside an argv argument list passes an
empty parameter to bsdtar, leading to:
bsdtar: Must specify one of -c, -r, -t, -u, -x
To allow the the "no parameter" case, it has to be argument list: that
can be empty. If not empty, it has to end with a comma.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
In case a proxy is used for downloading the manifest/pack, make sure to
honor those settings. And also, make sure certificate verification
occurs for the downloads.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This reverts commit f01d9ca6c8.
Upon further testing, this patch causes a double free/corruption with the
current master branch and crashes two of the tests. We need to investigate
more before fully enabling it to ensure we don't regress.
When a CI system (like the one from Ostro) captures the output of
commands, but not necessarily intermediate log files, then it is
useful to also log to stdout. Another use case is calling the tools
interactively during development.
The new --log-stdout option in all three commands enables logging to
stdout in addition to the traditional log files.
The implementation recycles the existing init_log_stdout() (not used
before) and gives it the slightly different meaning of "also log to
stdout".
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
The SWUPD_NUM_THREADS env variable is now understood by all three
commands and overrides the default number of threads. Setting it to 1
is useful while debugging the code that runs inside threads (only one
thread hits breakpoints there). If SWUPD_NUM_THREADS is invalid, a
warning is printed and the variable gets ignored, i.e. the default
parallelism is used.
The hard-coded parallelism of 12 threads when analysing the file system
gets replaced with n, where n is the number of available CPUs. The default
is the same as before elsewhere (n for packing, 3 * n for fullfiles).
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
In Ostro OS, we already have a "full" directory with all files.
Splitting it up into bundles just so that swupd-create-update can
reconstruct the "full" directory is a waste of IO, and noticably slow
when run under pseudo.
To streamline the required work, a new layout for the "image" input
directory gets introduced:
- The "full" directory gets created by the caller before invoking
swupd-create-update.
- For each bundle, instead of a <bundle> directory, there is a
<bundle>.content.txt file, listing all entries (including directories)
of the bundle.
The traditional mode of operation still works as before because each operation
which normally works with a bundle directory checks whether there is such a
directory and if not, switches to the new mode.
That way it is even possible to mix the two modes, i.e. replacing only
some bundles with a content list, although that's probably not all
that useful.
Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
Right now, the travis-ci config simply installs required build
dependencies for swupd-server and works around the umask discrepancy to
proper run the functional test suite.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Three functional tests depend on the effective UID being non-zero
(non-root), so skip the tests if running as root.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This conditional checks for state *directories* that are generally
installed by default, and the conditional immediately below this one
checks for state files within these directories. So, if we do strncmp()
instead of strcmp(), the fallthrough logic doesn't occur, and state
files are not marked as such.
This reverts commit 63fb5fb61b.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Since the enablement of signature verification in swupd-client, the
signature creation step has been decoupled from swupd-server, and is
instead performed as a separate step in a DevOps flow. As a result of
this decoupling, the signature code in swupd-server has remained unused.
This commit removes all the signature creation code with the assumption
that the separate DevOps step is going to work better long-term. Also,
the existing signature creation support does not accord with
swupd-client's verification support.
An example of how Manifest.MoM files can be signed is found in the
https://github.com/clearlinux/mixer-tools repo.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Matthew Johnson