The current format is very useful information to get from bug reports
so add it as default output to swupd update.
Signed-off-by: William Douglas <william.douglas@intel.com>
Allow users to update by stepping through each release between their
current version and the latest. This option is primarily for cases
where update is failing due to memory or disk space running out when
updating normally.
Signed-off-by: William Douglas <william.douglas@intel.com>
Currently sys_rm_recursive was used in any instance of deleting swupd
content from the system (update, repair and bundle-remove). This can
cause user data loss when unkown files are in directories that swupd
is deleting.
To prevent this, this patch changes how deleting content in swupd
operates. Swupd content removal is now done with sys_rm and the return
value is checked in case the removal failed due to a directory that
still had files in it. When this specific failure occurs, the
directory is added to a new list for reprocessing removals as it is
expected once the rest of the deletes on the system occur the failures
will go away as the directories will be empty (these deletes are
processed in alphabetical reverse order so leaf directories are
processed first). If the removal fails again it is presumed the
contents of the directory are not files swupd knows about and as such
should be kept somewhere else.
For handling the retention of user data, directories (with only the
content unknown to swupd) are renamed (currently using a
.deleted.$timestamp. prefix of the old name) and stored at the same
directory level they were previously found with one exception. The
exception is for nested deleted content best illustrated with an
example:
/swupd-dir1/user-file1
/swupd-dir1/swupd-dir2/user-file2
When swupd tries to remove the /swupd-dir1 content, it will store the
user files as follows:
/.deleted.$timestamp1.swupd-dir1/user-file1
/.deleted.$timestamp1.swupd-dir1/.deleted.$timestamp1.swupd-dir2/user-file2
To demarcate what was part of swupd content vs user content.
Signed-off-by: William Douglas <william.douglas@intel.com>
There are some problems with how we are using multiplexed curl causing
huge slowdowns (2-5 times slower).
For now rather than rework the multiplexed curl code, switch to using
the synchronous code path.
Signed-off-by: William Douglas <william.douglas@intel.com>
Instead of relying on a custom global_setup and global_teardown
functions, migrate to the use of bats' setup_file and teardown_file
functions.
Signed-off-by: William Douglas <william.douglas@intel.com>
Environment variables are used everywhere in testilb. This environment
variables are global variables that define the way testlib behaves.
However is was confusing to use the variables because they were
inconsistent between each other, for example some variables that define
paths would have absolute paths while other would have relative paths,
making it error prone while using them.
This commit makes the environment variables more consistent by following
a name convention for each type of variable, as an example, variables
that define absolute paths follow this convention ABS_<path_name>_DIR,
while variables that define relative paths are defined like this
<path_name>_DIR.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Currently there is only one location, known as "statedir", where swupd
stores data and cache. Normally it make sense to keep the data in a
path that is relative to the path prefix of the target system, since the
data is specific to that system, but cache can be independent of the
system, and that way it can be reused for other target systems if
desired.
This commit splits the cache and data into two different locations.
It also stores the cache in a path that is dependent of the mirror url
being used by swupd. The commit also makes all references to files in
the cache or data directories through getter functions thus removing the
hardcoded reference to the file's location, making it easier to move the
location in the future if needed. Lastly, this commit changes the read
permissions for the data directory and the manifest directories in the
cache to be user readable.
When swupd downloads manifests to the statedir they are downloaded in
directories that represent the version of clear that the manifests
belong to. However those version directories are stored in the statedir,
root directory. This commit moves them into their own "manifest"
directory so there is more consistency with the downloaded files.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Print a more useful error when we fail to locate a file to apply a
delta. Add a suggestion for the user to run `swupd repair` and make
it clear that the problem is in their system and not in the update
content.
Fixes#1425
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
When swupd downloads manifests to the statedir they are downloaded in
directories that represent the version of clear that the manifests
belong to. However those version directories are stored in the statedir,
root directory. This commit moves them into their own "manifest"
directory so there is more consistency with the downloaded files.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Ignoring error SC2119 because they are false positives and I don't see cases
where that would return anything useful for us.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
The show_target function shows a warning that is not applicable so this
commit ignores that warning wherever show_target is used.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Headers are printed at the beginning of the output for every repository
in operations that run in multipl repos. This header was using the word
"repo" but we are trying to use the whole "repository" word every time
we printed it to the user. This commit changes that.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This commit adds a flag that can be used to update content from
3rd-party repositories after updating the regular content if
successfull.
Closes#1473
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This commit adds the following files to trigger the execution of
clr-boot-manager:
- /usr/bin/clr-boot-manager
- /usr/share/syslinux/ldlinux.c32
Closes#1415
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Make sure files with unsual characters are successfully installed. Most
characters used in test are filtered by mixer, but there's no reason to
not support them in swupd
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
This commit provides a minimal output to be displayed when the --quiet
flag is used for these commands:
- swupd repair
- swupd 3rd-party repair
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Current implementation was only removing test enviroment for envs created
on test setup. And test_teardown function needed to be overriden in case
global_setup() was used.
Now always cleaning the enviroment for both use cases, test_setup() and
global_setup(). The user doesn't have the control anymore on that and
if the user wants to keep the env KEEP_ENV variable should be used.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Test environment can be created on global_setup() or test_setup() and that
change the behavior of several operations on testlib.
Because of that, let the user create the test environment as desired instead
of forcing it on test_setup()
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
2 groups were created:
- slow: Tests that are very slow because of sleeps, so we can run them all
in parallel
- system: Tests that make change to the system, so they can cause problems
when running in parallel, so we run them in series.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
This commit removes the extra slash '/' from the path from:
- the update_boot function
- the verifytime script
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When a manifest delta was incorrect, the full manifest was used. But in the
case of a correct delta that produces an invalid manifest, swupd was never
trying to download the full manifest.
Changing code to try to use deltas only once. On first error, always retry
downloading the full manifest.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
When trying to update if the mirror has an invalid signature file we should
try to use the default source url instead.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
- Reorganizing function to be easier to read
- If path is problematic and there's no access version url config, use the
one set using compile flags
- Use correct url to check for local
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
This patch adds 2 scripts used to balance test execution. The weight_tests.bash
runs all tests and sets a weight to them based on how long they take to run. The
other, filter_bats_list.bash, use this information to split the tests in groups to
be executed by github actions.
When a new test is added the script will consider it with an average weight, so this
shouldn't unbalance the system right away. After some time, if we notice that the
system is not balanced anymore we can just run the weight_tests.bash again to rebalance.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
When diagnosing/repairing a system, sometimes is useful to only
diagnose/repair a specific file or path.
This commit implements the --file option for diagnose/repair so a file
or path can be diagnosed only instead of doing it to the whole OS or a
whole bundle.
Closes#1150
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
In order to report progress accuratelly, currently swupd requires a
hardcoded number of steps per operation so we can report how far in the
operation we are. Most of these step totals are wrong.
This commit fixes the number of steps in many swupd functions.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This commit modifies some of the functions of the test library so it is
possible to update bundles that belong to 3rd-party repositories.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Print a "Validate downloaded files" message to separate 2 different progress bars
and improve the output when downloading extra fullfiles
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Changing the progress report lib to always print 0 or -1 when a step is
started and a 100 when a step finishes.
Also reorganizing all commands to have less steps (when possible) and that
are more meaningful to progress report. Hide some internal steps that doesn't
provide useful information to end users.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Reorganize bundle add main function to reuse more code and improve performance.
Stopped using subscription code from bundles and now using a new function to
recurse manifests, gaining around 30% cpu time on manifest processing.
Started using staging code from update and preventing checking hashes more than
once for some files on bundle-add operations gaining around of 30% CPU time too.
Overall executions of this new bundle-add uses 30% less CPU time but because this
operation is very IO intensitive this reflects to a gain in around 10% of total
time in systems I tested.
There was a bug in the logic of the setup function in testlib which was
causing the globale_teardown to be incorrectly called when tests were
being run using "bats <directory>/". This commit fixs the issue.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Instead of initializing curl library on start, initialize it only when needed.
The advantage of this approach is that we can run some commands offline, if
there's no file to download.
Fixes#801Fixes#895Fixes#277
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
When running the check-update command we get the latest version for the
format we are currently in, but we really want to get the latest version
regardless of the format.
This commit fixes the issue.
Closes#482
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Estimating the total download size is almost as slow as downloading the
manifests. So just don't do it.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
William Douglas