There are some problems with how we are using multiplexed curl causing
huge slowdowns (2-5 times slower).
For now rather than rework the multiplexed curl code, switch to using
the synchronous code path.
Signed-off-by: William Douglas <william.douglas@intel.com>
Instead of relying on a custom global_setup and global_teardown
functions, migrate to the use of bats' setup_file and teardown_file
functions.
Signed-off-by: William Douglas <william.douglas@intel.com>
The exact size difference changed for some reason but shouldn't cause
the tests to fail as it isn't the point of this test.
Signed-off-by: William Douglas <william.douglas@intel.com>
The test requires an exact size reporting, fix the size but consider
using a regex match instead.
Signed-off-by: William Douglas <william.douglas@intel.com>
Environment variables are used everywhere in testilb. This environment
variables are global variables that define the way testlib behaves.
However is was confusing to use the variables because they were
inconsistent between each other, for example some variables that define
paths would have absolute paths while other would have relative paths,
making it error prone while using them.
This commit makes the environment variables more consistent by following
a name convention for each type of variable, as an example, variables
that define absolute paths follow this convention ABS_<path_name>_DIR,
while variables that define relative paths are defined like this
<path_name>_DIR.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Currently there is only one location, known as "statedir", where swupd
stores data and cache. Normally it make sense to keep the data in a
path that is relative to the path prefix of the target system, since the
data is specific to that system, but cache can be independent of the
system, and that way it can be reused for other target systems if
desired.
This commit splits the cache and data into two different locations.
It also stores the cache in a path that is dependent of the mirror url
being used by swupd. The commit also makes all references to files in
the cache or data directories through getter functions thus removing the
hardcoded reference to the file's location, making it easier to move the
location in the future if needed. Lastly, this commit changes the read
permissions for the data directory and the manifest directories in the
cache to be user readable.
When swupd downloads manifests to the statedir they are downloaded in
directories that represent the version of clear that the manifests
belong to. However those version directories are stored in the statedir,
root directory. This commit moves them into their own "manifest"
directory so there is more consistency with the downloaded files.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When swupd downloads manifests to the statedir they are downloaded in
directories that represent the version of clear that the manifests
belong to. However those version directories are stored in the statedir,
root directory. This commit moves them into their own "manifest"
directory so there is more consistency with the downloaded files.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This commit provides a minimal output to be displayed when the --quiet
flag is used for these commands:
- swupd repair
- swupd 3rd-party repair
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Current implementation was only removing test enviroment for envs created
on test setup. And test_teardown function needed to be overriden in case
global_setup() was used.
Now always cleaning the enviroment for both use cases, test_setup() and
global_setup(). The user doesn't have the control anymore on that and
if the user wants to keep the env KEEP_ENV variable should be used.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Test environment can be created on global_setup() or test_setup() and that
change the behavior of several operations on testlib.
Because of that, let the user create the test environment as desired instead
of forcing it on test_setup()
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
2 groups were created:
- slow: Tests that are very slow because of sleeps, so we can run them all
in parallel
- system: Tests that make change to the system, so they can cause problems
when running in parallel, so we run them in series.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
This patch adds 2 scripts used to balance test execution. The weight_tests.bash
runs all tests and sets a weight to them based on how long they take to run. The
other, filter_bats_list.bash, use this information to split the tests in groups to
be executed by github actions.
When a new test is added the script will consider it with an average weight, so this
shouldn't unbalance the system right away. After some time, if we notice that the
system is not balanced anymore we can just run the weight_tests.bash again to rebalance.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
When diagnosing/repairing a system, sometimes is useful to only
diagnose/repair a specific file or path.
This commit implements the --file option for diagnose/repair so a file
or path can be diagnosed only instead of doing it to the whole OS or a
whole bundle.
Closes#1150
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
In order to report progress accuratelly, currently swupd requires a
hardcoded number of steps per operation so we can report how far in the
operation we are. Most of these step totals are wrong.
This commit fixes the number of steps in many swupd functions.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This commit enhance the consistency and readability, of the messages
shown when the signature verification is skipped or fails.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Heuristics and post-update scripts that apply to upstream bundles don't
apply to bundles installed from 3rd-party repositories. This commit
enables/disables these actions on demand so 3rd-party bundles are
installed correctly.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Insteaf of validating the tracking directory exists and is not empty
every time we want to track a bundle, we can validate it during swupd
initialization.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Print a "Validate downloaded files" message to separate 2 different progress bars
and improve the output when downloading extra fullfiles
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Changing the progress report lib to always print 0 or -1 when a step is
started and a 100 when a step finishes.
Also reorganizing all commands to have less steps (when possible) and that
are more meaningful to progress report. Hide some internal steps that doesn't
provide useful information to end users.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Reorganize bundle add main function to reuse more code and improve performance.
Stopped using subscription code from bundles and now using a new function to
recurse manifests, gaining around 30% cpu time on manifest processing.
Started using staging code from update and preventing checking hashes more than
once for some files on bundle-add operations gaining around of 30% CPU time too.
Overall executions of this new bundle-add uses 30% less CPU time but because this
operation is very IO intensitive this reflects to a gain in around 10% of total
time in systems I tested.
When installing a bundle using bundle-add, swupd also installs any
dependency not already installed in the system (if any). When the
bundle-add operation is complete, swupd reports how many of the
requested bundles were successfully installed and how many failed, but
if there are bundles that got installed as a side effect because they
are dependencies of the requested bundle(s), they are not included in
the summary of bundle-add.
This commit adds the number of dependencies installed in the summary of
bundle-add so it provides a more accurate view of the changes in the
system.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
There was a bug in the logic of the setup function in testlib which was
causing the globale_teardown to be incorrectly called when tests were
being run using "bats <directory>/". This commit fixs the issue.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Instead of initializing curl library on start, initialize it only when needed.
The advantage of this approach is that we can run some commands offline, if
there's no file to download.
Fixes#801Fixes#895Fixes#277
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
In the case that we have a bundle installed as also-add and removed later,
repair shouldn't reinstall that bundle. The same is valid for diagnose and update.
Os-install is the only exception. We should always install all bundles that are
listed as also-add in os-install.
Signed-off-by: Otavio Pontes <otavio.pontes@intel.com>
Swupd will install optional bundles on bundle-add by default,
but will skip them if specified by the user by using the
--skip-optional / -o flag..
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
An optional bundle is not required to be installed in the system while
included bundles are. Swupd will install optional bundles on bundle-add
by default.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Currently users can set content and version urls based on http or https
protocols. This pose a security risk if users decide to use http.
This commit blocks swupd from working with http unless is specifically
allowed by using the --allow-insecure-http flag.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This commit adds a few more cosmetic changes to the commands that run
verify in the back for consistency.
- Different steps in the update process are separated by a blank line.
- Messages from swupd should not finish with a '.'
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
When swupd is installing a bundle, it goes through the list of files to
install and removes those files that already exist in the target system
from the list, regardless of if the files were added by other bundle or
not.
This commit changes this process so the file is removed from the
list of files to install only if the file is listed as installed in the
manifest of one of the already installed bundles. This way if the file
is already present in the system, but not listed in any other manifest
it can be overwritten since we cannot assume the file is correct.
Closes#863
When staging a file using do_staging(), if the path of the file is
missing verify_fix_path() is called to try fix the missing path.
verify_fix_path() then removes bad directories (if any), downloads the
file and again and stages the new directory using do_staging() again, doing
a circular reference. On top of this bundle add is calling
verify_fix_path() after running do_staging(). All this circular
reference makes the code very difficult to understand and may cause
swupd to do extra work that is not necessary. This commit simplifies the
code by not allowing do_staging() to call verify_fix_path() so we can
manually call it later if necessary.
This commit also add some more comments in the code to make it easier to
follow.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
Adding messages where swupd could take some time to finish a step so
users know better where the process is at.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
The json tests had been disabled because they were unstable. This issue
was fixed in a previous commit, so these tests should be enabled again.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
This commit makes use of the swupd_progress_callback() function to
report progress downloading fullfiles based on how much data we have
downloaded vs the number of files downloaded. This callback will only be
used when the number of files to be downloaded are less than MAX_FILES,
calculating the total download size can be very costly if the files are
too many. If the files to be downloaded are more than MAX_FILES we will
fallback to reporting download progress based on file count as before.
When installing bundles or doing updates, swupd creates a list of files
that need to be downloaded. This list may contain files that were
already downloaded via packages and it often does. These files are then
skipped at the moment of downloading them since they are already in the
system. This causes a misleading output that shows the user that
fullfiles will be downloaded when they are actually not.
This commit filters the list of fullfiles to be downloaded to only
contain those ones that actually need to be downloaded.
Signed-off-by: Castulo Martinez <castulo.martinez@intel.com>
William Douglas