The content map can never be empty at this point due to the assignment
so remove the test.
We could have checked the files element in the map to see if it was
empty but printing a warning for an empty extras file isn't very
valuable.
Due to more systems being added for automated runtime dependency
injection for subpackages, we need a way for manual overrides of the
default detected dependencies.
This change provides support for specifying which subpackage runtime
dependencies should be added or removed and consolidates some of the
custom configuration file detection along with it.
In case autospec's output is logged to a file, it's better to not print
the escape sequences.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The `pkg_integrity` module defined its own progress message functions
that differed from the ones used by the rest of the code base: they used
slightly different colors (the "bright" color variants) and had
different formatting.
For better maintainability, remove the `pkg_integrity` progress message
functions, and extend/refactor the ones from `util` to cover the ones
`pkg_integrity` needs.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The pattern must be passed to the source class to handle pecl archive
files correctly. It would be good to have the configuration files be
loaded before the sources are processed but that's a later work item
and is unlikely to cause problems in this case.
Fixes#617.
The buildpattern is a bit of an odd standalone module that was a
vestiage of the pre-refactor days where modules owned global
data. These days are coming to a close so move this data to a more
sensible owner in config.
This resolves#612 though likely more will be done as part of #131.
Instruct pip to not use a keyring by setting an environment variable
(PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring). As far as I
know, there is no need to use a keyring for any of the pip commands
autospec runs.
Some users have encountered timeouts similar to the report from
https://github.com/pypa/pip/issues/7883, which describes this
environment variable setting as a workaround.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Many different components require the target download path and it is
needed early. Previously code existed to attempt to autodetect its
location and so defining the path was enforced later in the build
process. This created problems with ordering functions based on this
requirement but since the target path is now certain to be defined at
startup (due to previous changes) we can better align the location and
initialization of the data to the Config class.
As part of this change also avoid requiring the temporary working
directory to be setup as part of the Build class. This is done as a
cleanup due to the only real user of this location being the Content
class.
This also drops the stand-alone pkg_integrety main function that was
initially used as a one off test to avoid needing to run autospec to
validate the integrity of a package. This feature was seldom used and
doesn't function well without handling configuration so it was
dropped.
Similar to the URL, the common tooling always passes the target
directive to autospec. Using this convention we can better initialize
autospec state earlier and with less fallbacks.
Some test suites install files that can't be auto-detected, so support a
tests-extras file to add extra files to the -tests package (not
-extras-tests).
The logic behind the files extras, dev_extras and *_extras is
essentially identical but duplicated. Consolidate the code and internal
variables into a single parse and file_maps variable.
Many GNOME packages install their test suite so that it be run in situ
instead of in the limited environment of the builder. However, these
files for foo.spec are typically packaged into either foo or foo-libexec
and so always installed.
Instead, package /usr/(share|libexec)/installed-tests into foo-tests, so
that they are not always installed but the packages exist for potential
bundles in the future.
The fourth argument to `parse_config_files` is supposed to be the
content (tarball) version, not the Content instance itself. The
buildreq_cache file handling in `parse_config_files` is the only code
affected by the bug.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
There was a simple pattern for systemd (oddly not a pkgconfig pattern
since it is using pkg-config) that is usable by at least one other
package pattern so move it into a failed pattern. Also add the other
package to the failed_commands file.
This covers the "ENOENT" case. Also update the unittests to verify that
the updated pattern works.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The rationale behind this was to support extra tarball for R-RSQLite,
in order to bump the bundled sqlite version using sqlite's upstream
tarball instead of a local (huge) patch applied to the package.
Other installations welcome. I could have added them to the "-dev"
package but didn't think most people want the examples when
developing. This will be used in a new bundle in Clear Linux.
Needed to fixclearlinux/distribution#1781.
Fortran developers may be using either FCFLAGS, FFLAGS or both.
Common usage is:
FCFLAGS : Fortran 90 compiler flags
FFLAGS : Fortran 77 compiler flags
If both of them are initialized, they must be identical.
This patch addresses two issues:
1. https://github.com/clearlinux/autospec/issues/538
The assumption CFLAGS and FFLAGS are identical is no longer valid.
Using CFLAGS for FFLAGS will result in compiler warnings such as:
f951: Warning: command line option -Wformat=1 is valid for C/C++/ObjC/ObjC++ but not for Fortran
f951: Warning: command line option -Wformat-security is valid for C/C++/ObjC/ObjC++ but not for Fortran
Not only are the warnings a nuisance, but if warnings are treated as
errors, this can break builds. The remedy is to initilize FFLAGS using
the environment $FFLAGS instead of the environment $CFLAGS.
2. https://github.com/clearlinux/autospec/issues/580
AVX2/AVX512 instructions not generated for Fortran
When selecting builds with AVX2/AVX512 optimizations, there are no
FFLAGS or FCFLAGS specified, hence any resulting Fortran code will
not generate any AVX instructions. Fortran is mostly used for
intensive mathematical/numerical/scientific code and as such would
very likely benefit from AVX instructions a lot.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
In the recently removed `load_specfile()` function in buildreq.py (see
commit 2a181a2d), the pypi_requires set was sorted and converted to a
list via `sorted()`. Now, the set is no longer converted to a list, so
we can instead sort it immediately before writing out the contents in
`write_files_header()` in specfiles.py.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The build module had a number of globals that were referenced by many
other modules and has ordering dependencies with the config and
tarball module for some values. This made deciding on where certain
values get initialized difficult but before the initialization can be
addressed a refactor is helpful.
This change moves the global state (and functions that needed to act
on that global state) into a Build class. The goal of this work is
to better track what can be updated by a particular function, load
data in a sensible order and have it owned by a sensible component.
Some of this data is likely to be removed from the build class in the
future as initialization of download and base paths should take place
in other modules most likely.
The tarball module had a number of globals that were referenced by
many other modules and has ordering dependencies with the config
module for some values. This made deciding on where certain values get
initialized difficult but before the initialization can be addressed a
refactor is helpful.
This change moves the global state (and functions that needed to
act on that global state) into a Content class. The goal of this work
is to better track what can be updated by a particular function, load
data in a sensible order and have it owned by a sensible component.
I expect that future work will see the tarball, buildpattern and some
portions of the config moved around to better match what component
should own what data (and to better deliniate between the data that
has a similar use but comes from different sources).
Python openmpi files such as:
/usr/lib64/openmpi/lib/python3.8/site-packages/foo.py
should be packaged in <package>-openmpi instead of <package>-dev.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
The buildreq module had a large number of globals being touched across
many modules that would import it. This made changes to the buildreq
fragile.
It also made testing fragile due to certain globals needing to be
reset in order to properly test a change.
This change moves the global state (and functions that needed to act
on that global state) into a Requirements class. The goal of this work
is to better track what content can be updated by a particular
function and load data in a sensible order and have it owned by a
sensible component.
Omitting the package NVR from the conditional will result in an
IndexError if the NVR does not actually exist on that line.
This change fixes the packaging for `epm` in Clear Linux OS, which
prints "file not found" errors when running its unit tests, but with
the i386 arch and slightly different NVR.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The config module had a large amount of globals that were being
touched across many modules that would import. This made changes to
config very fragile as figuring out what would be modified in any
given call chain was difficult to diagnose.
It also made testing fragile as one would need to reset a given
module's config import to the best of their knowledge before rerunning
another test.
To get away from that (and to try and reduce the number of globally
modified variablies in autospec as a whole), refactor the config
module to provide its state as part config class. The long running
goal of changes like this is to better track what content can be
updated by a particular function (if a function would have access
to the config instance is now the hint rather than the config module
getting imported).
Initial support for concurrent serial and parallel build.
Parallel MPI build enabled via a new setting in "options.conf":
openmpi = false|true (false is default)
* Serial builds remain unaffected.
* Parallel build: Only OpenMPI is supported.
* Parallel builds always assume AVX2.
* Build patterns implemented: "configure" and "cmake".
The build process is controlled by additional files:
"cmake_arg_openmpi": same functionality as "cmake_args"
"configure_openmpi": same functionality as "configure"
The openmpi (configure, build, check, install) sections follow these steps:
module load openmpi
...[configure,build,install,check]
module unload openmpi
Once "openmpi=true", the packages "openmpi-dev", "modules" and "openssh"
are automatically added to BuildRequires. ("openssh" is generally required
to run any MPI test suites).
Packages created:
foo-openmpi: binaries, libraries, docs, ...
foo-dev : contains both serial and MPI devel. files
foo-staticdev: contains both serial and MPI devel. files
The package alias is intended to be used as a manual override in cases
where the upstream package name is not the one that Clear Linux used
as the package name (due to conflicts or other reasons) and there is
tooling that requires the use of the upstream package name that
autotools runs (pypi is the motivating usage currently).
The combined usage of "subdir" and AVX2/AVX512 builds fails to install
for certain build patterns (notably for "configure").
This can be verified, for example, by trying to build the package "ntl" with:
use_avx2 = true
use_avx512 = true
This will fail with an error during install, similar to this:
pushd: ../buildavx512/: No such file or directory
This patch fixes this by descending to the correct folder:
pushd ../buildavx512/src
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Get metadata about the package from pypi for distutils3 build patterns
then use that metadata to get summary, description and provides
information (license and requires maybe leveraged in the future).
The description isn't always very useful (often too long) so only use
it if there isn't one already.
This patch consolidates the code that manages the sources
(.zip, .tar, etc) in Source class. This class is written in a way
that allows the reuse of code for all current supported types of
sources and at the same time, it allows the easy addition of new
ones.
Signed-off-by: Athenas Jimenez <athenas.jimenez.gonzalez@intel.com>
Because rpmbuild dumps a list of build errors after the build has
stopped and any errors have occurred, the "Installed (but unpackaged)
file(s) found" line is repeated, and thus autospec will always parse the
list of unpackaged files twice. Also, the repeated listing of unpackaged
files is sometimes truncated, possibly due to a bug in rpmbuild.
Since autospec already understands how to locate the end of the initial
unpackaged file listing, it can avoid the duplicate parsing and any
possible truncation issues by ensuring that the `infiles` variable is
set to `1` exactly once. Then, `infiles` will continue to have value `2`
from the end of the first file listing until parsing stops.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Add a special archives list (go_archives), so we can remove Go specific
methods and simply make use of regular methods.
Signed-off-by: Athenas Jimenez <athenas.jimenez.gonzalez@intel.com>
Patrick McCarty