Initial support for concurrent serial and parallel build.
Parallel MPI build enabled via a new setting in "options.conf":
openmpi = false|true (false is default)
* Serial builds remain unaffected.
* Parallel build: Only OpenMPI is supported.
* Parallel builds always assume AVX2.
* Build patterns implemented: "configure" and "cmake".
The build process is controlled by additional files:
"cmake_arg_openmpi": same functionality as "cmake_args"
"configure_openmpi": same functionality as "configure"
The openmpi (configure, build, check, install) sections follow these steps:
module load openmpi
...[configure,build,install,check]
module unload openmpi
Once "openmpi=true", the packages "openmpi-dev", "modules" and "openssh"
are automatically added to BuildRequires. ("openssh" is generally required
to run any MPI test suites).
Packages created:
foo-openmpi: binaries, libraries, docs, ...
foo-dev : contains both serial and MPI devel. files
foo-staticdev: contains both serial and MPI devel. files
The package alias is intended to be used as a manual override in cases
where the upstream package name is not the one that Clear Linux used
as the package name (due to conflicts or other reasons) and there is
tooling that requires the use of the upstream package name that
autotools runs (pypi is the motivating usage currently).
The combined usage of "subdir" and AVX2/AVX512 builds fails to install
for certain build patterns (notably for "configure").
This can be verified, for example, by trying to build the package "ntl" with:
use_avx2 = true
use_avx512 = true
This will fail with an error during install, similar to this:
pushd: ../buildavx512/: No such file or directory
This patch fixes this by descending to the correct folder:
pushd ../buildavx512/src
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
Get metadata about the package from pypi for distutils3 build patterns
then use that metadata to get summary, description and provides
information (license and requires maybe leveraged in the future).
The description isn't always very useful (often too long) so only use
it if there isn't one already.
This patch consolidates the code that manages the sources
(.zip, .tar, etc) in Source class. This class is written in a way
that allows the reuse of code for all current supported types of
sources and at the same time, it allows the easy addition of new
ones.
Signed-off-by: Athenas Jimenez <athenas.jimenez.gonzalez@intel.com>
Because rpmbuild dumps a list of build errors after the build has
stopped and any errors have occurred, the "Installed (but unpackaged)
file(s) found" line is repeated, and thus autospec will always parse the
list of unpackaged files twice. Also, the repeated listing of unpackaged
files is sometimes truncated, possibly due to a bug in rpmbuild.
Since autospec already understands how to locate the end of the initial
unpackaged file listing, it can avoid the duplicate parsing and any
possible truncation issues by ensuring that the `infiles` variable is
set to `1` exactly once. Then, `infiles` will continue to have value `2`
from the end of the first file listing until parsing stops.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Add a special archives list (go_archives), so we can remove Go specific
methods and simply make use of regular methods.
Signed-off-by: Athenas Jimenez <athenas.jimenez.gonzalez@intel.com>
For the meson pattern, scan for any calls to `test(...)` within
meson.build files in the source tree.
The scan exits early if the toplevel srcdir lacks a meson.build file,
but we can probably remove that check in the future to support test
execution of projects like `zstd`.
Fixes#208
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Support the output of `meson test` in addition to `ninja test`, since
the meson docs recommend running `meson test`.
Also add a new unit test to cover some of the test result summary line
differences.
Fixes#323
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
In this function, returning `None` indicates success, so instead, a
GPGCliStatus instance should be returned for the new error cases.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Older versions of gnupg do not print the "packet separator" lines in
`gpg --list-packets` output, so the length information is not parsed by
autospec and thus verification must be skipped for now.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
There are only two GPG packet types that autospec needs to know about
for now ("signature" and "user ID"), so update `parse_gpg_packets` to
only add those packets for tracking.
Also, update unit tests to account for packets that are now skipped,
thus reducing the size of the packets list.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The only package in Clear Linux OS that (occasionally) has its source
archive signed by multiple keys is `gnupg`, so to unblock updates for
it, only verify the first signature from the .sig file.
When only one signature is present in the sig file (the common case),
pass the entire sig file to `gpg --verify ...`, keeping the existing
behavior.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
To prepare for more general parsing of GPG key and signature files,
implement a new function, parse_gpg_packets(), and start using it where
appropriate.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
These characters should never appear in version strings, because rpm
complains. If there are additional percent-encoded characters to add in
the future, we can update the list.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
In the process of updating packages to new releases, the build may fail,
and the spec file and other files are left in an intermediate,
uncommitted state.
This intermediate state is useful for some of the files, but for the
spec file (generally), we care more about the state transition from
commit to commit.
This change begins teaching autospec how to read spec file state from
the latest commit, specifically for the "old version" of the package and
the patch list from that old version.
Fixes#433
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The buildreq_cache is only meant to apply when re-packaging the same
version. To prevent the cache from becoming stale during version
changes, remove it initially. If the new version has cached buildreqs,
the file will be written later.
Fixes#481
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
PECL tarballs have an annoying `package.xml` in the toplevel. We
detect PECL by URL and eliminate this file from the files list
so that we do not need to compensate for path magic. This fixes
all the PECL builds. Thanks to Athenas for coming up with the
magic bit :).
The following pattern match expression:
match = re.search(r"(---\s+)?FAIL:?\s*", line)
would match both XFAIL and FAIL.
In the case of XFAIL in the line, the match would be interpreted as FAIL.
Remedy: use this instead:
match = re.search(r"(---\s+)?(?<!X)FAIL:?\s*", line).
Kudos to phmccarty.
write_configure_pattern: for avx512 the user defined
"build_prepend" was executed outside of the avx512 build
folder.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
In the section of the mock build.log that lists unpackaged files, rpm
4.14 will print warnings if any files have build ID issues.
Because file names must begin with a forward slash, it's easy to filter
out these warnings, since the warnings themselves do not begin with a
forward slash.
Fixes#547
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
The recently added read_script_file() function preserves newline
characters, and because the specfile writer functions already add
newlines, extra blank lines were being written to spec files.
Avoid the extra newline characters by stripping them with rstrip(). Note
that any trailing whitespace is also stripped.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
If a banned dependency is added to the buildreqs, buildreq_cache, or
requires sets before the configuration files are read, the dependency
will not actually be banned.
Fix this issue by removing banned dependencies from the sets if present
during *_ban configuration file parsing.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Don't skip empty lines or lines starting with # or simplify
whitespace. This makes non-trivial scripts nicer to read in the .spec
file, like those found in llvm.spec:
for f in *; do
...indented lines go here...
mv $f $f-$VERSION
ln -s -f $f-$VERSION $f
done
We skip the first line if it is a shebang or a file format annotation.
If CPL license is found by autospec, we end up with an
error:
ERROR: License 'CPL' is not an allowed SPDX license ID
Translate CPL to a valid SPDX license CPL-1.0
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
William Douglas