Commit Graph

1221 Commits

Author SHA1 Message Date
William Douglas 3fa292013c Fixup cmake_module names
pypi ecosystem migration for -python3 suffixed names to use pypi()
namespace instead.

Signed-off-by: William Douglas <william.douglas@intel.com>
2022-02-03 10:27:39 -08:00
William Douglas cfaaa04537 Have setuid content honor attrs
Previously setuid files would only set ownership to root,root (and no
other attrs). In cases where content should be setuid with different
ownership, the configuration was only set in the attrs file for a
path. Unfortunately filemaps only look at the setuid file to know
which paths to skip over for calling elf-move and so setuid files were
not being accounted for.

This change allows attrs for setuid files to be set, enabling checking
for content with setuid bits more readily in the process.

Signed-off-by: William Douglas <william.douglas@intel.com>
2022-01-27 15:57:11 -08:00
William Douglas 4fb4a2780b Do elf_move after install_append
It is easier to use install_append to change content before elf_move
runs than it is to modify content in install_append after elf_move
runs so order them thusly. Could potentially need a new install_append
file that runs after elf_move but so far it isn't needed

Signed-off-by: William Douglas <william.douglas@intel.com>
2022-01-21 16:41:28 -08:00
William Douglas 5b0b5fd85e Skip setuid optimized files
Optimized files that need to be setuid are difficult to handle, the
filemap would need to be scanned and the files themselves individually
accounted for in the spec file %files section. To work around this
difficulty, don't allow setuid files to be put into the optimized file
contents.

Signed-off-by: William Douglas <william.douglas@intel.com>
2022-01-21 16:12:41 -08:00
William Douglas 748df4781f Make install_append last
The install_append contents should be the last thing written to the
%install section.

Signed-off-by: William Douglas <william.douglas@intel.com>
2022-01-21 16:12:41 -08:00
William Douglas c712642011 Fix test processing
If a file addition only failure happens and a quick rebuilt is done,
the build log will be missing test results. Check the previous build
log in that case.

Signed-off-by: William Douglas <william.douglas@intel.com>
2022-01-21 11:42:56 -08:00
Patrick McCarty ed01ee40c8 license: also support license directory containing license texts
In dpdk 20.11.3, the REUSE approach is partially adopted, with the
`LICENSES` directory named `license` instead.

Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
2022-01-13 11:10:41 -08:00
William Douglas 61af4ac6f5 pypi conversion test fixups 2022-01-12 04:44:34 -08:00
William Douglas 124291cf03 Full rebuilds when excludes are active
This could be improved if we wanted to try and cross check excluded
files from the files list but this is more trouble than it is worth
most likely (usually mostly an issue for compat packages).

Signed-off-by: William Douglas <william.douglas@intel.com>
2022-01-12 04:44:34 -08:00
William Douglas 5b25cfeaa1 More pypi renames 2022-01-12 03:33:04 -08:00
William Douglas 00baf595e1 More pypi renames 2022-01-11 13:30:36 -08:00
William Douglas 1f5592743f Add pypi six failed command 2022-01-10 17:37:04 -08:00
William Douglas 6e3e9e430f More failed command pypi fixes 2022-01-10 16:28:52 -08:00
William Douglas 5e6beca394 failed command typo 2022-01-10 14:13:01 -08:00
William Douglas ea3345266a More pypi fallout changes 2022-01-10 14:06:44 -08:00
William Douglas 279c2a4aa5 pkg_integrity: Allow fallback methods to be used
In cases where the signature file can't be verified as no key can be
found, fallback to other verification methods instead of failing
verification completely.

Signed-off-by: William Douglas <william.douglas@intel.com>
2022-01-10 13:26:21 -08:00
William Douglas 22acdf5346 Update python packages from pypi migration
The Clear Linux python ecosystem has migrated the bulk of the pypi
packages to its own package namespace. Update autospec to account for
these changes.

Still needs test fixes but merging for now.

Signed-off-by: William Douglas <william.douglas@intel.com>
2022-01-10 13:24:18 -08:00
William Douglas aabe801c3e Avoid rebuilding the package after only file adds
Try to avoid doing a full rebuild when the build.log only has new
files in it.

This has a two downsides. The source buildroot needs to be distinct
from the binary buildroot. The built rpms are no longer installable as
rpmbuild flags them.

The upside is packages that take forever to build or don't have
build times improved by ccache will not need a duplicate round for
just adding the files.

The rpms can still be generated in an installable fashion with 'make
build' and the autospec generated rpms can be installed by force
installing if need be.

Signed-off-by: William Douglas <william.douglas@intel.com>
2022-01-05 08:23:17 -08:00
William Douglas 225aa03851 Add Zstd failed command
Signed-off-by: William Douglas <william.douglas@intel.com>
2022-01-04 16:31:26 -08:00
William Douglas 5cf936749f buildreq: Ignore badly formed pypi package bits
Signed-off-by: William Douglas <william.douglas@intel.com>
2021-12-17 15:14:44 -08:00
William Douglas e3d4b53ecb config: Add new cmake failed pattern
Add new cmake failed pattern that was found trying to detect
yaml-cpp.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-12-17 15:14:44 -08:00
William Douglas cbc91dfe71 buildreq: Sometimes install requires are also build requires
This imporves the ability for pypi ecosystem pyproject packages to
build more consistently. Might consider making this a last attempt if
build fails instead though to avoid adding spurious buildreqs.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-12-17 11:56:08 -08:00
William Douglas 0c780b3205 Remove support for the java build systems
These are no longer used in Clear Linux so the code can be removed
without issue.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-12-17 11:56:08 -08:00
William Douglas c525138663 config.py: Extend meson C library missing regex
Signed-off-by: William Douglas <william.douglas@intel.com>
2021-12-09 22:57:54 -08:00
William Douglas 442b42e0bb Enable the 32bit install to use the install_macro
Signed-off-by: William Douglas <william.douglas@intel.com>
2021-12-09 22:57:54 -08:00
William Douglas 81e2121ab2 Handle the new pypi- prefix
The pypi automation has learned to add a pypi- prefix to packages from
that ecosystem so make autospec look for that as well. Note the pypi_
being looked for is just our lowercase of pypi-.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-12-02 09:34:48 -08:00
William Douglas 36c607ad12 specfiles: correct pyproject installation
The pyproject installs are intended to use pip to install but pip
installs need to avoid adding dependencies or caring about what is
currently installed on the system.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-11-23 09:27:20 -08:00
William Douglas cbe6aa1eae buildreq: Fix python_req_in_filtered_path
The python_req_in_filtered_path function was missing skips where the
directory was the last thing in the dirpath (as it matched an ending
slash and there isn't one in that case).

Since we need the full path to the item to scan for dependencies, make
that path first. Then use the full file path as the path to filter
because it will always have an ending slash for the skip portion.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-11-23 09:27:20 -08:00
William Douglas f20cc8a1d3 Add missing python requirement symbol scrubber
Signed-off-by: William Douglas <william.douglas@intel.com>
2021-11-22 08:46:31 -08:00
William Douglas 91e04be03f buildreq.py: More python requirement detection work
The previous changes to the build/runtime requirement detection for
python added a few too many spurious dependencies. This change adds a
few more directories to exclude from searching and makes pyproject
files primary over setup.py files.

Finally we stop parsing requirements.txt and requires.txt files once
we encounter the first '[' as most times the content we care about for
runtime is listed first.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-11-19 15:41:13 -08:00
William Douglas a19a557d53 Revamp autospec python dependency management
This migrates python dependencies (both build and runtime) to use
'pypi()' where possible (when detecting dependencies from setup.py,
setup.cfg, pyproject.toml, requires.txt and requirements.txt).

It also adds a few little cleanups to python detection in order to
make the above work more evenly and removes some python2 cruft.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-11-18 13:45:55 -08:00
William Douglas 416efdca94 buildreq: Use pip resolves for pyproject patterns 2021-11-09 23:28:58 -08:00
William Douglas c3a5bbf607 buildreq: Always ban setuptools_scm[toml]
This is a pyprojectism that we don't need to worry about, we'll always
have the toml bits needed with our python version.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-11-09 22:10:59 -08:00
William Douglas 46573f1035 Add pyproject build_pattern
Enable building projects without setup.py and with a pyproject.toml.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-11-09 15:02:26 -08:00
William Douglas 5e05989bf0 Add make_command override file
Enable using an alternate command instead of make. This is useful as
some projects are enabling the use of ninja with cmake for instance.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-11-09 15:02:06 -08:00
William Douglas 644706645e cargo build changes
Rework build cargo build pattern to no longer include sources and hack
the cargo config as the goal isn't to package dependencies at this
time.

Also reworks handling of install as cargo install is no longer working
for the buildroot install.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-10-25 11:41:05 -07:00
William Douglas 281e89833a Add build_append_once configuration support
Include the ability to have a build_append_once configuration that is
only run once at the top of the build section (before the
build_append, which has gotten a documentation update).

This is motivated by some packages wanting their base build to be
slightly different than the alternate builds.

It is possible instead of this change to instead move to having these
special builds be handled entirely with configuration files or using
another package.

Modified-by: William Douglas <william.douglas@intel.com>
2021-10-20 16:39:14 -07:00
William Douglas 9dfeae395a flake8 fixes for line length 2021-10-20 16:39:14 -07:00
William Douglas 448916c193 Run pypi dependency version override fix over the buildroot too
In cases where the requires.txt is generated, the dependency version
override will need to be run over the buildroot and not just the
source root.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-10-20 16:00:54 -07:00
William Douglas 668822df6c Add pypi module version overrides
Enable selectively removing version dependencies for a given set of
python modules. The motiviation for this change is to avoid breaking
package builds when there are packages that often lag pypi in updating
their max version requirements.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-10-19 13:25:35 -07:00
William Douglas db39793d81 config: Avoid shadowing local variable with loop temporary
Signed-off-by: William Douglas <william.douglas@intel.com>
2021-10-19 13:25:35 -07:00
William Douglas 311ba1f822 config: Add missing class variable initialization
Signed-off-by: William Douglas <william.douglas@intel.com>
2021-10-19 13:25:35 -07:00
Arjan van de Ven 9120dc23f7 set the linker flags to get the right identity to our binaries 2021-10-19 13:25:09 -07:00
William Douglas b206fd6a9b Add translation for BSD-3-Clause 2021-10-19 10:59:30 -07:00
William Douglas 37ef2f55df Revert "specfiles: Add new package files to subpackages"
This reverts commit 124ee37874.

Turns out this isn't quite correct for a few cases and doesn't fix the
issue well enough to continue to fix. Moving to a solution with
mixer-tools instead.

Signed-off-by: William Douglas <william.douglas@intel.com>
2021-10-14 15:44:31 -07:00
William Douglas 124ee37874 specfiles: Add new package files to subpackages
Useful to track what packages are installed on the system if the rpm
db isn't being used.
2021-10-14 14:46:39 -07:00
Arjan van de Ven fb197fc176 move the elf-move run to after install_append so that install_append can fix up messes correctly 2021-10-12 16:03:55 -07:00
Arjan van de Ven b5655686b9 the R world is fully AVX so no need for vzeroupper 2021-10-12 14:45:07 -07:00
Arjan van de Ven 650a564677 one more pattern 2021-10-12 14:45:07 -07:00
Arjan van de Ven e9517c8d87 one more file category 2021-10-10 23:42:38 -07:00