Given more packages are using ninja as the build system of choice
instead of make, add flag to enable ninja usage.
Signed-off-by: William Douglas <william.douglas@intel.com>
Previously support for cmake's find_package parsing was only able to
handle single line entries. This change adds support for multiline
versions and allows for module namespacing (currently qt6 and kf6
support is available).
The change for cmake_modules shows this namespacing in use.
Signed-off-by: William Douglas <william.douglas@intel.com>
Previous code would emit:
SyntaxWarning: invalid escape sequence
Correct this by removing the escapes as they are not necessary.
Also add a missing buildreq that is detected correctly after the
fix.
Signed-off-by: William Douglas <william.douglas@intel.com>
Some packages should not be able to be added to any subpkg. Add
handling for this case.
Also as pypi-nose is no longer usable so ban it from being added as a
dependency.
Signed-off-by: William Douglas <william.douglas@intel.com>
When autospec find a requires.txt, set the requirements_path not the
requires_path (renamed to a more fitting pyproject_path). Also fix up
unittests and add a new one for this case.
Signed-off-by: William Douglas <william.douglas@intel.com>
Reorder some items for build pattern detection so that build
requirements for specific patterns are added last. This is to help
prevent build requirements that don't apply to a given pattern from
being added unintentionally.
As part of this remove some outdated bits (catkin and disutils36) that
no longer are used.
Signed-off-by: William Douglas <william.douglas@intel.com>
These patterns are being removed as they are not used or are going to
be reworked.
This change also removes support for multi-versioned packages which
were considered for go and cargo purposes.
Signed-off-by: William Douglas <william.douglas@intel.com>
Add support for (*_)provides_{add,ban} configuration files.
This is occasionally useful for cases where a pypi() provide isn't
picked up as the packages isn't a python package but does provide a
python library. It also can be handy when doing some renames to
prevent mass rebuilds.
Signed-off-by: William Douglas <william.douglas@intel.com>
If a python requirement has a python_version bounded by a maximum
(python_version < 3.8) than ignore it. The reasoning for this change
is to avoid dependencies that no longer apply with the most recent
python version.
Signed-off-by: William Douglas <william.douglas@intel.com>
This migrates python dependencies (both build and runtime) to use
'pypi()' where possible (when detecting dependencies from setup.py,
setup.cfg, pyproject.toml, requires.txt and requirements.txt).
It also adds a few little cleanups to python detection in order to
make the above work more evenly and removes some python2 cruft.
Signed-off-by: William Douglas <william.douglas@intel.com>
Autospec's DESCRIPTION file parsing for CRAN packages is working well
enough that I think we can unconditionally add the detected requirements
as both build and runtime dependencies.
Also remove the propagation of CRAN build->runtime deps via fail
pattern, since we no longer need it; the deps detected via fail pattern
are either Suggests (optional) or not listed in DESCRIPTION at all.
Either way, they are not required dependencies.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
When parsing setup.py files, if already in multiline and install_requires
is used as a variable, don't try and treat the line as if it was an
install_requires assignment.
Signed-off-by: William Douglas <william.douglas@intel.com>
Due to more systems being added for automated runtime dependency
injection for subpackages, we need a way for manual overrides of the
default detected dependencies.
This change provides support for specifying which subpackage runtime
dependencies should be added or removed and consolidates some of the
custom configuration file detection along with it.
The buildpattern is a bit of an odd standalone module that was a
vestiage of the pre-refactor days where modules owned global
data. These days are coming to a close so move this data to a more
sensible owner in config.
This resolves#612 though likely more will be done as part of #131.
Many different components require the target download path and it is
needed early. Previously code existed to attempt to autodetect its
location and so defining the path was enforced later in the build
process. This created problems with ordering functions based on this
requirement but since the target path is now certain to be defined at
startup (due to previous changes) we can better align the location and
initialization of the data to the Config class.
As part of this change also avoid requiring the temporary working
directory to be setup as part of the Build class. This is done as a
cleanup due to the only real user of this location being the Content
class.
This also drops the stand-alone pkg_integrety main function that was
initially used as a one off test to avoid needing to run autospec to
validate the integrity of a package. This feature was seldom used and
doesn't function well without handling configuration so it was
dropped.
The buildreq module had a large number of globals being touched across
many modules that would import it. This made changes to the buildreq
fragile.
It also made testing fragile due to certain globals needing to be
reset in order to properly test a change.
This change moves the global state (and functions that needed to act
on that global state) into a Requirements class. The goal of this work
is to better track what content can be updated by a particular
function and load data in a sensible order and have it owned by a
sensible component.
The config module had a large amount of globals that were being
touched across many modules that would import. This made changes to
config very fragile as figuring out what would be modified in any
given call chain was difficult to diagnose.
It also made testing fragile as one would need to reset a given
module's config import to the best of their knowledge before rerunning
another test.
To get away from that (and to try and reduce the number of globally
modified variablies in autospec as a whole), refactor the config
module to provide its state as part config class. The long running
goal of changes like this is to better track what content can be
updated by a particular function (if a function would have access
to the config instance is now the hint rather than the config module
getting imported).
Get metadata about the package from pypi for distutils3 build patterns
then use that metadata to get summary, description and provides
information (license and requires maybe leveraged in the future).
The description isn't always very useful (often too long) so only use
it if there isn't one already.
Update support for handling go packages by handling missing go
dependencies. Make use of go modules and the go proxy (see
https://tip.golang.org/cmd/go/#hdr-Module_proxy_protocol for more
details on the proxy implementation) to discover what artifacts are
needed.
A new go-dep build_pattern that will package all versions that the
go-proxy finds is also added as part of this change. Additional rework
of the golang build_pattern (and Makefile build_pattern as a number of
go packages use Makefiles) to use module vendoring when module
configuration is detected was added as well.
The new functionality is also intended to be used by a higher level
tool that will wrap autospec calls to build missing dependencies as
well as the requested package.
Add additional logic to handle: "pkg ; dep <= 'X'", style statements
for python requirements in setup.py. Also update tests to validate the
results are as expected.
If a python package is python3-only and has a setup.py,
"distutils23" will be used as build pattern even if the project is
explicitely specified as python 2 or python 3 only.
This change tries to find if either of these are the case by
looking at the classifiers in the setup.py file for the following
string:
"Programming Language :: Python :: 3 :: Only"
"Programming Language :: Python :: 2 :: Only"
The default is still "distutils23" unless either of these strings
are found. If they are "distutils2" or "distutils3" will be used
corresponding to the version in the classifier.
Signed-off-by: Kevron Rees <kevron.m.rees@intel.com>
Improve rust/cargo support by:
1) Better handling only toplevel Cargo configuration files for dependency
lookup.
2) Adding runtime requirements for buildreq packages in order to get
dependencies from non directly dependent packages resolved.
3) Add cargo as a buildreq for rust packages to handle the rustc/cargo
split.
4) Add cargo detection from failed_commands.
5) Fix rust source directory destination to not append version in case
the package name already had version information in it.
6) Get name and version information from crates.io package files.
Minor clean up of buildreq module, including the following changes:
- Remove unused import "tarball"
- Add function docstrings
- Convert to list-driven pattern checks instead of a long list of
conditionals.
- Rename variable names that were overriding keywords (file, buffer,
etc.)
- Remove unused setup_py_python3 function
- Remove unused arguments and variables from scan_for_configure function
(and corresponding function calls in test_buildreq.py and autospec.py)
Most changes made at pylint and pycodestyle suggestions.
Shortcomings were identified in setup.py parsing for python build
requirements. These tests more exhaustively test the cases we have been
seeing in the wild.
This patch makes some simplifications to the buildreq module and related
modules (autospec and config). These are mostly readability changes and
code cleanup.
William Douglas