The buildpattern is a bit of an odd standalone module that was a
vestiage of the pre-refactor days where modules owned global
data. These days are coming to a close so move this data to a more
sensible owner in config.
This resolves#612 though likely more will be done as part of #131.
The download code for licenses, tarballs, and signature files are
similar enough to warrant consolidation into a unified interface, so add
a new function do_curl() that satisfies the needs of all the callers.
Note that I made one change that noticeably changes the output,
hopefully to reduce confusion: The signature download loop no longer
prints error messages, since the end user should not need to care at all
unless a signature cannot be downloaded and verification is required. If
the latter case is true, autospec will still exit and print the fatal
error, so the issue should be straightforward to diagnose after seeing
that error.
A couple of bugs are fixed as a result of using the new function:
- The FAILONERROR pycurl option is now set for tarball downloads. This
prevents certain undesirable side-effects like downloading 404
response pages.
- Responses from FTP servers are now handled better, since a more
common "success" code from FTP servers is 226, not 200.
And unit tests have been updated:
- Added new tests for download.do_curl().
- Replaced "mock" import with "unittest.mock" for pkg_integrity tests.
- Revised license and pkg_integrity tests to reflect the new download
paths.
- Added missing mocks for the download paths in PyPIVerifier and
GEMShaVerifier.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Because the Python standard library provides a package named "test",
pep8 thinks autospec is using it, but really it's importing
autospec/test.py. Rename test.py to check.py to avoid this issue and
tidy up the imports more.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Update autospec to be largely flake8 clean with 2 exceptions added to
the Makefile. One for Module docs that autospec is intentionally not
including as the file splits aren't real modules at this point. And
the second for import groups needing newline separation that autospec
is ignoring to avoid splitting up each autospec "module" being
imported as autospec today isn't really built on modules the python
way.
configparser.ConfigParser attempts to interpolate "%" characters in the
values of the key-value pairs when parsing. This is absolutely unwanted
behavior, especially when parsing URLs such as
<sourceforge url>/joe-editor/files/JOE%20sources/<joe tar>
which cause an uncaught exception.
Turn off interpolation by passing the interpolation=None argument to
ConfigParser.
Add a unit tests to report when ConfigParser(...) is called in the
codebase without setting interpolation to None.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This test suite is extremely hard to keep up-to-date and is no longer as
valuable now that unit tests have been written for much of the code
base. Actual functional testing can be done against package repos
without requiring them to be included in the autospec repository.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Create 3 separate infile files, the infile_handler reads in the
inputs and determnes how to parse them. The --infile argument now
allows a url, file, or directory of files to be passed as the
input. The infile_handler determines that type of input, calls the
correct parser depending on the filetype, and calls the update to
the specfile.
The infile parsers contains the scraping and parsing for all file
types. Currently it scrapes both .bb and .inc files and stores the
data into one dictionary. Because these files are both for recipes
their data corresponds and the .inc file takes presedence.
The infile update spec file udpates aspects of the specfile that
have been scraped and stored into a dict from the infile parsers
file. If there exists a value for ROS_SPN in the .bb file, use that
when replace the ROS_SNP value in the replace PV function.
Signed-off-by: Gabi Beyer <gabib@live.com>
Add --infile argument that takes a url, path, or directory
to a bitbake/recipe file. It scrapes the .bb/.inc file for
configuration data and stores it in a dictionary. This
architecture can be used for multiple file formats.
Also adds some initial unit tests for scraping functionality.
Signed-off-by: Gabi Beyer <gabib@live.com>
The previous ignore, import not at top of file (E402) was unneeded. Bare
excepts are handy when we want to fail gracefully from any error. This
is a script, not an imported library.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
This allows python3 unittest to properly generate the tests when running
all the tests in the tests directory.
Also adds a fix to test_test.py to reset buildpattern.default_pattern
every test. This was a bug introduced by running all tests at once -
this was not being cleared correctly.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
The unittest discover method did not run any code within the
if __name__ == "__main__": guard, which prevented it from picking up
generated tests for some of our large pattern-based tests. Just run them
manually so we can actually see all the tests being run.
Add unit tests for the very large count.py module/script. These unit
tests attempt to test every regular expression present in the parse_log
function and helped catch several bugs in the original regexes ported
from perl.
Unit tests added for the specdescription module. At the time of this
commit, there are two failing tests due to bugs in the module.
Failures:
- test_description_from_pkginfo
This test fails because a string.find() method is checked as a boolean
as if find returned 0 if the substring was not present. find()
actually returns -1 in this case. Because find() was being used as a
guard to find the end of a section, the end was found immediately and
the section was not parsed.
- test_description_from_spec_no_info
This test fails due to the default description being overridden, even
when the override is just an empty string.
The complexity of the buildpattern has been significantly reduced with
the move to a centralized specfiles class to handle specfile writes.
This patch tests the remaining functionality and fixes a typo in
buildpattern.py (pattern_strengh -> pattern_strength).
Several changes to the license module including changes to where local
hashes are stored. Tests were added as well. Detailed changes below.
config:
- Move the license hashes from an internal repo to a config file
(license_hashes) in autospec.
test_license:
- Unit tests added
- Unit test target added to autospec Makefile
- COPYING_TEST file added with GPL-3.0 text for test purposes
license:
- scan_for_licenses: separate out search strings from conditional to
make the code more readable and remove unused parameter (required a
change to where autospec.py called the function).
- remove duplicate licenses from license_translation list.
- convert license_blacklist dict to list (values were unused, it was
being treated as a list anyways).
- separate out search strings from code in license_from_copying_hash
and license_from_doc.
- move curl code out of `with open(file) as fname:` clause in
license_from_copying_hash.
The files module was being treated like a class, in that it was
necessary to keep state throughout the entire autospec run. Convert the
files module to a class and make some other general clean ups. This
changed how the files module was instantiated and called by the main
autospec module and other modules such as tarball and config.
Test the files module with unit tests and add the target to the
makefile.
Tests most helper functions in the abireport module. Excludes some
helpers that are just wrappers for os and shutil calls. Also excludes
the 100-line examine_abi main method, which may require functional
tests.
Functional tests take a long time to run, some longer than others. This
adds the option to run specific test cases defined by passing
whitespace-separated options to the '-c' flag, allowing the tester to
avoid long-running tests like libjpeg-turbo if they don't need to. This
can be run with make by setting the CASES argument:
make test_autospec CASES="c-helloworld c-helloworld-32"
Since pkg_integrity now depends on config.py, it is necessary to change
how imports are done in the tests so mocking can be achieved. This also
required renaming the tests/pkg_integrity.py file to
tests/test_pkg_integrity.py to avoid a name collision. Before each test
the necessary config option is set correctly and the configuration file
rewrite is mocked away.
Instead of using the date the configure file was modified, use the UNIX
epoch time for the SOURCE_DATE_EPOCH environment variable. This allows
reproducible package binaries per autospec build.
Functional tests updated to match change and 'unittest' make target
added to ease testing.
Includes a rename of specfile.py -> specfiles.py for easier imports and
related change to how it imported in autospec.py. Also includes a minor
fix to how lang files are written to the specfile. The error was
identified by these unit tests.
This unit test reads test cases from tests/packageurls and attempts to
determine the version and name from the url specified. tests/packageurls
defines these cases and their expected results in the format:
<package url>,<expected name>,<expected version>
There are currently some test cases that fail. This unit test will make
it easier to make changes to the tarball.name_and_version() function.
This involves moving the tests to the top-level tests/ directory, adding
a Makefile target (make test_autospec) to run the functional tests, and
adding a tests/README.rst to explain the various testing files and test
structure.
Signature verification was extended to do hash
validation for ruby gems, so integrity validation
is a better description of module. The module
was changed from verify_sign to pkg_integrity
autospec code was moved into autospec/ folder and current
Makefile does not check it, this commit changes the check
target to run flake8 in autospec/*.py instead of ./*.py
Patrick McCarty