A (target [0]) package can independently declare installing in various
locations: target, staging, or images. The default is to only install
in target.
When a package opts out from installing to target, but does not opts
in to install in any other location, the package is not downloaded,
extracted, patched, configured, nor built at all. As a consequence, none
of the per-step instrumentation is executed, specifically the listing
of files before/after the package sequence.
Down the line, the package infra does not cope well with that situation,
because the gathering-install step, the one that synchronises all the
optional target, staging, or images install steps, still gets run.
And as #13836 shows, this does not go well:
/bin/sh: /home/tbuild/myboard/build/foo/.files-list.after: No such file or directory
make[1]: *** [/home/tbuild/myboard/build/foo/.stamp_installed] Error 1
make: *** [_all] Error 2
So, we should have ensured that the gathering-install step itself
depends on the build step, which would have solved the issue.
However, this bug really illustrates a more fundamental issue: does it
even make sense to have a package that installs nothing in any location?
Indeed, why even bother with that package to begin with if it will not
provide anything at all?
It turns out that yes, this makes sense. We have some packages, that
do not install anything at all, and do not even build anything; they are
there just to ensure that we can download something that will ultimately
be used by another package. This is the case for example for packages
that provide linux extensions, like aufs [1].
Additionally, some ugly out-of-tree packages could conceivably install
things during the build (or even configure!) steps. That's not unheard
of... [2]
So, the solution is to ensure that the gathering-install step does
depend on the build step, to trigger the proper dependency chain and
have the instrumentation hooks properly run even in that degenerate
case.
Fixes: #13836
[0] a host package can't opt out of installing anything.
[1] that one is actually missing AUFS_INSTALL_TARGET = NO, so this
hides the issue.
[2] even us are not 100% clean on that topic: gcc will install files in
staging and target as part of the same step (not the build, granted,
but still...)
Reported-by: "Weber, Matthew L Collins" <Matthew.Weber@collins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Matthew Weber <matthew.weber@collins.com
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
When a package has both a target and a host variant, and uses git
submodules, and the host variant is downloaded before the target one, we
end up with the generated archive missing the submodules.
This happens in exactly one package in our tree: c-capnproto.
This issue was not caught before because after a few days, the full
sources are added to sources.buildroot.net. So when the hash check
fails, the full tarball is simply downloaded from there.
Propagate the git submodule setting from the target variant to the host
variant, unless the host variant explicitly opted-out.
Fixes:
http://autobuild.buildroot.org/results/2de9c6c8ce83569d18cc7140ebc60d6fe1aadcbf/
Reported-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
The .tar.gz default extension is historical, and we initially used
to only fetch tarballs from remote sites.
When we introduced downloads from VCS repositories, we kept that
extension, and kept compressing with gz, by lack of good reason to
switch to some other compression scheme.
However, nowadays, we will want to change the way we construct the
tarballs we generate from VCS. This will de facto change the hashes
of those tarballs.
So we will want that the archives we generate do not clash with the
existing ones, so we need another filename. Thus, we need a way to
be able to use a different extension when we generate archives from
VCS.
Use a macro as suggested by Arnout.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Vincent Fazio <vfazio@xes-inc.com>
Acked-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
Currently, the match between Buildroot packages and CVEs is solely
based on the package names. Unfortunately, as one can imagine, there
isn't necessarily a strict mapping between Buildroot package names,
and how software projects are referenced in the National Vulnerability
Database (NVD) which we use.
The NVD has defined the concept of CPE (Common Platform Enumeration)
identifiers, which uniquely identifies software components based on
string looking like this:
cpe:2.3:a:netsurf-browser:libnsbmp:0.1.2:*:*:*:*:*:*:*
In particular, this CPE identifier contains a vendor name (here
"netsurf-browser"), a product name (here "libnsbmp") and a version
(here "0.1.2").
This patch series introduces the concept of CPE ID in Buildroot, where
each package can be associated to a CPE ID. A package can define one
or several of:
- <pkg>_CPE_ID_VENDOR
- <pkg>_CPE_ID_PRODUCT
- <pkg>_CPE_ID_VERSION
- <pkg>_CPE_ID_VERSION_MINOR
- <pkg>_CPE_ID_PREFIX
If one or several of those variables are defined, then the
<pkg>_CPE_ID will be defined by the generic package infrastructure as
follows:
$(2)_CPE_ID = $$($(2)_CPE_ID_PREFIX):$$($(2)_CPE_ID_VENDOR):$$($(2)_CPE_ID_NAME):$$($(2)_CPE_ID_VERSION):$$($(2)_CPE_ID_VERSION_MINOR):*:*:*:*:*:*
<pkg>_CPE_ID_* variables that are not explicitly specified by the
package will carry a default value defined by the generic package
infrastructure.
If a package is happy with the default <pkg>_CPE_ID, and therefore
does not need to define any of <pkg>_CPE_ID_{VENDOR,PRODUCT,...}, it
can set <pkg>_CPE_ID_VALID = YES.
If any of the <pkg>_CPE_ID_{VENDOR,PRODUCT,...} variables are defined
by the package, then <pkg>_CPE_ID_VALID = YES will be set by the
generic package infrastructure.
Then, it's only if <pkg>_CPE_ID_VALID = YES that a <pkg>_CPE_ID will
be defined. Indeed, we want to be able to distinguish packages for
which the CPE ID information has been checked and is considered valid,
from packages for which the CPE ID information has never been
verified. For this reason, we cannot simply define a default value
for <pkg>_CPE_ID.
The <pkg>_CPE_ID_* values for the host package are inherited from the
same variables of the corresponding target package, as we normally do
for most package variables.
Signed-off-by: Matt Weber <matthew.weber@rockwellcollins.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Matt Weber <matthew.weber@rockwellcollins.com>
Reviewed-by: Heiko Thiery <heiko.thiery@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Allow packages to have an 'selinux' subfolder containing SELinux modules
(sources) to be synced and compiled within the refpolicy, if the package
is selected.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Add support for packages to enable SELinux modules already supported by
the refpolicy, but not selected by default in its policy.
With this commit, packages will be able to do something like:
SYSTEMD_SELINUX_MODULES = systemd udev
to enable additional SELinux modules.
Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Virtual packages are not built but it's useful to allow them to have
permission/device/user tables and target-finalize/rootfs-pre-cmd hooks.
With this change we don't need to duplicate data as currently done in
eudev and systemd user tables, or test if the package is selected to add
a target-finalize hook as in luainterpreter.
Signed-off-by: Carlos Santos <unixmania@gmail.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
These functions are no longer using the GLOBAL_INSTRUMENTATION_HOOKS
mechanism, so it doesn't make much sense for them to be in the section
of pkg-generic.mk related to those hooks.
Move them to the "Helper functions" section.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Now that we are checking the host directory changes throughout all
installation steps and not just during the "host installation step",
it means that changes done within the staging directory (which is a
subdir of the host directory) are also visible in the
.files-list-host.txt file.
Note that this problem already potentially occurs if a host package is
installing files in the staging directory: they would be listed in
.files-list-host.txt even without the changes in this series.
To fix this up, we simply exclude files that are beneath the
$(STAGING_SUBDIR). Note that we do that in all cases, so when
searching $(HOST_DIR), $(HOST_DIR)/$(STAGING_SUBDIR) is excluded, but
when searching $(TARGET_DIR), $(TARGET_DIR)/$(STAGING_SUBDIR) is
excluded, and when search $(STAGING_DIR),
$(STAGING_DIR)/$(STAGING_SUBDIR) is excluded. This is not a problem in
practice since $(TARGET_DIR)/$(STAGING_SUBDIR) and
$(STAGING_DIR)/$(STAGING_SUBDIR) don't exist, but it's not very
nice. However, it allows to keep the code simple.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
This commits reworks the pkg_size logic to no longer use the
GLOBAL_INSTRUMENTATION_HOOKS mechanism, but instead be directly
implemented within the configure step and install step.
The problem with the current implementation in the
GLOBAL_INSTRUMENTATION_HOOKS is that we only capture what is installed
in $(HOST_DIR) during the "host installation step", what is installed
in $(TARGET_DIR) during the "target installation step" and what is
installed in "$(STAGING_DIR)" during the staging installation step.
While this seems reasonable in principle, it is in fact not completely
true. For example, "toolchain", which is a target package, installs
tons of files in $(HOST_DIR). "qt5base", which is also a target
package, also installs things in $(HOST_DIR). Same with the "syslinux"
package.
The idea behind this patch is pretty simple:
- At the beginning of the configure step, right after the per-package
directories have been populated (if BR2_PER_PACKAGE_DIRECTORIES=y),
we capture the state of the HOST_DIR, TARGET_DIR and STAGING_DIR.
- At the end of all install steps (which is possible thanks to the
newly introduced "install" step), we capture again the state of
HOST_DIR, TARGET_DIR and STAGING_DIR, and compare it to what we
have saved at the configure step.
So regardless of whether a file has been installed in $(HOST_DIR)
during the target or staging installation steps of a target package,
or if a host package has installed a file in $(TARGET_DIR), we will
detect it.
The pkg_size_before and pkg_size_after macros are intentionally left
where they are (even if they now fall in the middle of the
GLOBAL_INSTRUMENTATION_HOOKS implementations) to minimize the diffstat
and facilitate review.
Note that we also have to change check_bin_arch to be explicitly
called from the install step rather than through a
GLOBAL_INSTRUMENTATION_HOOKS as it depends on the .files-list.txt file
produced by the pkg_size_after function.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
We currently create HOST_DIR, TARGET_DIR, STAGING_DIR and BINARIES_DIR
in their respective installation steps. However, as we are about to
change how the logic to capture files installed by packages is
implemented, we will need these directories to exist at the configure
step to keep things simple.
Note that when BR2_PER_PACKAGE_DIRECTORIES=y, the HOST_DIR, TARGET_DIR
and STAGING_DIR are anyway already created at the configure step, when
populating the per-package HOST_DIR and TARGET_DIR. This also means
that we can drop the "mkdir" from per-package-rsync.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr:
- create directories after MESSAGE
- use $(Q) not @
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
We currently have four different install steps: target installation,
staging installation, images installation and host installation. These
steps are directly triggered from the $(1)-install make target, so
there is no place where we can run some logic once all installation
steps have completed.
However, as part of improving the reliability of the logic done in
step_pkg_size_before and step_pkg_size_after to detect the files
installed by packages, we would in fact need to run some logic after
all installation steps have completed. This will allow us to make sure
that all files are detected, even if a host package installs something
in the target directory, or if a target package installs something in
the host directory.
To achieve this, this commit implements a new stamp file,
.stamp_installed, which is a step that depends on all four install
steps. Currently, this step does nothing except creating the stamp
file.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: remove stampfile on foo-reinstall]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The $(1) argument passed to step_pkg_size_{before,after}, which
contains the package name, is no longer used. We simply use $(PKG) to
get the upper-case version of the package name.
So, let's drop this first argument that isn't needed.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Since we're already using $(PKG)_DIR in step_pkg_size_after, we can
also just use $(PKG)_NAME. This allows to make $(1) useless, which
means it can be dropped in a follow-up commit.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Currently, the linux kernel will apply some fixups on its .config file,
based on whether some packages are enabled or not. That list of
conditional fixups is getting bigger and bigger with each new package
that needs such fixups, culminating with the pending firewalld one [0].
Furthermore, these fixups are not accessible to packages in br2-external
trees.
Add a new per-package variable, that packages may set to the commands to
run to fixup the kernel .config file, which is added at the end of the
linux' own fixups.
This opens the possibility to write things like;
define FOO_LINUX_CONFIG_FIXUPS
$(call KCONFIG_ENABLE_OPT,BLA)
endef
Of course, it also opens the way to run arbitrary commands in there, but
any alternative that would be declarative only, such as a list of
options to enable or disable (as an example):
FOO_LINUX_CONFIG_FIXUPS = +BAR -FOO +BUZ="value"
.. is not very nice either, and such lists fall flat when a value would
have a space.
For packages that we have in-tree, we can ensure they won't play foul
with their _LINUX_CONFIG_FIXUPS. For packages in br2-external trees,
there's nothing we can do; users already have the opportunity to hack
into the linux configure process by providing LINUX_PRE_CONFIGURE_HOOKS
or LINUX_POST_CONFIGURE_HOOKS anyway...
.. which brings the question of why we don't use that to implement the
per-package fixups. We don't, because _PRE or _POST_CONFIGURE_HOOKS are
run after we run 'make oldconfig' to sanitise the mangled .config.
[0] http://lists.busybox.net/pipermail/buildroot/2020-March/278683.html
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Adam Duskett <aduskett@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
By their very nature, host packages have no target, staging, or images
install steps; they have a single install step, that is always
performed.
As such, setting the corresponding _INSTALL_{TARGET,STAGING,IMAGES}
variables does not make sense for host packages.
However, people (and scripts) may get confused when they process the
output of printvars, e.g.:
$ make printvars VARS=HOST_LIBTOOL_INSTALL_TARGET
HOST_LIBTOOL_INSTALL_TARGET=YES
Only set those variables for target packages. There is no
corresponding variable for host packages, as they are always installed
(and only once).
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Vadim Kochan <vadim4j@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: eeppeliteloop@gmail.com
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
After the staging installation, we replace a number of paths in libtool
.la files so that those paths point to STAGING_DIR instead of a location
in the build machine.
However, we replace only paths that start with /usr. And it turns out
that the linux-pam package is configured with --libdir=/lib (linux-pam
seems to always be installed in /lib rather than /usr/lib).
Due to this, libpam.la contains the following line:
libdir='/lib'
When building a configuration that has:
- BR2_ROOTFS_MERGED_USR=y
- BR2_PACKAGE_LINUX_PAM=y
- BR2_PACKAGE_POLKIT=y
on a system that has its system-wide PAM library installed in /lib,
the build fails with:
/lib/libpam.so: file not recognized: File format not recognized
For some reason, libtool searches only in STAGING_DIR/usr/lib, but
when BR2_ROOTFS_MERGED_USR=y, STAGING_DIR/lib points to
STAGING_DIR/usr/lib, so libtool finds libpam.la. And this libpam.la
contains a bogus libdir='/lib' path. libtool then goes on, finds
/lib/libpam.so, and links with it, causing the build failure.
By doing the proper replacement of libdir='/lib', we have a correct
libpam.la, and solve the build issue.
There is no autobuilder failure associated to this issue, as it
requires /lib/libpam.so to exist. This is the case on ArchLinux, on
which Xogium reported the issue, which can also be reproduced in an
ArchLinux container.
Reported-by: Xogium <contact@xogium.me>
Cc: Xogium <contact@xogium.me>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Tested-by: Yann E. MORIN <yann.morin.1998@free.fr>
[yann.morin.1998@free.fr:
- tested by manually creating a symlink to libpam.so in /lib
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
If a package sets a FOO_SUBDIR (meaning its sources are not under
output/build/foo-123 but under output/build/foo-123/$(FOO_SUBDIR)), the
.files-list.txt file were also created under FOO_SUBDIR, due to which the
logic in the Makefile would not find it.
Change the instrumentation steps so that the file list is directly under the
package dir, ignoring the subdir.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
In commit 0e2be4db8a
("package/pkg-generic: make file list logic parallel build
compatible"), the logic to create the list of files installed by a
particular package was significantly reworked to be compatible with
top-level parallel build.
Before this commit, there was only a after-install step of listing the
files in HOST_DIR/TARGET_DIR/STAGING_DIR. But after this commit, we
now have a before-install logic and an after-install logic.
It turns out that when the before-install logic is called for the very
first host package, $(HOST_DIR) doesn't exist yet, and therefore the
cd $(2) fails, with an error message:
/bin/sh: line 0: cd: /home/thomas/buildroot/output/host: No such file or directory
In fact, $(HOST_DIR), $(STAGING_DIR), $(TARGET_DIR) and
$(BINARIES_DIR) are created by the make rules for host installation,
staging installation, target installation and images installation, but
*after* calling the step_start hooks.
So, we simply fix this problem by creating the directories *before*
calling the step_start hooks.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
In all steps, we print the message indicating the start of the step
using the MESSAGE macro before running pre-hooks. Except in the image
installation step, where the message is printed after the pre-hooks.
Let's fix this inconsistency.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The current solution used to collect the list of files installed by
packages does not work for top-level parallel build. Indeed, we rely
on a file created after the installation of the previous package to
build the list of files installed by the current package.
This works well when packages are built sequentially, but badly fails
when using top-level parallel build.
More specifically, top-level parallel build can fail with:
comm: /home/thomas/buildroot/output/build/.files-list-host.new: No such file or directory
Because that file has been removed concurrently by the build process
of another package.
This commit reworks the logic in a very straight-forward way. Before
the installation of each package, we store the list of files that are
already installed and store it in the package build directory. After
the installation of each package, we store again that list of files,
calculate the difference with the before file, and store that as the
list of files installed by that package, still in the package build
directory.
At the end of the build, in target-finalize we collect all the
collected information into the global package file lists, that
continue to be installed in the same location as before, with the same
name.
There are however some differences:
(1) The files are no longer ordered in build order, but by alphabetic
ordering of packages. Indeed, "build order" no longer makes any
sense in the context of top-level parallel build.
(2) Some files which were incorrectly tracked are no longer
tracked. For example, the toolchain package is a target package,
but it installs files in $(HOST_DIR). In the previous logic, the
files installed by the toolchain package in $(HOST_DIR) were
incorrectly affected to the next host package that was installed
after the toolchain package. With our new logic, those files are
no longer tracked at all. To fix this, we would have to change
the logic to scan HOST_DIR/TARGET_DIR/STAGING_DIR for all
installation steps, not just for the install-host, install-target
and install-staging steps respecitively. But the result was
already incorrect anyway, and therefore this should be fixed
separately.
Note that the check_bin_arch hook needs to be adjusted: it was using
the global package-file-list.txt file, but this file is now created
only at the very end of the build. So instead, we use the current
package .file-list.txt file to know which packages have been installed
by the current package in $(TARGET_DIR).
Fixes:
http://autobuild.buildroot.net/results/4e60fa31b1cd08bc7fdf9c5dd3a3f4941e029ba3/
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Use the same trick in step_pkg_size as the one used in check_bin_arch:
factorize the two $(filter ...) calls into one, checking in one step
the step and whether it's the beginning or end of the step.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
The package instrumentation step 'step_pkg_size' is populating the files:
output/build/packages-file-list.txt
output/build/packages-file-list-staging.txt
output/build/packages-file-list-host.txt
by comparing the list of files before and after installation of a package,
with some clever tricks to detect changes to existing files etc.
As an optimization, instead of gathering this list before and after each
package, where the 'after-state' of one package is the same as the
'before-state' of the next package, only the 'after-state' is used and
is shared between packages.
This works fine, except at the end of the build, as explained next.
In the target-finalize step, many files will be touched. For example, files
like /etc/hosts, /etc/os-release, but also all object files that are
stripped, and all files touched by post-build scripts or created by rootfs
overlays. This means that the 'after-state' of the last package does not
reflect the actual situation after target-finalize is run.
For a single complete build this poses no problem. But, if one incrementally
rebuilds a package after the initial build, e.g. with 'make foo-rebuild',
then all changes that happened in target-finalize at the end of the initial
build (the 'after-state' of the last package built) will be detected as
changes caused by the rebuild of package foo. As a result, all these files
will incorrectly be treated as 'owned' by package foo.
Correct this situation by capturing a new state at the end of
target-finalize, so that the 'before-state' of an incremental build will be
correct.
Note: the reasoning above talks about packages-file-list.txt and
target-finalize, but also applies to
packages-file-list-staging.txt/staging-finalize and
packages-file-list-host.txt/host-finalize.
Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
For now, the extractor dependencies were only calculated for
<pkg>_SOURCE, so if the package manually downloads another file using
<pkg>_EXTRA_DOWNLOADS and then extracts it with $(call
suitable-extractor), we are missing the corresponding dependency on
the appropriate extracting tool.
Since the vast majority of <pkg>_EXTRA_DOWNLOADS are compressed files
that will be uncompressed at build time, it makes sense to derive the
corresponding extractor dependencies directly in the common package
infrastructure, rather than having each and every package using
<pkg>_EXTRA_DOWNLOADS making this effort.
On a system without xzcat, before this patch:
$ make printvars VARS=HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES
HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES=host-tar
After this patch:
$ make printvars VARS=HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES
HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES=host-tar host-xz
This commit most notably fixes the build of host-gettext-tiny on
systems without xzcat, and with per-package support enabled. Indeed,
the main _SOURCE for gettext-tiny is a .gz file, but it has a .xz file
in its _EXTRA_DOWNLOADS, which is then extracted. Except that xzcat
being missing from the dependencies, it is not built.
Fixes:
http://autobuild.buildroot.net/results/83c6d47c06334bef27791a59bdd491b1de124c49/
Suggested-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Instead of manually calculating the EXTRACT_DEPENDENCIES value based
on the archive extension, let's use the newly introduced
extractor-pkg-dependency macro.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
The extractor-dependency macro returns which system-provided tools are
needed to be able to extract the archive passed as argument. The
result of this macro is added to DL_TOOLS_DEPENDENCIES so that the
logic in support/dependencies/ verifies that the necessary tools are
provided by the system.
However, we are going to add another macro, extractor-pkg-dependency,
which says which Buildroot packages are needed to extract the archive
passed as argument. Indeed, for those archive types, if the extractor
is not provided system-wide, we build it as a host Buildroot package.
To clarify the distinction between the upcoming
extractor-pkg-dependency and existing extractor-dependency, we rename
the latter to extractor-system-dependency.
We take this opportunity to extend the documentation of this macro.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Reviewed-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Commit c4e6d5c8be ("core: implement
per-package SDK and target") had a mistake on the regexp that is used
to match $(PER_PACKAGE_DIR)/<something>/, and due to this, the regexp
was never matched.
The + sign in [^/]+ which was suggested by Yann E. Morin during the
review of the per-package patch series (instead of [^/]*) needs to be
escaped to be taken into account correctly. Without this, the regexp
doesn't match, and the replacement is not done, causing:
(1) For the libtool fixup in pkg-generic.mk, the lack of replacement
causes libtool .la files to not be tweaked as expected, which it
turn causes build failures reported by the autobuilder.
(2) For the fix-rpath, the RPATH of host binaries in the SDK were not
correct.
Interestingly, we have the same regexp in
support/scripts/check-host-rpath, but here the + sign does not need to
be escaped.
Fixes:
http://autobuild.buildroot.net/results/d4d996f3923699e266afd40cc7180de0f7257d99/ (libsvg-cairo)
http://autobuild.buildroot.net/results/56330f86872f67a2ce328e09b4c7b12aa835a432/ (bind)
http://autobuild.buildroot.net/results/9e0fc42d2c9f856b92954b08019b83ce668ef289/ (ibrcommon)
and probably a number of other similar issues
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Fixes:
http://autobuild.buildroot.net/results/3b6/3b6280b0b7a9634b747db2865b21c6266007c725/
The PYTHON_KEEP_PY_FILES global variable conflicts with the per-package
<pkg>_KEEP_PY_FILES variable for the python package, causing make to
complain:
package/zlib/zlib.mk:7: *** Recursive variable 'PYTHON_KEEP_PY_FILES' references itself (eventually). Stop.
As a workaround, rename the global variable to KEEP_PYTHON_PY_FILES so it
cannot conflict with the per-package variable.
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When BR2_PACKAGE_PYTHON{,3}_PYC_ONLY=y, we force remove all .py files
from the system, as they have all been byte-compiled into their .pyc
variants.
However, it turns out that some packages (e.g: OpenCV) do some funky
things with a few .py files: they pass them through Python's
execfile() facility, which only works with .py files and not .pyc
files. It is used by OpenCV for example to read two small
configuration files.
In order to support such use cases, this commit introduces a very
simple mechanism by which packages can exclude some path patterns from
the .py removal: a per-package <pkg>_KEEP_PY_FILES variable that is
collected into a global PYTHON_KEEP_PY_FILES variable, then used by
the python/python3 target-finalize hooks.
This variable is intentionally not documented, this is really a hack
that we ideally would like to see go away, and we'd rather not see its
usage spread too much.
This is necessary to be able to fix bug #12171.
[Peter: check if PYTHON_KEEP_PY_FILES contains non-white space]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Libtool .la files unfortunately contain a number of absolute paths,
which now refer to per-package directories. Due to this, when building
package A, .la files may contain absolute paths referring to
directories in package B per-package sysroot. This causes some -L
flags referring to other sysroot from being added, which doesn't work
as the linker no longer realizes that such paths are within its
sysroot.
To fix this, we introduce a replacement step of .la files in the
configure step, to make sure all paths refer to this package
per-package directory.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
This commit implements the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target directories, that only contain
their explicit dependencies.
There are two main benefits:
- Packages will now see only the dependencies they explicitly list in
their <pkg>_DEPENDENCIES variable, and the recursive dependencies
thereof.
- We can support top-level parallel build properly, because a package
only "sees" its own host directory and target directory, isolated
from the build of other packages that can happen in parallel.
It works as follows:
- A new output/per-package/ directory is created, which will contain
one sub-directory per package, and inside it, a "host" directory
and a "target" directory:
output/per-package/busybox/target
output/per-package/busybox/host
output/per-package/host-fakeroot/target
output/per-package/host-fakeroot/host
This output/per-package/ directory is PER_PACKAGE_DIR.
- The global TARGET_DIR and HOST_DIR variable now automatically point
to the per-package directory when PKG is defined. So whenever a
package references $(HOST_DIR) or $(TARGET_DIR) in its build
process, it effectively references the per-package host/target
directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
is handled as well.
- Of course, packages have dependencies, so those dependencies must
be installed in the per-package host and target directories. To do
so, we simply rsync (using hard links to save space and time) the
host and target directories of the direct dependencies of the
package to the current package host and target directories.
We only need to take care of direct dependencies (and not
recursively all dependencies), because we accumulate into those
per-package host and target directories the files installed by the
dependencies. Note that this only works because we make the
assumption that one package does *not* overwrite files installed by
another package.
This is done for "extract dependencies" at the beginning of the
extract step, and for "normal dependencies" at the beginning of the
configure step.
This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:
- The filesystem image creation code cannot work.
- We don't have a SDK to build code outside of Buildroot.
In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.
This change to per-package directories has an impact on the RPATH
built into the host binaries, as those RPATH now point to various
per-package host directories, and no longer to the global host
directory. We do not try to rewrite such RPATHs during the build as
having such RPATHs is perfectly fine, but we still need to handle two
fallouts from this change:
- The check-host-rpath script, which verifies at the end of each
package installation that it has the appropriate RPATH, is modified
to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is
a correct RPAT.
- The fix-rpath script, which mungles the RPATH mainly for the SDK
preparation, is modified to rewrite the RPATH to not point to
per-package directories. Indeed the patchelf --make-rpath-relative
call only works if the RPATH points to the ROOTDIR passed as
argument, and this ROOTDIR is the global host directory. Rewriting
the RPATH to not point to per-package host directories prior to
this is an easy solution to this issue.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Just like _SITE, _SOURCE, _SITE_METHOD, it is very likely that if
<pkg>_DL_OPTS is defined, the same value should be used for
HOST_<pkg>_DL_OPTS, so let's have the same inheritance logic than the
one we have for other variables.
Fixes:
https://bugs.busybox.net/show_bug.cgi?id=12321
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
When a package specifies extra downloads, it has the option to only name
the basename of the extra download, in which case that extra download
will be retrieved from the same location the main download is retrieved
from.
In that case, if the extra download contains a '+', it would confuse the
dl-wrapper, which believes the LHS of the '+' is the site method, and
the RHS the actual URI, and so the dl-wrapper mangles and damages the
URI when fetching such extra downloads, like that happens with android
tools, where the proper URI and mangled URIs of the extra download are,
respectively:
https://launchpad.net/ubuntu/+archive/primary/+files/android-tools_4.2.2+git20130218-3ubuntu41.debian.tar.gzhttp://archive/primary/+files/android-tools_4.2.2+git20130218-3ubuntu41.debian.tar.gz
We fix that by always propagating the site method to extra downloads,
but only when they are specified as relative to the main download URI.
For the extra downloads that specify a full URI, it is not systematic
that it is the same site method. For example, a main download could be a
git clone, but an extra download a pure http download; in that case we
can't replicate the site method for extra downloads, so they'll have to
take appropriate care to specify the required method and encoding if
needed.
Reported-by: Jemy Zhang <jemy.zhang@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Jemy Zhang <jemy.zhang@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, the formatting we impose on the _LICENSE variable requires
that we also use the rarely used := assignment operator, which makes
the _LICENSE variable the only variable that users have to write with
this operator.
This really departs from the simplicity and consistency of using the
append-assignment, which we use for every other variable.
This is because the append-assignment operator surreptiously
introduces a space between the original value and the appended one. But
we can use this knowledge, to match any instance of a space followed by
a comma, and turn it into a single comma.
This allows users to now have a consistent use of the '=' and '+='
operators we use everywhere else in .mk files.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Add an OpenRC service that starts and stops sysv-init scripts. We order
that script 'after local' so that it is started after all other native
openrc services.
Signed-off-by: Michał Łyszczek <michal.lyszczek@bofc.pl>
[yann.morin.1998@free.fr:
- don't propagate the micro optimisation for running .sh scripts
- use spaces, not TABs
- stop services in reverse order
- reword commit log
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, foo-depends only builds build dependencies. This means that
download and extract dependencies are not built.
First, but a minor point, this is inconsistent with foo-show-depends,
which does display all dependencies:
$ make host-gzip-show-depends
host-skeleton host-tar
$ make host-gzip-depends
# Only host-skeleton is built and installed
Second, and more important, it makes it more difficult to preapre a
debug build, like so:
$ make foo-depends
$ tar cf output.tar output
$ make foo
# bummer, broken
# edit foo.mk to try and fix it
$ rm -rf output; tar xf output.tar
# rince and repeat
Change foo-depends so that it really builds all the dependencies for
foo, bringing it on-par with foo-show-depends.
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
We have some internal packages which automatically bake a version string
from the git checkout, and we usually combine these with a
*_OVERRIDE_SRCDIR. I would like to let Buildroot *not* skip the .git
directory when picking up sources from the local checkout. It turns out
that the existing mechanism (*_OVERRIDE_SRCDIR_RSYNC_EXCLUSIONS) only
supports adding to the exclude list because `rsync` simply uses the
first match from the provided filtering rules.
Solve this by using the user-provided values first. If they match, then
`rsync` won't exclude stuff based on the generic VCS exclude patterns.
Signed-off-by: Jan Kundrát <jan.kundrat@cesnet.cz>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
* package/pkg-generic.mk
Add <pkg>_INSTALL_INIT_OPENRC so packages can define their own steps
to install openrc service scripts.
* docs/manual/adding-packages-generic.txt
update documentation about new hook.
Signed-off-by: Michał Łyszczek <michal.lyszczek@bofc.pl>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
show-dependency-tree was introduced in this release cycle, as a way to
quickly and easily provide the dependency tree to graph-depends.
show-dependency-tree is no longer used, now that graph-depends has been
switched over to using the more versatile show-info.
Beside, show-dependency-tree has never been part of a release.
Drop it.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
Sometimes, it is need to quickly get the metadata of a subset of
packages, without resorting to a full-blown JSON query.
Introduce a new per-package (and per-filesystem) foo-show-info rule,
that otputs a per-entity valid JSON blob.
Note that calling it for multiple packages and.or filesystems at once
will not generate a valid JSON blob, as there would be no separator
between the JSON elements:
$ make {foo,bar}-show-info
{ "foo": { foo stuff } }
{ "bar": { bar stuff } }
However, jq is able to absorb this, with its slurping ability, which
generates an array (ellipsed and manualy reformated for readability):
$ make {foo,bar}-show-info |jq -s . -
[
{ "foo": { foo stuff } },
{ "bar": { bar stuff } }
]
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, the DOWNLOAD macro is context-dependent and expects
the PKG variable to be set to the current package.
This is not so nice.
Change the macro to expect the upper-case package name as a
parameter, rather than expect it from a variable.
Adapt the caller accordingly.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Currently, host-xz and host-lzip are built as soon as the
corresponding tools are not provided by the system, independently of
whether they are really needed by the Buildroot configuration. This is
particularly annoying for host-lzip, which is only needed for very few
packages.
This commit modifies the generic package infrastructure to only add
host-lzip and host-xz as dependencies when really needed.
Signed-off-by: Norbert Lange <nolange79@gmail.com>
[Thomas:
- improve commit log
- as suggested by Yann E. Morin, make the lzip case similar to the xz
case]
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
The %-graph-rdepends should be marked PHONY, as the %-graph-depends is.
The %-show-dependency-tree should also be a PHONY rule, because it does
not actually create a file. Furthermore, to avoid the "Nothing to be
done for .." message, give that rule an actual recipe that just does
nothing.
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Currently, when we tweak the .la files, we do so unconditionally on all
.la files, even those we already fixed in a previous run.
This has the nasty side effect that each .la file will be reported as
being touched by all packages that are installed after the package that
actually installed said .la file.
Since we can't easily know what files were installed by a package (that
is an instrumentation hook, and comes after the mangling), we use a
trick (like is done in libtool?): we do mangle all files, each into a
temporary location; if the result is identical to the input, we remove
the temporary, while if the result differs from the output, we move
the temporary to replace the input.
Reported-by: Nicolas Cavallari <nicolas.cavallari@green-communications.fr>
Reported-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Nicolas Cavallari <nicolas.cavallari@green-communications.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Yann E. MORIN