Implement the logging package for mixer.
A common log file can be set for all the mixer commands in the
builder.conf. E.g. `mixer config set Mixer.LOG <filepath>`
The log file and level can also be set for individual mixer
commands using the `--log` and `--log-level` flags respectively.
The various log levels are:
ERROR (1), WARNING (2), INFO (3), DEBUG (4) and VERBOSE (5).
Default log level is 4.
Fixes#666
Signed-off-by: Ashlesha Atrey ashlesha.atrey@intel.com
When calling the copyFileWithFlags function with the preserveSrc flag
set, file permissions must be set after creating the destination file
and setting the file owner. This is necessary because umask can prevent
newly created files from acquiring the source file's permissions and
file ownership changes can impact the setuid/setgid permissions.
Similarly directory permissions are set after creating the directory to
avoid unexpected permission behavior caused by umask.
Signed-off-by: John Akre <john.w.akre@intel.com>
When using content chroots, file uid/gid values should be preserved when
copying them to the full chroot.
Fixes#727
Signed-off-by: John Akre <john.w.akre@intel.com>
The CopyFileWithOptions function has the ability to disable symlink
resolution, preserve the source file permissions, and disable disk syncs.
Signed-off-by: John Akre <john.w.akre@intel.com>
golangci-lint identified a couple errors that were not found by
gometalinter. This patch fixes the detected errors.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This patch changes how Warnings and Error messages are printed
throughout the code to use the `log` package instead of `fmt`. This
allows for finer control over what is printed and where it is printed
to.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
If an error occurred when running an external program, the user should
receive a full report of that program's output. If mixer is unable to
write that output, the user should get and error indicating the problem
instead of an incomplete output, making it easier to debug the problem.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
This reverts commit 9cf854d8d6.
The reduction to the timeout has caused failures for users on slow
networks, so this must be reverted to the default values.
The reduction to the timeout was added to improve user experience when
connection failed due to a proxy error. It would take mixer too long to
fail and the error message didn't indicate what the problem could be.
The error is now properly indicated, so even if it takes a while for the
error to occur the first time, the user should be able to identify the
problem on the spot.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
In some situations dnf may try to combine files from different
encodings. If the command is not running is a UTF-8 locale, this
combination will be missing a new line, causing and error.
An example of this occurrence is the command `dnf repoquery --quiet -l
systemd ca-certs-static ...` which combines a file list from systemd and
ca-certs-static
This patch enforces UTF-8 locale on every dnf execution.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
The error when mixer failed to download the upstream version was too
confusing since there were many layers of wrapped errors.
This patch removes the excessive wrapping and suggests a missing proxy
configuration to the user in case this error occurs.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Set timeout to 1 minute for bsdiff. The majority of all delta creations
take significantly less than 1 minute to run, which signifies that bsdiff
is working on a delta that may be very large, or very difficult to diff. In
all the cases where bsdiff took multiple minutes to finish, the delta ended
up not being used because it was larger than the compressed fullfile. This
attempts to skip those cases and improve delta creation time.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Fix a number of AFB bugs:
- Previous build's upstream format was being incorrectly looked up,
resulting in upstream format boundary crossing detection failing.
- PreRun check was incorrectly skipping format build commands, resulting
in "stage for mix" being run twice.
- Only the UInt32 upstream version was being modified, resulting in the
second half of 'mixer build upstream-format' being run in the wrong
container.
- The wrong format was being printed out for 'mixer versions' if
crossing an upstream format boundary.
Refactor Cobra command PreRun
Simplified the PreRun logic that happens before every command. This also
fixes an issue where format bump build commands could be run with
--native=false, when they need to be run natively.
Fix AFB mixversion and LAST_VER setting
1) Changes how the format bump builds set mixversion so that it no
longer skips a version number in some situations.
2) Updates the update/image/LAST_VER file after the +10 build back
to the +20 build value, so mixer is ready to do the next mix.
Refactor file download code
This patch refactors and consolidates the different file download code
scattered throughout helper.go and builder.go.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Download a file is a generic operation. This patch moves this
functionality from builder to helpers to allow it to be reused
by other packages.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Previously the code was using ioutils.ReadDir which does a stat call
on every file, but none of the code required the results of the stat
calls, just the names.
In addition programs such as git would like to hold information in the
directories. The usual unix way to do this is to have filenames
beginning with '.', and these are ignored by default by things such as
shell globbing and the ls program.
fixes#269
Signed-off-by: Icarus Sparry <icarus.w.sparry@intel.com>
Deprecate use of chroot-building for each individual bundle. This allows
us to scale much easier on machines with limited disk space. Because
chroots are decoupled from manifest creation this also makes it possible
in the future to move to doing bundle builds on distributed machines and
allows our bundle numbers to scale. The bundle file list is stored in a
JSON file that is read later by the manifest creation step.
This leaves a fallback in swupd/fullchroot.go to allow users to add
content to a chroot in order to extend a typical build.
Signed-off-by: Matthew Johnson <matthew.johnson@intel.com>
Adds new command, 'mixer bundle edit', that allows a user to edit local
and upstream bundle definition files. This command will locate the
bundle (looking first in local-bundles, then in upstream-bundles), and launch
an editor to edit it. If the bundle is only found upstream, the bundle file will
first be copied to your local-bundles directory for editing. When the editor
closes, the bundle file is then parsed for validity.
The editor is configured via environment variables. VISUAL takes precedence to
EDITOR. If neither are set, the tool defaults to nano. If nano is not installed,
the tool will skip editing, and act as if '--copy-only' had been passed.
Passing '--copy-only' will suppress launching the editor, and will thus only copy
the bundle file to local-bundles if it is only found upstream. This can be
useful if you want to add a bundle to local-bundles, but wish to edit it at a
later time.
Passing '--add' will also add the bundle(s) to your mix. Please note that
bundles are added after all bunles are edited, and thus will not be added if any
errors are encountered earlier on.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
When RunCommandOutput (or RunCommandSilent) fails, only add the STDOUT
and STDERR to the error string if they have any content (otherwise
omit).
Also add some text without a newline in the end of the string to allow
the underlying error to be printed correctly. If you've seen lines
with ": exit status NNN" this last change will make things looks
better.
In RunCommand, simplify the error message taking in account that the
wrap will append the underlying error when printed.
Previously, helpers.CopyFile left off os.O_TRUNC, meaning that
if the destination is larger than the source, the rest of the
destination remained at the end. Now the file is overwritten
completely.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
This commit refactors the way you specify what bundles are included
in your mix, as well as how the mix-bundles directory gets
generated.
Major changes in this commit:
- Introduces the Mix Bundles List, a newline-separated file in the
working directory that lists the bundles that must be in the mix.
Other bundles (included by those in the list) will be added
automatically.
- Introduces the local-bundles directory that stores any bundles
that the user created or that have been edited from upstream.
Bundles in the local-bundles directory take precedence over
upstream bundles of the same name.
- Generates the mix-bundles/ directory (used as input to the BCB)
automatically on-the-fly when chroots are built. The Mix Bundles
List is recursed to find all bundles needed for the mix, and the
bundle definition files are copied from either the local or
upstream bundles.
- If '--new-chroots' is passed, the mix-bundles/ directory is
skipped entirely, as the new BCB is capable of reading the
files from anywhere.
- Modifies the 'mixer bundle add' command to now edit the Mix
Bundles List. Bundles are verified to exist in either the local
or upstream bundles. The user is informed whether the added
bundles came from local or upstream.
- Adds a new 'list' command to 'mixer bundle' that prints out
the bundles in the Mix Bundle List (including those
recursively included), all bundles available in local bundles,
or all bundles available in upstream bundles. The mix bundle
list, bundle names are annotated with information about their
origin (local or upstream) and whether they are part of the
Mix Bundle List or just included. For the local and upstream
lists, bundle names are annotated with whether or not they are
currently included in the mix. All three of the above list
types support a '--tree' option that prints them as a formatted
tree view, visually showing the include relationship for bundles
and how the full list is being generated.
Minor changes in this commit:
- As the mix-bundles directory is now created on-the-fly, it is no
longer tracked as a git repository. Instead, the entire mixer
working directory is tracked, with temporary, tool-generated
files ignored. This allows one to track and correlate the Mix
Bundle List, mix version, and upstream version as they are changed.
- The above-mentioned git revision tracking is now optional, and
is only set up if '--git' is passed to 'mixer init'. 'mixer
bundle add' still supports '--git', which mirrors this behavior.
- The '--all' flags for 'mixer init' and 'mixer bundle add' have
been split up into '--all-local' and '--all-upstream'. This
allows a user to add or start with all local or upstream bundles,
or both.
This rewrites bundle-chroot-builder.py in Go to be part of Mixer
code. Mixer is the only user of that software, and both Mixer and
bundle-chroot-builder.py the same configuration file, with overlapping
fields.
Main differences from bundle-chroot-builder.py:
- New bundleset type was added, that cares about collecting as much
information as possible from the bundles themselves. This type and
related functions also sets us up for success when upcoming changes
to how bundles are specified happen. There is no assumption all
bundles are in the same directory.
- We are not using m4, instead a bundleset takes care of parsing. If
format of individual bundle files change. The upside is that we can
give nicer error messages, specially for the circular case.
- Read the configuration file directly (with go-ini) to peek at values
that Mixer didn't read before. Done that to avoid conflicting with
existing patch in-flight that parses configuration.
- Some individual steps were reordered for code clarity. E.g.: since
we have bundleset, we can upfront generate all the *-include files.
- Fixed the output for versions file. Due to the way yum list output
works, parsing it is not very friendly. Comments around the code
tells the story.
- Removed the network testing step. It wasn't covering every case in
the Python version, so I'm leaning to let the failure come from
yum/dnf itself. I'm usually in favor of such early tests, but in
this case the price of parsing yet another config file didn't felt
worth.
- Removed the "yum clean all" step from the bootstrap. There isn't any
cache at that point, and the next yum call will bootstrap the
necessary files for yum to operate.
- Removed generation of files-* files (and the pkgmap-* files used to
generate them). I couldn't find any tool or team making use of this
information. Those (or their content) might be relevant in future
changes to use a single chroot, but we should add when we need them.
- Added more detailed commentary to individual steps, collecting
information from the developers of bcb and related software.
- The port still don't parallelize the work into multiple
goroutines. I plan to do this in a similar way than what was done in
CreateFullfiles, but in a separated patch.
Fixes#42.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
When initializing the mix workspace, if a file that would be initialized
already exists, it should be preserved. This patch introduces a generic
implementation of CopyFile that accepts custom flags and provides two
interfaces with pre-defined flags: CopyFile, which keeps the original
implementation and CopyFileNoOverwrite, which adds the new functionality.
Signed-off-by: Rodrigo Chiossi <rodrigo.chiossi@intel.com>
Split up the download/unpacking of upstream bundles and the
creation of the mix-bundles directory.
Fetching upstream bundles now lives in a standalone function that
is called by any function that relies on having upstream bundle
definitions. This means the tool automatically handles the caching
of upstream bundles, making the 'mixer bundle get' command
obsolete.
Creation of the mix-bundles directory has been moved to InitMix.
This patch also includes general cleanup and error handling,
especially for InitMix and AddBundles.
This patch has the following side-effects:
1) The upstream bundles are now considered temporary, sausage-
making data. As such, they now live in .mixer/upstream-bundles/,
and the .tar.gz file used to fetch them is cleaned up.
2) The method for fetching upstream bundles has an optional
parameter ('prune') that will clean up the bundle cache for other
versions of upstream. This should be considered the default value
as part of standard clean-up behavior; a value of false should only
be used in cases where a method needs to have multiple versions
of upstream bundles cached simultaneously.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
When unpacking upstream bundle defintions, previously an exec
was made to 'tar'. This patch replaces the exec call with a
native function that uses the Go tar and gzip packages instead.
Signed-off-by: Kevin C. Wells <kevin.c.wells@intel.com>
Main changes:
- Rename parameters to reflect better their intention. In particular
"signflag" -> "skipSigning"
- Remove unnecessary Stat check for dir creation: MkdirAll will do
that and not fail if the directory already exists.
- Changed commands to do like the swupd_create_update and their output
the error directly to the os.Stdout/os.Stderr.
- Changed most of file path building to use filepath.Join.
- Inline setVersion, since these two steps make sense at build
update level.
- Only create upstreamver file if we create an upstreamurl file, and do
that before the final publishing step.
- Added comments to clarify some of the steps.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
Make it return errors, check for more cases and avoid the Chdir by
setting the relevant directory in the exec.Cmd.
Signed-off-by: Caio Marcelo de Oliveira Filho <caio.oliveira@intel.com>
The cobra package is very mature for writing CLIs, and makes the code
much easier to maintain and add new commands to. This feature ports the
current implementation over to using cobra, maintaining the same calling
convention for the commands. However, it would be trivial and perhaps
better to update the form from:
mixer build-chroots --flags
to
mixer build chroots --flags
with chroots being a subcommand of build, as the code is currently
structured in cmd.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Follow the expected Go convention for packages, and put the project
under github.com/clearlinux/mixer-tools with the correct hierarchy.
Simplify Makfile for the time being.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Because Clear version numbers use numeric digits only, it's possible for
the tag names to conflict with the prefix of a git object hash in the
repo. To remove the possibility of conflict, use the refs/tags/ prefix
when querying the tag names.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
With the current workflow, the "mix-bundles" repo is used for
mix-specific bundle changes, and the "clr-bundles" repo is treated as
read-only.
So in clr-bundles, the CLRVER_mix branches should always point to the
commit tagged with CLRVER. To make this handling more robust, first
verify that CLRVER exists and is a tag. Then create the CLRVER_mix
branch based on CLRVER, or if the branch already exists, reset it
appropriately.
Since this is the best place to detect invalid CLEAR_VERSION numbers
passed in the builder.conf, I've added that error handling too.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
Because 'set -e' is enabled for init-mix.sh, the exit status of 'type'
is not checked, so an error raised by 'type' would exit immediately, not
running the $? conditional logic.
To fix, disable 'set -e' for the dependency check function.
Signed-off-by: Patrick McCarty <patrick.mccarty@intel.com>
This will do a lot of the ugly work that does not need to be taking up space
in the main scripts, and should be abstracted out so they are easier to read.
Signed-off-by: Tudor Marcu <tudor.marcu@intel.com>
Reagan Lopez