clearlinux/clear-linux-documentation
2
0
Fork 0
mirror of https://github.com/clearlinux/clear-linux-documentation.git synced 2026-08-27 10:55:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

Adds swupd-3rd-party option; removes instruction on Docker. (#1050)

Browse Source 
- Use of Docker with mixer is now deprecated.
- Recreates grid table as list-table for future maintenance.
- Adds new section, Bundle definition files, for illustration
- Revises builder.conf in References, removing docker path.
- Updates mixer builder.conf to VERSION 1.2.
- Fix typo in filepath for LOCAL_RPM_DIR in builder.conf.

Signed-off-by: Michael Vincerra <michael.vincerra@intel.com>
This commit is contained in:
michael vincerra
2020-03-30 17:02:59 -07:00
committed by GitHub
parent 154beef96e
commit 2ffff6fa9e
1 changed files with 154 additions and 197 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/guides/clear/mixer.rst
+154 -197
View File
@@ -3,10 +3,9 @@
mixer
#####
**mixer** is the tool used by the |CL-ATTR| team to generate official update
content and releases. The update content generated by mixer is then consumed
by swupd on a downstream client. The same mixer tool is available as part of
|CL| to create your own customized update content and releases.
The |CL-ATTR| team uses **mixer** to generate official update content and
releases. The update content generated by mixer is then consumed by swupd on
a downstream client. The same mixer tool is available to those who wish to create customized update content and releases.
.. contents::
:local:
@@ -20,13 +19,21 @@ mixer uses the following sources as inputs to generate update content:
* Upstream |CL| bundles with their corresponding RPM packages
* Locally-defined bundles with their corresponding local RPM packages
* Locally-defined bundles with upstream RPM packages
* Locally-defined bundles with non-RPM content
Using the mixer tool, you select which set of content from these sources
will be part of your update. You can select content from each of these sources to make a unique combination of functionality for your custom update content, known as a **mix**.
Using the mixer tool, you select which content from these sources that
becomes part of your update. Your selection of sources produces a unique
combination of functionality for your custom update content, known as
a **mix**.
The update content that mixer generates consists of various pieces of OS
content, update metadata, as well as a complete image. The OS content
includes all files in an update, as well as zero- and delta-packs for improved update performance. The update metadata, stored as manifests, describes all of the bundle information for the update. Update content produced by mixer is then published to a web server and consumed by clients via :command:`swupd`. Refer to :ref:`swupd <swupd-guide>` for additional information regarding updates and update content.
includes all files in an update, as well as zero- and delta-packs for
improved update performance. The update metadata, stored as manifests,
describes all of the bundle information for the update. Update content
produced by mixer is then published to a web server and consumed by clients
via :command:`swupd`. Refer to :ref:`swupd <swupd-guide>` for additional
information regarding updates and update content.
How it works
************
@@ -45,26 +52,6 @@ Prerequisites
Add the mixer tool by installing the :command:`mixer` bundle. Refer to
:ref:`swupd-guide` for more information on installing bundles.
* Docker\* container
mixer by default runs all build commands in a Docker container to ensure
the correct tool versions are used. This also allows custom mixes to
automatically perform downstream format bumps when the upstream releases a
format bump. See `Format version`_ for additional information regarding
format bumps.
Refer to `Configure and enable Docker`_ for instruction.
* Docker proxy (optional)
If you use a proxy server, you must set your proxy environment variables and
create a proxy configuration file for the Docker daemon and container.
Consult your IT department for the correct values if you are behind a
corporate proxy.
Refer to `Configure Docker proxy info`_ for instruction.
* Location to host the update content and images
In order for :command:`swupd` to make use of your mix, the update content for your mix must be hosted on a web server. Your mix will be configured with an update location URL, which :command:`swupd` will use to pull down updates.
@@ -256,11 +243,13 @@ these include the :command:`native-kernel` bundle that is intended to be
used on a bare metal system instead of a VM. So we will modify the default
bundle set to get a smaller kernel image, which will also be faster to load.
The only bundles available to :command:`swupd` for a given release are those
that were added to the mix during build time. A mix doesn’t automatically
inherit upstream bundles.
.. note::
The only bundles available to :command:`swupd` for a given release are
those that were added to the mix during build time. A mix doesn’t
automatically inherit all upstream bundles.
#. Assure that you have run `mixer init`, shown in Example 1.
#. Ensure that you have run `mixer init`, shown in Example 1.
#. Update bundles in mix:
@@ -710,11 +699,6 @@ Follow the `afb.sh reference script`_ to learn how to do a manual format bump. T
* Do a format bump to remove the deprecated bundle
..
Example: Create a mix with custom RPM
..
TODO future example to show copy into local-rpms...
References
**********
@@ -738,32 +722,89 @@ content.
#builder.conf
#VERSION 1.0
#VERSION 1.2
[Builder]
CERT = "/home/clr/mix/Swupd_Root.pem"
SERVER_STATE_DIR = "/home/clr/mix/update"
VERSIONS_PATH = "/home/clr/mix"
YUM_CONF = "/home/clr/mix/.yum-mix.conf"
CERT = "/home/clr/mix/Swupd_Root.pem"
SERVER_STATE_DIR = "/home/clr/mix/update"
VERSIONS_PATH = "/home/clr/mix"
YUM_CONF = "/home/clr/mix/.yum-mix.conf"
[Swupd]
BUNDLE = "os-core-update"
CONTENTURL = "<URL where the content will be hosted>"
VERSIONURL = "<URL where the version of the mix will be hosted>"
BUNDLE = "os-core-update"
CONTENTURL = "<URL where the content will be hosted>"
VERSIONURL = "<URL where the version of the mix will be hosted>"
COMPRESSION = ["external-xz"]
UPSTREAM_BUNDLES_URL = "https://github.com/clearlinux/clr-bundles/archive/"
[Server]
DEBUG_INFO_BANNED = "true"
DEBUG_INFO_LIB = "/usr/lib/debug"
DEBUG_INFO_SRC = "/usr/src/debug"
DEBUG_INFO_BANNED = "true"
DEBUG_INFO_LIB = "/usr/lib/debug"
DEBUG_INFO_SRC = "/usr/src/debug"
[Mixer]
LOCAL_BUNDLE_DIR = "/home/clr/mix/local-bundles"
LOCAL_REPO_DIR = ""
LOCAL_RPM_DIR = ""
DOCKER_IMAGE_PATH = "clearlinux/mixer"
LOCAL_BUNDLE_DIR = "/home/clr/mix/local-bundles"
LOCAL_REPO_DIR = "/home/clr/mix/local-yum"
LOCAL_RPM_DIR = "/home/clr/mix/local-rpms"
OS_RELEASE_PATH = ""
Additional explanation of variables in :file:`builder.conf` is provided in Table
1.
Additional explanation of variables in :file:`builder.conf` is provided in
Table 1.
.. list-table:: **Table 1**: Variables in builder.conf
:widths: 50, 50
:header-rows: 1
* - **Variable**
- **Description**
* - `CERT`
- Sets the path where mixer stores the certificate file used to sign
content for verification. mixer automatically generates the
certificate if you do not provide the path to an existing one, and
signs the :file:`Manifest.MoM` file to provide security for the
updated content you create.
chroot-builder uses the certificate file to sign the root :file:`
Manifest.MoM` file to provide security for content verification.
swupd uses this certificate to verify the :file:`Manifest.MoM` file's
signature.
For now, we strongly recommend that you do not modify this variable,
as swupd expects a certificate with a very specific configuration to sign and verify properly.
* - `CONTENTURL` and `VERSIONURL`
- Set these variables to the IP address of the web server hosting the
update content.
VERSIONURL is the IP address where the swupd client
looks to determine if a new version is available.
CONTENTURL is the location from which swupd pulls content updates. If
the web server is on the same machine as the SERVER_STATE_DIR
directory, you can create a symlink to the directory in your web
server's document root to easily host the content.
These URLs are embedded in the images created by mixer.
* - `LOCAL_BUNDLE_DIR`
- Sets the path where mixer stores the local bundle definition files.
The bundle definition files include any new, original bundles you
create, along with any edited versions of upstream bundles.
* - `SERVER_STATE_DIR`
- Sets the path to which mixer outputs content. By default, mixer
automatically sets the path.
* - `VERSIONS_PATH`
- Sets the path for the mix version and upstream version's two state
files: :file:`mixversion` and :file:`upstreamversion`. mixer creates
both files for you when you set up the workspace.
* - `YUM_CONF`
- Sets the path where mixer automatically generates the
:file:`.yum-mix.conf` file. The yum configuration file points the
chroot-builder to where the RPMs are stored.
+-------------------------------+----------------------------------------------------------+
| **Variable** | **Explanation** |
@@ -803,9 +844,6 @@ Additional explanation of variables in :file:`builder.conf` is provided in Table
| | |
| | These URLs are embedded in the images created by mixer. |
+-------------------------------+----------------------------------------------------------+
| `DOCKER_IMAGE_PATH` | Sets the base name of the docker image that mixer pulls |
| | down to run builds in the proper container. |
+-------------------------------+----------------------------------------------------------+
| `LOCAL_BUNDLE_DIR` | Sets the path where mixer stores the local bundle |
| | definition files. The bundle definition files include |
| | any new, original bundles you create, along with any |
@@ -877,13 +915,12 @@ Bundles
=======
mixer stores information about the bundles included in a mix in a flat file
called :file:`mixbundles`, which is located in the path set by the VERSIONS_PATH variable in :file:`builder.conf`. :file:`mixbundles` is automatically created when the mix is initiated. mixer will refresh the file each time you change the bundles in the mix.
called :file:`mixbundles`, which is located in the path set by the
VERSIONS_PATH variable in :file:`builder.conf`. :file:`mixbundles` is
automatically created when the mix is initiated. mixer will refresh the file
each time you change the bundles in the mix.
Bundles can include other bundles. Nested bundles can themselves include
other bundles. If you see an unexpected bundle in your mix, it is likely a
nested bundle in one of the bundles you explicitly added.
A bundle will fill into one of two categories: upstream or local. Upstream
Bundles belong in one of two categories: upstream or local. Upstream
bundles are those provided by |CL|. Local bundles are either modified upstream bundles or new local bundles.
Upstream bundles
@@ -897,8 +934,8 @@ contents of this directory before repopulating it on-the-fly if a new
version must be downloaded.
The mixer tool automatically caches the bundles for the |CL| version
configured in the :file:`upstreamversion` file. mixer also cleans up old
versions once they are no longer needed.
configured in the :file:`upstreamversion` file. :command:`mixer` also
cleans up old versions once they are no longer needed.
Local bundles
-------------
@@ -913,13 +950,62 @@ precedence over any upstream bundles that have the same name. This
precedence enables you to copy upstream bundles locally, and edit into a
local variation.
Bundle definition files
-----------------------
A ``bundle definition`` file consists of a header, followed by a list
of packages and directives. The header holds important meta-data, like
the TITLE, DESCRIPTION, and STATUS. Other meta-data include TAGS, which
define a bundle's function in the ecosystem, and MAINTAINER, which gives
contact information.
Following the header are the directives, shown in Table 2.
.. list-table:: **Table 2**: Bundle directives
:widths: 50,50
:header-rows: 1
* - **Directive**
- **Description**
* - ``include(<required-bundle-name>)``
- Add <required-bundle-name> with this bundle
* - ``also-add(<optional-bundle-name>)``
- Add <optional-bundle-name> unless the option ``--skip-optional`` is used with ``swupd bundle-add``.
* - ``content(<full/path/to/non-packaged/content>)``
- Add the non-packaged content to the bundle. Refer to :ref:`swupd-3rd-party` for usage of this directive.
Following is `cluster-tools`, an upstream bundle definition file. The
directives are highlighted, and the rest are packages.
.. code-block:: bash
:emphasize-lines: 8-12
[TITLE]: cluster-tools
[DESCRIPTION]: Utilities to manage computer clusters
[STATUS]: Active
[CAPABILITIES]: HPC
[TAGS]: Tools and Utilities
[MAINTAINER]: Juro Bystricky <juro.bystricky@intel.com>
include(curl)
include(libglib)
include(libX11client)
also-add(openmpi)
also-add(modules)
munge
pmix
pdsh
slurm
Bundle configuration
--------------------
mixer provides commands to configure the bundles for a mix, such as to add a
bundle to a mix, to create a new bundle for a mix, or to remove a bundle from a
mix. View the `mixer.bundle man page`_ for a full list of commands and more
information on configuring bundles in a mix.
bundle to a mix, to create a new bundle for a mix, or to remove a bundle from a mix. View the `mixer.bundle man page`_ for a full list of commands and more information on configuring bundles in a mix.
Editing an existing local bundle is as simple as opening the bundle definition
file in your favorite editor, making the desired edits, and saving your changes.
@@ -935,135 +1021,6 @@ file in your favorite editor, making the desired edits, and saving your changes.
.. rst-class:: content-collapse
Configure and enable Docker
===========================
Use these steps to enable Docker for the mixer tool. Make sure to
`Configure Docker proxy info`_ first if needed.
#. Start the Docker daemon:
.. code-block:: bash
sudo systemctl start docker
sudo chmod 777 /var/run/docker.sock
sudo docker info
#. Add user to the docker group
.. code-block:: bash
sudo usermod -G docker -a <username>
Pull Docker container manually (optional)
-----------------------------------------
By default, mixer automatically pulls a Docker container for mixing if one
does not already exist. If you need to troubleshoot the mixer container, it
may be useful to manually pull a mixer Docker container.
Versions of the mixer Docker container are available under the tags for the
`clearlinux/mixer repo <https://hub.docker.com/r/clearlinux/mixer/tags/>`_
on Docker Hub. Each version of the mixer Docker container is named after the
associated |CL| upstream format version. Refer to `Format version`_ for
additional information on upstream format versions.
Use the following steps to manually pull a mixer Docker container:
#. Find the version of the container you need by viewing the tags for the
`clearlinux/mixer repo <https://hub.docker.com/r/clearlinux/mixer/tags/>`_
on Docker Hub.
#. Pull the latest container version:
.. code-block:: bash
docker pull clearlinux/mixer:<upstream-format-version>
#. View local docker images:
.. code-block:: bash
docker images
.. rst-class:: content-collapse
Configure Docker proxy info
===========================
If needed, use these steps to configure the Docker proxy information.
#. Create the Docker daemon proxy config directory:
.. code-block:: bash
sudo mkdir -p /etc/systemd/system/docker.service.d
#. Create :file:`/etc/systemd/system/docker.service.d/http-proxy.conf` and
add the following using your own proxy values:
.. code-block:: console
[Service]
Environment="HTTP_PROXY=<HTTP proxy URL>:<port number>"
Environment="HTTPS_PROXY=<HTTPS proxy URL>:<port number>"
#. Reload the Docker daemon:
.. code-block:: bash
sudo systemctl daemon-reload
Configure the Docker container proxies, to pass proxy settings to
containers:
#. Create a directory for your container config:
.. code-block:: bash
mkdir ~/.docker
#. Create the config file :file:`~/.docker/config.json` and add the following
entries, using your own proxy values:
.. code-block:: console
{
"proxies":
{
"default":
  {
   "httpProxy": "<proxy-url>:<port>",
    "httpsProxy": "<proxy-url>:<port>"
  }
}
}
#. Set ownership and permission on the docker config directory:
.. code-block:: bash
sudo chown "$USER":"$USER" /home/"$USER"/.docker -R
sudo chmod g+rwx "$HOME/.docker" -R
Configure proxies to allow mixer to access upstream content from behind
a firewall.
#. Open your :file:`$HOME/.bashrc` file and add proxy and port values for the
following:
.. code-block:: console
export http_proxy="<proxy-url>:<port>"
export https_proxy="<proxy-url>:<port>"
export HTTP_PROXY="<proxy-url>:<port>"
export HTTPS_PROXY="<proxy-url>:<port>"
export no_proxy="<...>"
#. Log out and log back in for the proxies to take effect.
.. rst-class:: content-collapse
.. _set-up-nginx-web-server-start:
Set up a nginx web server for mixer
@@ -1136,8 +1093,8 @@ Related topics
* :ref:`autospec`
* :ref:`bundles-guide`
* :ref:`swupd-guide`
* :ref:`swupd-3rd-party`
.. _Docker Hub: https://hub.docker.com/r/clearlinux/mixer/tags/
.. _mixer man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.1.rst
.. _mixer.init man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.init.1.rst
.. _mixer.bundle man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.bundle.1.rst