From 2ffff6fa9ed3695ccdf76163dcccb7d52962c6fd Mon Sep 17 00:00:00 2001 From: michael vincerra <37549381+mvincerx@users.noreply.github.com> Date: Mon, 30 Mar 2020 17:02:59 -0700 Subject: [PATCH] Adds swupd-3rd-party option; removes instruction on Docker. (#1050) - Use of Docker with mixer is now deprecated. - Recreates grid table as list-table for future maintenance. - Adds new section, Bundle definition files, for illustration - Revises builder.conf in References, removing docker path. - Updates mixer builder.conf to VERSION 1.2. - Fix typo in filepath for LOCAL_RPM_DIR in builder.conf. Signed-off-by: Michael Vincerra --- source/guides/clear/mixer.rst | 351 +++++++++++++++------------------- 1 file changed, 154 insertions(+), 197 deletions(-) diff --git a/source/guides/clear/mixer.rst b/source/guides/clear/mixer.rst index c475cf08..03897532 100644 --- a/source/guides/clear/mixer.rst +++ b/source/guides/clear/mixer.rst @@ -3,10 +3,9 @@ mixer ##### -**mixer** is the tool used by the |CL-ATTR| team to generate official update -content and releases. The update content generated by mixer is then consumed -by swupd on a downstream client. The same mixer tool is available as part of -|CL| to create your own customized update content and releases. +The |CL-ATTR| team uses **mixer** to generate official update content and +releases. The update content generated by mixer is then consumed by swupd on +a downstream client. The same mixer tool is available to those who wish to create customized update content and releases. .. contents:: :local: @@ -20,13 +19,21 @@ mixer uses the following sources as inputs to generate update content: * Upstream |CL| bundles with their corresponding RPM packages * Locally-defined bundles with their corresponding local RPM packages * Locally-defined bundles with upstream RPM packages +* Locally-defined bundles with non-RPM content -Using the mixer tool, you select which set of content from these sources -will be part of your update. You can select content from each of these sources to make a unique combination of functionality for your custom update content, known as a **mix**. +Using the mixer tool, you select which content from these sources that +becomes part of your update. Your selection of sources produces a unique +combination of functionality for your custom update content, known as +a **mix**. The update content that mixer generates consists of various pieces of OS content, update metadata, as well as a complete image. The OS content -includes all files in an update, as well as zero- and delta-packs for improved update performance. The update metadata, stored as manifests, describes all of the bundle information for the update. Update content produced by mixer is then published to a web server and consumed by clients via :command:`swupd`. Refer to :ref:`swupd ` for additional information regarding updates and update content. +includes all files in an update, as well as zero- and delta-packs for +improved update performance. The update metadata, stored as manifests, +describes all of the bundle information for the update. Update content +produced by mixer is then published to a web server and consumed by clients +via :command:`swupd`. Refer to :ref:`swupd ` for additional +information regarding updates and update content. How it works ************ @@ -45,26 +52,6 @@ Prerequisites Add the mixer tool by installing the :command:`mixer` bundle. Refer to :ref:`swupd-guide` for more information on installing bundles. -* Docker\* container - - mixer by default runs all build commands in a Docker container to ensure - the correct tool versions are used. This also allows custom mixes to - automatically perform downstream format bumps when the upstream releases a - format bump. See `Format version`_ for additional information regarding - format bumps. - - Refer to `Configure and enable Docker`_ for instruction. - -* Docker proxy (optional) - - If you use a proxy server, you must set your proxy environment variables and - create a proxy configuration file for the Docker daemon and container. - - Consult your IT department for the correct values if you are behind a - corporate proxy. - - Refer to `Configure Docker proxy info`_ for instruction. - * Location to host the update content and images In order for :command:`swupd` to make use of your mix, the update content for your mix must be hosted on a web server. Your mix will be configured with an update location URL, which :command:`swupd` will use to pull down updates. @@ -256,11 +243,13 @@ these include the :command:`native-kernel` bundle that is intended to be used on a bare metal system instead of a VM. So we will modify the default bundle set to get a smaller kernel image, which will also be faster to load. -The only bundles available to :command:`swupd` for a given release are those -that were added to the mix during build time. A mix doesn’t automatically -inherit upstream bundles. +.. note:: + + The only bundles available to :command:`swupd` for a given release are + those that were added to the mix during build time. A mix doesn’t + automatically inherit all upstream bundles. -#. Assure that you have run `mixer init`, shown in Example 1. +#. Ensure that you have run `mixer init`, shown in Example 1. #. Update bundles in mix: @@ -710,11 +699,6 @@ Follow the `afb.sh reference script`_ to learn how to do a manual format bump. T * Do a format bump to remove the deprecated bundle -.. - Example: Create a mix with custom RPM -.. - TODO future example to show copy into local-rpms... - References ********** @@ -738,32 +722,89 @@ content. #builder.conf - #VERSION 1.0 + #VERSION 1.2 [Builder] - CERT = "/home/clr/mix/Swupd_Root.pem" - SERVER_STATE_DIR = "/home/clr/mix/update" - VERSIONS_PATH = "/home/clr/mix" - YUM_CONF = "/home/clr/mix/.yum-mix.conf" + CERT = "/home/clr/mix/Swupd_Root.pem" + SERVER_STATE_DIR = "/home/clr/mix/update" + VERSIONS_PATH = "/home/clr/mix" + YUM_CONF = "/home/clr/mix/.yum-mix.conf" [Swupd] - BUNDLE = "os-core-update" - CONTENTURL = "" - VERSIONURL = "" + BUNDLE = "os-core-update" + CONTENTURL = "" + VERSIONURL = "" + COMPRESSION = ["external-xz"] + UPSTREAM_BUNDLES_URL = "https://github.com/clearlinux/clr-bundles/archive/" [Server] - DEBUG_INFO_BANNED = "true" - DEBUG_INFO_LIB = "/usr/lib/debug" - DEBUG_INFO_SRC = "/usr/src/debug" + DEBUG_INFO_BANNED = "true" + DEBUG_INFO_LIB = "/usr/lib/debug" + DEBUG_INFO_SRC = "/usr/src/debug" [Mixer] - LOCAL_BUNDLE_DIR = "/home/clr/mix/local-bundles" - LOCAL_REPO_DIR = "" - LOCAL_RPM_DIR = "" - DOCKER_IMAGE_PATH = "clearlinux/mixer" + LOCAL_BUNDLE_DIR = "/home/clr/mix/local-bundles" + LOCAL_REPO_DIR = "/home/clr/mix/local-yum" + LOCAL_RPM_DIR = "/home/clr/mix/local-rpms" + OS_RELEASE_PATH = "" -Additional explanation of variables in :file:`builder.conf` is provided in Table -1. +Additional explanation of variables in :file:`builder.conf` is provided in +Table 1. + +.. list-table:: **Table 1**: Variables in builder.conf + :widths: 50, 50 + :header-rows: 1 + + * - **Variable** + - **Description** + + * - `CERT` + - Sets the path where mixer stores the certificate file used to sign + content for verification. mixer automatically generates the + certificate if you do not provide the path to an existing one, and + signs the :file:`Manifest.MoM` file to provide security for the + updated content you create. + + chroot-builder uses the certificate file to sign the root :file:` + Manifest.MoM` file to provide security for content verification. + swupd uses this certificate to verify the :file:`Manifest.MoM` file's + signature. + + For now, we strongly recommend that you do not modify this variable, + as swupd expects a certificate with a very specific configuration to sign and verify properly. + + * - `CONTENTURL` and `VERSIONURL` + - Set these variables to the IP address of the web server hosting the + update content. + + VERSIONURL is the IP address where the swupd client + looks to determine if a new version is available. + + CONTENTURL is the location from which swupd pulls content updates. If + the web server is on the same machine as the SERVER_STATE_DIR + directory, you can create a symlink to the directory in your web + server's document root to easily host the content. + + These URLs are embedded in the images created by mixer. + + * - `LOCAL_BUNDLE_DIR` + - Sets the path where mixer stores the local bundle definition files. + The bundle definition files include any new, original bundles you + create, along with any edited versions of upstream bundles. + + * - `SERVER_STATE_DIR` + - Sets the path to which mixer outputs content. By default, mixer + automatically sets the path. + + * - `VERSIONS_PATH` + - Sets the path for the mix version and upstream version's two state + files: :file:`mixversion` and :file:`upstreamversion`. mixer creates + both files for you when you set up the workspace. + + * - `YUM_CONF` + - Sets the path where mixer automatically generates the + :file:`.yum-mix.conf` file. The yum configuration file points the + chroot-builder to where the RPMs are stored. +-------------------------------+----------------------------------------------------------+ | **Variable** | **Explanation** | @@ -803,9 +844,6 @@ Additional explanation of variables in :file:`builder.conf` is provided in Table | | | | | These URLs are embedded in the images created by mixer. | +-------------------------------+----------------------------------------------------------+ -| `DOCKER_IMAGE_PATH` | Sets the base name of the docker image that mixer pulls | -| | down to run builds in the proper container. | -+-------------------------------+----------------------------------------------------------+ | `LOCAL_BUNDLE_DIR` | Sets the path where mixer stores the local bundle | | | definition files. The bundle definition files include | | | any new, original bundles you create, along with any | @@ -877,13 +915,12 @@ Bundles ======= mixer stores information about the bundles included in a mix in a flat file -called :file:`mixbundles`, which is located in the path set by the VERSIONS_PATH variable in :file:`builder.conf`. :file:`mixbundles` is automatically created when the mix is initiated. mixer will refresh the file each time you change the bundles in the mix. +called :file:`mixbundles`, which is located in the path set by the +VERSIONS_PATH variable in :file:`builder.conf`. :file:`mixbundles` is +automatically created when the mix is initiated. mixer will refresh the file +each time you change the bundles in the mix. -Bundles can include other bundles. Nested bundles can themselves include -other bundles. If you see an unexpected bundle in your mix, it is likely a -nested bundle in one of the bundles you explicitly added. - -A bundle will fill into one of two categories: upstream or local. Upstream +Bundles belong in one of two categories: upstream or local. Upstream bundles are those provided by |CL|. Local bundles are either modified upstream bundles or new local bundles. Upstream bundles @@ -897,8 +934,8 @@ contents of this directory before repopulating it on-the-fly if a new version must be downloaded. The mixer tool automatically caches the bundles for the |CL| version -configured in the :file:`upstreamversion` file. mixer also cleans up old -versions once they are no longer needed. +configured in the :file:`upstreamversion` file. :command:`mixer` also +cleans up old versions once they are no longer needed. Local bundles ------------- @@ -913,13 +950,62 @@ precedence over any upstream bundles that have the same name. This precedence enables you to copy upstream bundles locally, and edit into a local variation. + +Bundle definition files +----------------------- + +A ``bundle definition`` file consists of a header, followed by a list +of packages and directives. The header holds important meta-data, like +the TITLE, DESCRIPTION, and STATUS. Other meta-data include TAGS, which +define a bundle's function in the ecosystem, and MAINTAINER, which gives +contact information. + +Following the header are the directives, shown in Table 2. + +.. list-table:: **Table 2**: Bundle directives + :widths: 50,50 + :header-rows: 1 + + * - **Directive** + - **Description** + + * - ``include()`` + - Add with this bundle + + * - ``also-add()`` + - Add unless the option ``--skip-optional`` is used with ``swupd bundle-add``. + + * - ``content()`` + - Add the non-packaged content to the bundle. Refer to :ref:`swupd-3rd-party` for usage of this directive. + +Following is `cluster-tools`, an upstream bundle definition file. The +directives are highlighted, and the rest are packages. + +.. code-block:: bash + :emphasize-lines: 8-12 + + [TITLE]: cluster-tools + [DESCRIPTION]: Utilities to manage computer clusters + [STATUS]: Active + [CAPABILITIES]: HPC + [TAGS]: Tools and Utilities + [MAINTAINER]: Juro Bystricky + + include(curl) + include(libglib) + include(libX11client) + also-add(openmpi) + also-add(modules) + munge + pmix + pdsh + slurm + Bundle configuration -------------------- mixer provides commands to configure the bundles for a mix, such as to add a -bundle to a mix, to create a new bundle for a mix, or to remove a bundle from a -mix. View the `mixer.bundle man page`_ for a full list of commands and more -information on configuring bundles in a mix. +bundle to a mix, to create a new bundle for a mix, or to remove a bundle from a mix. View the `mixer.bundle man page`_ for a full list of commands and more information on configuring bundles in a mix. Editing an existing local bundle is as simple as opening the bundle definition file in your favorite editor, making the desired edits, and saving your changes. @@ -935,135 +1021,6 @@ file in your favorite editor, making the desired edits, and saving your changes. .. rst-class:: content-collapse -Configure and enable Docker -=========================== - -Use these steps to enable Docker for the mixer tool. Make sure to -`Configure Docker proxy info`_ first if needed. - -#. Start the Docker daemon: - - .. code-block:: bash - - sudo systemctl start docker - sudo chmod 777 /var/run/docker.sock - sudo docker info - -#. Add user to the docker group - - .. code-block:: bash - - sudo usermod -G docker -a - -Pull Docker container manually (optional) ------------------------------------------ - -By default, mixer automatically pulls a Docker container for mixing if one -does not already exist. If you need to troubleshoot the mixer container, it -may be useful to manually pull a mixer Docker container. - -Versions of the mixer Docker container are available under the tags for the -`clearlinux/mixer repo `_ -on Docker Hub. Each version of the mixer Docker container is named after the -associated |CL| upstream format version. Refer to `Format version`_ for -additional information on upstream format versions. - -Use the following steps to manually pull a mixer Docker container: - -#. Find the version of the container you need by viewing the tags for the - `clearlinux/mixer repo `_ - on Docker Hub. - -#. Pull the latest container version: - - .. code-block:: bash - - docker pull clearlinux/mixer: - -#. View local docker images: - - .. code-block:: bash - - docker images - -.. rst-class:: content-collapse - -Configure Docker proxy info -=========================== - -If needed, use these steps to configure the Docker proxy information. - -#. Create the Docker daemon proxy config directory: - - .. code-block:: bash - - sudo mkdir -p /etc/systemd/system/docker.service.d - -#. Create :file:`/etc/systemd/system/docker.service.d/http-proxy.conf` and - add the following using your own proxy values: - - .. code-block:: console - - [Service] - Environment="HTTP_PROXY=:" - Environment="HTTPS_PROXY=:" - -#. Reload the Docker daemon: - - .. code-block:: bash - - sudo systemctl daemon-reload - -Configure the Docker container proxies, to pass proxy settings to -containers: - -#. Create a directory for your container config: - - .. code-block:: bash - - mkdir ~/.docker - -#. Create the config file :file:`~/.docker/config.json` and add the following - entries, using your own proxy values: - - .. code-block:: console - - { - "proxies": - { - "default": -   { -    "httpProxy": ":", -     "httpsProxy": ":" -   } - } - } - -#. Set ownership and permission on the docker config directory: - - .. code-block:: bash - - sudo chown "$USER":"$USER" /home/"$USER"/.docker -R - sudo chmod g+rwx "$HOME/.docker" -R - -Configure proxies to allow mixer to access upstream content from behind -a firewall. - -#. Open your :file:`$HOME/.bashrc` file and add proxy and port values for the - following: - - .. code-block:: console - - export http_proxy=":" - export https_proxy=":" - export HTTP_PROXY=":" - export HTTPS_PROXY=":" - export no_proxy="<...>" - -#. Log out and log back in for the proxies to take effect. - -.. rst-class:: content-collapse - .. _set-up-nginx-web-server-start: Set up a nginx web server for mixer @@ -1136,8 +1093,8 @@ Related topics * :ref:`autospec` * :ref:`bundles-guide` * :ref:`swupd-guide` +* :ref:`swupd-3rd-party` -.. _Docker Hub: https://hub.docker.com/r/clearlinux/mixer/tags/ .. _mixer man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.1.rst .. _mixer.init man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.init.1.rst .. _mixer.bundle man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.bundle.1.rst