.. _mixer:

mixer
#####

**mixer** is the tool used by the |CL-ATTR| team to generate official update
content and releases. The update content generated by mixer is then consumed
by swupd on a downstream client. The same mixer tool is available as part of
|CL| to create your own customized update content and releases.

.. contents::
   :local:
   :depth: 1

Description
***********

mixer uses the following sources as inputs to generate update content:

* Upstream |CL| bundles with their corresponding RPM packages
* Locally-defined bundles with their corresponding local RPM packages
* Locally-defined bundles with upstream RPM packages

Using the mixer tool, you select which set of content from these sources
will be part of your update. You can select content from each of these sources to make a unique combination of functionality for your custom update content, known as a **mix**.

The update content that mixer generates consists of various pieces of OS
content, update metadata, as well as a complete image. The OS content
includes all files in an update, as well as zero- and delta-packs for improved update performance. The update metadata, stored as manifests, describes all of the bundle information for the update. Update content produced by mixer is then published to a web server and consumed by clients via swupd. Refer to :ref:`swupd <swupd-guide>` for additional information regarding updates and update content.

How it works
************

Learn the mixer tool set up and workflow.

.. contents::
   :local:
   :depth: 1

Prerequisites
=============

* :command:`mixer` bundle

  Add the mixer tool with the :command:`mixer` bundle. Refer to
  :ref:`swupd-guide` for more information on installing bundles.

* Docker\* container

  mixer by default runs all build commands in a Docker container to ensure
  the correct tool versions are used. This also allows custom mixes to
  automatically perform downstream format bumps when the upstream releases a
  format bump. See `Format version`_ for additional information regarding
  format bumps.

  Refer to `Configure and enable Docker`_ for instruction.

* Docker proxy (optional)

  If you use a proxy server, you must set your proxy environment variables and
  create a proxy configuration file for the Docker daemon and container.

  Consult your IT department for the correct values if you are behind a
  corporate proxy.

  Refer to `Configure Docker proxy info`_ for instruction.

* Location to host the update content and images

  In order for swupd to make use of your mix, the update content for your mix
  must be hosted on a web server. Your mix will be configured with an update
  location URL, which swupd will use to pull down updates.

  Refer to `Set up a nginx web server for mixer`_ for an simple example of
  setting up an update location.

Mix setup
==========

Follow these steps to create and initialize the mixer workspace. Complete
the setup before you create a mix.

#. Create workspace.

   The mixer tool uses a simple workspace to contain all input and output in a
   basic directory structure. The workspace is simply an empty folder from
   which you execute the mixer commands. Each mix uses its own separate
   workspace.

#. Initialize the workspace and mix.

   Before you create a mix, you must explicitly initialize the mixer workspace.
   During initialization, the mixer workspace is configured and the base for
   your mix is defined. By default, your mix is based on the latest
   upstream version and starts with the minimum set of bundles. Your first custom
   mix version number starts at 10. Alternatively, you can select other
   versions or bundle sets from which to start.

   Initialization creates the directory structure within the workspace and adds
   the :file:`builder.conf` file, which is used to configure the mixer tool.

   View the `mixer.init man page`_ for more information on mixer
   initialization.

   View the list of suitable `releases`_ from which to mix.

#. Edit builder.conf.

   :file:`builder.conf` tells the mixer tool how to configure the mix. For
   example, it allows you to configure where mixer output is located and where
   swupd update content will be located.

   At minimum, set the URL of your update server so your custom OS knows where
   to get update content.

   Refer to the `builder.conf`_ section for more information.

Create a mix
============

A mix is created with the following steps:

#. Add custom RPMs and set up local repo (optional).

   If you are adding custom RPMs to your mix, you must add the RPMs to
   your mix workspace and set up a corresponding local repository.

   Go to the :ref:`autospec<autospec>` guide to learn to build RPMs from
   scratch. If the RPMs are not built on |CL|, make sure your
   configuration and toolchain builds them correctly for |CL|. Otherwise there
   is no guarantee they will be compatible.

   Refer to the :ref:`autospec` guide for more information on using autospec to
   build RPMs.

#. Update and build bundles.

   Add, edit, or remove bundles that will be part of your content and build
   them. mixer automatically updates the :file:`mixbundles` file when you
   update the bundles in your mix.

   View the `mixer.bundle man page`_ for more information on configuring bundles
   in a mix.

   View the `mixer.build man page`_ for more information on building bundles.

   View the `Bundles`_ section for more information on how mixer manages
   bundles.

#. Create the update content.

   mixer creates update content with this step. Zero-packs are created
   automatically, and delta-packs can be optionally created at the same time
   (for all builds after version 0).

   A zero-pack is the full set of content needed to go from mix version 0
   (nothing) to the mix version for which you just built content.

   A delta-pack provides the content *delta* between a `PAST_VERSION` to a
   `MIX_VERSION` that allows the transition from one mix version to another.

   View :ref:`swupd-guide`  for more information on update content.

#. Create image.

   mixer creates a bootable image from your updated content using
   the :ref:`ister` tool. In this step you can specify which bundles you want
   *preinstalled* in the image. Users can later install other bundles available
   in your mix.

#. Make update available.

   Deploy update content and images to your update server.

   View the `Example 3: Deploy updates to target`_ for a simple deployment
   scenario.

Maintain or modify mix
======================

Update or modify your content to a new version by following the steps to
create a mix. Increment the mix version number for the next mix.

Examples
********

The following examples are designed to work together and in order. The examples
use:

* A stock installation of |CL|.
* A web server that comes with |CL| to host the content updates.
* A simple VM that updates against the locally produced content created in
  Example 2.

Complete all `Prerequisites`_ before using these examples.

Example 1: Mix set up
======================

This example shows the basic steps for the first-time setup of 
mixer for a new mix.

#. Create an empty directory to use as a workspace for mixer:

   .. code-block:: bash

      mkdir ~/mixer

#. In your mixer workspace, generate an initial mix based on the latest upstream
   |CL| version, with minimum bundles. In the initialization output, be aware 
   that your initial mix version is set to 10 and that the minimum bundles have 
   been added.

   .. code-block:: bash

      cd ~/mixer
      mixer init

#. Edit :file:`builder.conf` to set the value of CONTENTURL and VERSIONURL to
   the IP address of  the nginx\* server you set up in the prerequisite
   `Set up a nginx web server for mixer`_. For example:

   .. code-block:: console

      CONTENTURL="http://192.168.25.52"
      VERSIONURL="http://192.168.25.52"

Example 2: Create a simple mix
==============================

This example shows how to create a simple custom mix using upstream content.
We'll create an image for a QEMU virtual machine that we can use later to test
our mix.

We can use the default bundles that were added during initialization, but these
include the :command:`native-kernel` bundle that is intended to be used on a
bare metal system instead of a VM. So we will modify the default bundle
set to get a smaller kernel image, which will also be faster to load.

#. Update bundles in mix:

   .. code-block:: bash

      mixer bundle remove kernel-native
      mixer bundle add kernel-kvm

#. In this case, we will add the `editors` bundle from upstream, but we will
   remove the `joe` editor.

   .. code-block:: bash

      mixer bundle add editors
      mixer bundle edit editors

#. Use an editor and manually remove `joe` from the bundle definition.

   .. code-block:: bash

      $EDITOR ./local-bundles/editors

#. List the bundles in the mix again to confirm removal.

   .. code-block:: bash

      mixer bundle list  --tree


#. Build bundles:

   .. code-block:: bash

      mixer build bundles

   Look in ~/mixer/update/image/<mix version>/full for the full chroot after the
   :command:`build` command completes.

#. Build update content. Browse to your \http://localhost site and you'll see
   the web page is now up, but with no update content. Build the update content:

   .. code-block:: bash

      mixer build update

   Refresh your \http://localhost site and now you can see the update
   content for mix version 10.

   Look in ~/mixer/update/www/<mix version> to see the update content in your
   workspace.

#. Configure image. Edit the ister configuration file for your image to include
   all of the bundles you want preinstalled in the image. If this is the first
   time creating an image, first get a copy of the
   :file:`release-image-config.json` template file:

   .. code-block:: bash

      curl -O https://raw.githubusercontent.com/bryteise/ister/master/release-image-config.json

   For this example, edit :file:`release-image-config.json` so that the root
   partition size is "5G" and replace the "kernel-native" bundle with
   "kernel-kvm".

   .. code-block:: console

      {
        "DestinationType" : "virtual",
        "PartitionLayout" : [ { "disk" : "release.img", "partition" : 1, "size" : "32M", "type" : "EFI" },
                              { "disk" : "release.img", "partition" : 2, "size" : "16M", "type" : "swap" },
                              { "disk" : "release.img", "partition" : 3, "size" : "5G", "type" : "linux" } ],
        "FilesystemTypes" : [ { "disk" : "release.img", "partition" : 1, "type" : "vfat" },
                              { "disk" : "release.img", "partition" : 2, "type" : "swap" },
                              { "disk" : "release.img", "partition" : 3, "type" : "ext4" } ],
        "PartitionMountPoints" : [ { "disk" : "release.img", "partition" : 1, "mount" : "/boot" },
                                   { "disk" : "release.img", "partition" : 3, "mount" : "/" } ],
        "Version": "latest",
        "Bundles": ["kernel-kvm", "os-core", "os-core-update"]
      }

#. Build the image.

   .. code-block:: bash

      sudo mixer build image

   The output from this step will be :file:`release.img`, which is a live image.

#. Make the next mix. Create a new version of your mix, for the live image to
   update to. Increment your mix version by 10:

   .. code-block:: bash

      mixer versions update

   Repeat steps 1-3 to add the upstream :command:`curl` bundle to the mix:

   .. code-block:: bash

      mixer bundle add curl
      mixer build bundles
      mixer build update

   Build optional delta-packs, which helps reduce client update time:

   .. code-block:: bash

      mixer build delta-packs --from 10 --to 20

   Refresh your \http://localhost site to see the update content for
   mix version 20.

   Look in ~/mixer/update/www/<mix version> to see the update content in your
   workspace.

Example 3: Deploy updates to target
===================================

The image created in Example 2 is directly bootable in QEMU. In this example,
we'll boot the image from Example 2 to verify it, and update the image from
mix version 10 (from which the image was built), to mix version 20.

#. Set up the QEMU environment.

   Install the :command:`kvm-host` bundle to your |CL|:

   .. code-block:: bash

      sudo swupd bundle-add kvm-host

   Get the virtual EFI firmware, download the image launch script, and make it
   executable:

   .. code-block:: bash

      curl -O https://download.clearlinux.org/image/OVMF.fd
      curl -O https://download.clearlinux.org/image/start_qemu.sh
      chmod +x start_qemu.sh

#. Start your VM image (created in Example 2):

   .. code-block:: bash

      sudo ./start_qemu.sh release.img

#. Log in as root and set a password

#. Try out your mix.

   Take a look at the default bundles installed in your mix:

   .. code-block:: bash

      swupd info
      swupd bundle-list
      swupd bundle-list -a

#. Now we will add the `editors` bundle that we modified.

   .. code-block:: bash

      swupd bundle add editors

#. Try to start the `joe` editor.  It should not appear because we removed it
   from the original `editors` bundle.

#. Next we will update from version 10 to 20 to capture the newly
   available bundles. Use :command:`swupd` to update your mix:

   .. code-block:: bash

      swupd check-update
      swupd update
      swupd bundle-list -a

#. Now your mix should be at version 20 and curl is now available. Try using
   curl. This will fail because curl is not yet installed:

   .. code-block:: console

      curl: command not found
      To install curl use: swupd bundle-add curl

   Add the new bundle from your update server to your VM. Retry curl. It works!

   .. code-block:: bash

      swupd bundle-add curl
      curl -O https://download.clearlinux.org/image/start_qemu.sh

   Shutdown your VM:

   .. code-block:: bash

      poweroff


.. Example: Create a mix with custom RPM
.. -------------------------------------
.. TODO future example to show copy into local-rpms...

References
**********

Reference the `mixer man page`_ for details regarding mixer commands and options.

.. contents::
   :local:
   :depth: 1

.. rst-class:: content-collapse

builder.conf
============

mixer initialization creates a :file:`builder.conf` that stores the basic
configuration for the mixer tool. The items of primary interest are CONTENTURL
and VERSIONURL, which will be used by systems updating against your custom
content.

.. code-block:: console

   #builder.conf

   #VERSION 1.0

   [Builder]
     CERT = "/home/clr/mix/Swupd_Root.pem"
     SERVER_STATE_DIR = "/home/clr/mix/update"
     VERSIONS_PATH = "/home/clr/mix"
     YUM_CONF = "/home/clr/mix/.yum-mix.conf"

   [Swupd]
     BUNDLE = "os-core-update"
     CONTENTURL = "<URL where the content will be hosted>"
     VERSIONURL = "<URL where the version of the mix will be hosted>"

   [Server]
     DEBUG_INFO_BANNED = "true"
     DEBUG_INFO_LIB = "/usr/lib/debug"
     DEBUG_INFO_SRC = "/usr/src/debug"

   [Mixer]
     LOCAL_BUNDLE_DIR = "/home/clr/mix/local-bundles"
     LOCAL_REPO_DIR = ""
     LOCAL_RPM_DIR = ""
     DOCKER_IMAGE_PATH = "clearlinux/mixer"

Additional explanation of variables in :file:`builder.conf` is provided in Table
1.

+-------------------------------+----------------------------------------------------------+
| **Variable**                  | **Explanation**                                          |
+-------------------------------+----------------------------------------------------------+
| `CERT`                        | Sets the path where mixer stores the certificate file    |
|                               | used to sign content for verification. mixer             |
|                               | automatically generates the certificate if you do not    |
|                               | provide the path to an existing one, and signs the       |
|                               | :file:`Manifest.MoM` file to provide security for the    |
|                               | updated content you create.                              |
|                               |                                                          |
|                               | chroot-builder uses the certificate file to sign         |
|                               | the root :file:`Manifest.MoM` file to provide            |
|                               | security for content verification.                       |
|                               |                                                          |
|                               | swupd uses this certificate to verify the                |
|                               | :file:`Manifest.MoM` file's signature.                   |
|                               |                                                          |
|                               | For now, we strongly recommend that you do not modify    |
|                               | this variable, as swupd expects a certificate with a     |
|                               | very specific configuration to sign and verify           |
|                               | properly.                                                |
+-------------------------------+----------------------------------------------------------+
| `CONTENTURL` and `VERSIONURL` | Set these variables to the IP address of the web server  |
|                               | hosting the update content.                              |
|                               |                                                          |
|                               | VERSIONURL is the IP address where the swupd client      |
|                               | looks to determine if a new version is available.        |
|                               |                                                          |
|                               | CONTENTURL is the location from which swupd pulls        |
|                               | content updates.                                         |
|                               |                                                          |
|                               | If the web server is on the same machine as the          |
|                               | SERVER_STATE_DIR directory, you can create a symlink to  |
|                               | the directory in your web server's document root to      |
|                               | easily host the content.                                 |
|                               |                                                          |
|                               | These URLs are embedded in the images created by mixer.  |
+-------------------------------+----------------------------------------------------------+
| `DOCKER_IMAGE_PATH`           | Sets the base name of the docker image that mixer pulls  |
|                               | down to run builds in the proper container.              |
+-------------------------------+----------------------------------------------------------+
| `LOCAL_BUNDLE_DIR`            | Sets the path where mixer stores the local bundle        |
|                               | definition files. The bundle definition files include    |
|                               | any new, original bundles you create, along with any     |
|                               | edited versions of upstream bundles.                     |
+-------------------------------+----------------------------------------------------------+
| `SERVER_STATE_DIR`            | Sets the path to which mixer outputs content. By         |
|                               | default, mixer automatically sets the path.              |
+-------------------------------+----------------------------------------------------------+
| `VERSIONS_PATH`               | Sets the path for the mix version and upstream version's |
|                               | two state files: :file:`mixversion` and                  |
|                               | :file:`upstreamversion`. mixer creates both files for    |
|                               | you when you set up the workspace.                       |
+-------------------------------+----------------------------------------------------------+
| `YUM_CONF`                    | Sets the path where mixer automatically generates the    |
|                               | :file:`.yum-mix.conf` file.                              |
|                               |                                                          |
|                               | The yum configuration file points the chroot-builder to  |
|                               | where the RPMs are stored.                               |
+-------------------------------+----------------------------------------------------------+
| **Table 1**: *Variables in builder.conf*                                                 |
+-------------------------------+----------------------------------------------------------+

Format version
--------------

Compatible versions of an OS are tracked with an OS *compatibility epoch*.
Versions of an OS within an epoch are fully compatible and can update to any
other version within that epoch. The compatibility epoch is set as the
`Format` variable in the :file:`mixer.state` file. Variables in the
:file:`mixer.state` are used by mixer between executions and should not be
manually changed.

A format bump is like modifying the foundation of a house to create a new
level. If `Format` increments to a new epoch (a "format bump"), the OS has
changed in such a way that updating from build A in format X to build B in
format Y will not work.

A format bump is required when:

* The software updater, :command:`swupd`, or the software is no longer
  compatible with the previous update scheme

* A package is removed from the update stream and the update must ensure the
  files associated with that package are removed from the system

Using a format increment, we make sure pre- and co-requisite changes flow out
with proper ordering. The updated client will only update to the latest
release in its respective format version, unless overridden by command line
flags. In this way, we can guarantee that all clients update to the final
version in their given format.

The given format *must* contain all the changes needed to understand the content built in the next format. Only after reaching the final release in the old format can a client continue to update to releases in the new format.

The format version is incremented only when a compatibility breakage is
introduced. Normal updates, such as updating a software package, do not require a format increment.

.. rst-class:: content-collapse

Bundles
=======

mixer stores information about the bundles included in a mix in a flat file
called :file:`mixbundles`, which is located in the path set by the VERSIONS_PATH variable in :file:`builder.conf`. :file:`mixbundles` is automatically created when the mix is initiated. mixer will refresh the file each time you change the bundles in the mix.

Bundles can include other bundles. Nested bundles can themselves include
other bundles. If you see an unexpected bundle in your mix, it is likely a
nested bundle in one of the bundles you explicitly added.

A bundle will fill into one of two categories: upstream or local. Upstream
bundles are those provided by |CL|. Local bundles are either modified upstream bundles or new local bundles.

Upstream bundles
----------------

mixer automatically downloads and caches upstream bundle definition files.
These definition files are stored in the upstream-bundles directory in the
workspace. Do not modify the files in this directory. This directory is
simply a mirror for mixer to use. mixer will automatically delete the
contents of this directory before repopulating it on-the-fly if a new
version must be downloaded.

The mixer tool automatically caches the bundles for the |CL| version
configured in the :file:`upstreamversion` file. mixer also cleans up old
versions once they are no longer needed.

Local bundles
-------------

Local bundles are bundles that you create, or are edited versions of upstream
bundles. Local bundle definition files are stored in the local-bundles
directory in the workspace. The LOCAL_BUNDLE_DIR variable sets the path of this directory in the :file:`builder.conf` file.

*mixer always checks for local bundles first and the upstream bundles
second.* So bundles in the local-bundles directory will always take
precedence over any upstream bundles that have the same name. This
precedence enables you to copy upstream bundles locally, and edit into a
local variation.

Bundle configuration
--------------------

mixer provides commands to configure the bundles for a mix, such as to add a
bundle to a mix, to create a new bundle for a mix, or to remove a bundle from a
mix. View the `mixer.bundle man page`_ for a full list of commands and more
information on configuring bundles in a mix.

Editing an existing local bundle is as simple as opening the bundle definition
file in your favorite editor, making the desired edits, and saving your changes.

.. note::

   Removing bundles from a mix: By default, removing a bundle will only
   remove the bundle from the mix. The local bundle definition file will
   still remain. To completely remove a bundle, including its local bundle definition file, use the :command:`--local` flag.

   If you remove the bundle definition file for a local, edited version of an
   upstream bundle in a mix, the mix reverts to reference the original upstream version of the bundle.

.. rst-class:: content-collapse

Configure and enable Docker
===========================

Use these steps to enable Docker for the mixer tool. Make sure to
`Configure Docker proxy info`_ first if needed.

#. Start the Docker daemon:

   .. code-block:: bash

      sudo systemctl start docker
      sudo chmod 777 /var/run/docker.sock
      sudo docker info

#. Add user to the docker group

   .. code-block:: bash

      sudo usermod -G docker -a <username>

Pull Docker container manually (optional)
-----------------------------------------

By default, mixer automatically pulls a Docker container for mixing if one
does not already exist. If you need to troubleshoot the mixer container, it
may be useful to manually pull a mixer Docker container.

Versions of the mixer Docker container are available under the tags for the
`clearlinux/mixer repo <https://hub.docker.com/r/clearlinux/mixer/tags/>`_
on Docker Hub. Each version of the mixer Docker container is named after the
associated |CL| upstream format version. Refer to `Format version`_ for
additional information on upstream format versions.

Use the following steps to manually pull a mixer Docker container:

#. Find the version of the container you need by viewing the tags for the
   `clearlinux/mixer repo <https://hub.docker.com/r/clearlinux/mixer/tags/>`_
   on Docker Hub.

#. Pull the latest container version:

   .. code-block:: bash

      docker pull clearlinux/mixer:<upstream-format-version>

#. View local docker images:

   .. code-block:: bash

      docker images

.. rst-class:: content-collapse

Configure Docker proxy info
===========================

If needed, use these steps to configure the Docker proxy information.

#. Create the Docker daemon proxy config directory:

   .. code-block:: bash

      sudo mkdir -p /etc/systemd/system/docker.service.d

#. Create :file:`/etc/systemd/system/docker.service.d/http-proxy.conf` and
   add the following using your own proxy values:

   .. code-block:: console

      [Service]
      Environment="HTTP_PROXY=<HTTP proxy URL>:<port number>"
      Environment="HTTPS_PROXY=<HTTPS proxy URL>:<port number>"

#. Reload the Docker daemon:

   .. code-block:: bash

      sudo systemctl daemon-reload

Configure the Docker container proxies, to pass proxy settings to
containers:

#. Create a directory for your container config:

   .. code-block:: bash

      mkdir ~/.docker

#. Create the config file :file:`~/.docker/config.json` and add the following
   entries, using your own proxy values:

   .. code-block:: console

      {
        "proxies":
        {
          "default":
          {
            "httpProxy": "<proxy-url>:<port>",
            "httpsProxy": "<proxy-url>:<port>"
          }
        }
      }

#. Set ownership and permission on the docker config directory:

   .. code-block:: bash

      sudo chown "$USER":"$USER" /home/"$USER"/.docker -R
      sudo chmod g+rwx "$HOME/.docker" -R

Configure proxies to allow mixer to access upstream content from behind
a firewall.

#. Open your :file:`$HOME/.bashrc` file and add proxy and port values for the
   following:

   .. code-block:: console

      export http_proxy="<proxy-url>:<port>"
      export https_proxy="<proxy-url>:<port>"
      export HTTP_PROXY="<proxy-url>:<port>"
      export HTTPS_PROXY="<proxy-url>:<port>"
      export no_proxy="<...>"

#. Log out and log back in for the proxies to take effect.

.. rst-class:: content-collapse

Set up a nginx web server for mixer
===================================

A web server is needed to host your update content. In this example, we use
the nginx web server, which comes with |CL|.

Set up a nginx web server for mixer with the following steps:

#. Install the :command:`nginx` bundle:

   .. code-block:: bash

      sudo swupd bundle-add nginx

#. Make the directory where mixer updates will reside:

   .. code-block:: bash

      sudo mkdir -p /var/www

#. Create a symbolic link between your workspace updates and the updates on
   the local nginx web server. In this example, `$HOME/mixer` is the
   workspace for the mix.

   .. code-block:: bash

      sudo ln -sf $HOME/mixer/update/www /var/www/mixer

#. Set up ``nginx`` configuration:

   .. code-block:: bash

      sudo mkdir -p  /etc/nginx/conf.d

#. Copy the default example configuration file:

   .. code-block:: bash

      sudo cp -f /usr/share/nginx/conf/nginx.conf.example /etc/nginx/nginx.conf

#. Configure the mixer update server. Create and add the following server
   configuration content to :file:`/etc/nginx/conf.d/mixer.conf` (sudo required):

   .. code-block:: console

      server {
           server_name localhost;
           location / {
                     root /var/www/mixer;
                     autoindex on;
           }
      }

#. Restart the daemon, enable nginx on boot, and start the service.

   .. code-block:: bash

      sudo systemctl daemon-reload

      sudo systemctl enable nginx

      sudo systemctl start nginx

#. Verify the web server is running at \http://localhost. At this point
   you should no longer see a "404 Not Found" message.

Related topics
**************

* :ref:`autospec`
* :ref:`bundles-guide`
* :ref:`swupd-guide`

.. _Docker Hub: https://hub.docker.com/r/clearlinux/mixer/tags/
.. _mixer man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.1.rst
.. _mixer.init man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.init.1.rst
.. _mixer.bundle man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.bundle.1.rst
.. _mixer.build man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.build.1.rst
.. _releases: https://github.com/clearlinux/clr-bundles/releases
