clearlinux/clear-linux-documentation
2
0
Fork 0
mirror of https://github.com/clearlinux/clear-linux-documentation.git synced 2026-08-25 01:06:37 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #438 from intelkevinputnam/rtd-theme

Browse Source 
Updates to RTD themed version of documentation
This commit is contained in:
Kevin Putnam
2019-04-02 16:00:50 -07:00
committed by GitHub
parent 92c79d862c 101020bd84
commit e8b6ba46ba
12 changed files with 1213 additions and 1201 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/clear-linux/get-started/bootable-usb/bootable-usb.rst
+6 -2
View File
@@ -78,7 +78,7 @@ Burn the |CL| image onto a USB drive
├─sda8 8:8 0 30G 0 part /
└─sda6 8:6 0 7.9G 0 part [SWAP]
#. You must unmount a USB drive before you can burn an image onto it. Note that
#. You must unmount a USB drive before burning an image onto it. Note that
some Linux distros automatically mount a USB drive when it is plugged in.
Unmount a USB drive with the :command:`umount` command followed by the device
identifier/partition. For example:
@@ -93,7 +93,11 @@ Burn the |CL| image onto a USB drive
.. code-block:: bash
dd if=./clear-[version number]-[image type] of=<your USB device> bs=4M status=progress && sync
dd if=./clear-[version number]-[image type] of=<your USB device> oflag=sync bs=4M status=progress
.. caution::
|CAUTION-UNMOUNT-USB-PARTITIONS|
.. _bootable-usb-mac:
source/clear-linux/guides/maintenance/autospec.rst → source/clear-linux/guides/clearlinux/autospec.rst
View File
source/clear-linux/guides/maintenance/ister.rst → source/clear-linux/guides/clearlinux/ister.rst
View File
source/clear-linux/guides/clearlinux/mixer.rst
+843
View File
@@ -0,0 +1,843 @@
.. _mixer:
mixer
#####
mixer is the tool used by the |CL-ATTR| team to generate official update content
and releases. The update content generated by mixer is then consumed by swupd on
a downstream client. The same mixer tool is available as part of |CL| to create
your own customized update content and releases.
.. contents::
:local:
:depth: 1
Description
***********
mixer uses the following sources as inputs to generate update content:
* Upstream |CL| bundles with their corresponding RPM packages
* Locally-defined bundles with their corresponding local RPM packages
* Locally-defined bundles with upstream RPM packages
Using the mixer tool, you select which set of content from these sources will be
part of your update. You can select content from each of these sources to make a
unique combination of functionality for your custom update content (known as a
mix).
The update content that mixer generates consists of various pieces of OS
content, update metadata, as well as a complete image. The OS content includes
all files in an update, as well as zero- and delta-packs for improved update
performance. The metadata, stored as manifests, describes all of the bundle
information for the update. Update content produced by mixer is then published
to a web server and consumed by clients via swupd. Refer to
:ref:`swupd <swupd-guide>` for additional information regarding updates and
update content.
How it works
************
Learn the mixer tool set up and workflow.
.. contents::
:local:
:depth: 1
Prerequisites
=============
* :command:`mixer` bundle
Add the mixer tool with the :command:`mixer` bundle. Refer to
`Install a bundle`_ for more details.
* Docker container
mixer by default runs all build commands in a Docker container to make sure
the correct tool versions are used. This also allows custom mixes to
automatically perform downstream format bumps when the upstream releases a
format bump. See `Format version`_ for additional information regarding
format bumps.
Refer to `Configure and enable Docker`_ for instruction.
* Docker proxy (optional)
If you use a proxy server, you must set your proxy environment variables and
create a proxy configuration file for the Docker daemon and container.
Consult your IT department if you are behind a corporate proxy for the correct
values.
Refer to `Configure Docker proxy info`_ for instruction.
* Location to host the update content and images
In order for swupd to make use of your mix, the update content for your mix
must be hosted on a web server. Your mix will be configured with an update
location URL, which swupd will use to pull down updates.
Refer to `Set up a nginx web server for mixer`_ for an simple example of
setting up an update location.
Mix setup
==========
Follow these steps to create and initialize the mixer workspace. Complete
setup before you create a mix.
#. Create workspace.
The mixer tool uses a simple workspace to contain all input and output in a
basic directory structure. The workspace is simply an empty folder that you
will execute the mixer commands from. Each mix will use its own separate
workspace.
#. Initialize the workspace and mix.
Before you create a mix, you must explicitly initialize the mixer workspace.
During initialization, the mixer workspace is configured and the base for
your mix is defined. By default, your mix will be based on the latest
upstream version and start with the minimum set of bundles. Your first custom
mix version number will start at 10. You can alternately select other
versions or bundle sets to start from.
Initialization creates the directory structure within the workspace and adds
the :file:`builder.conf` file, which is used to configure the mixer tool.
View the `mixer.init man page`_ for more information on mixer
initialization.
View the list of `suitable versions`_ to mix from.
#. Edit builder.conf.
:file:`builder.conf` tells the mixer tool how to configure the mix. For
example, it allows you to configure where mixer output is located and where
swupd update content will be located.
At minimum, set the URL of your update server so your custom OS knows where
to get update content.
Refer to the `builder.conf`_ section for more information.
Create a mix
============
A mix is created with the following steps:
#. Add custom RPMs and set up local repo (optional).
If you are adding custom RPMs to your mix, you will need to add the RPMs to
your mix workspace and set up a corresponding local repository.
Go to the :ref:`autospec<autospec>` guide to learn to build RPMs from
scratch. If the RPMs are not built on |CL|, make sure your
configuration and toolchain builds them correctly for |CL|. Otherwise there
is no guarantee they will be compatible.
Refer to the :ref:`autospec` guide for more information on using autospec to
build RPMs.
#. Update and build bundles.
Add, edit, or remove bundles that will be part of your content and build
them. mixer will automatically update the :file:`mixbundles` file when you
update the bundles in your mix.
View the `mixer.bundle man page`_ for more information on configuring bundles
in a mix.
View the `mixer.build man page`_ for more information on building bundles.
View the `Bundles`_ section for more information on how mixer manages
bundles.
#. Create the update content.
mixer creates update content with this step. Zero-packs are created
automatically, and delta-packs can be optionally created at the same time
(for all builds after version 0).
A zero-pack is the full set of content needed to go from mix version 0
(nothing) to the mix version you just built content for.
A delta-pack provides the content *delta* between a `PAST_VERSION` to a
`MIX_VERSION` which allows the transition from one mix version to another.
View :ref:`swupd-guide` for more information on update content.
#. Create image.
mixer creates a bootable image from your updated content using
the ister tool. In this step you can specify which bundles you want
*preinstalled* in the image. Users can later install other bundles available
in your mix.
#. Make update available.
Deploy update content and images to your update server.
View the `Example 3: Deploy updates to target`_ for a simple deployment
scenario.
Maintain or modify mix
======================
Update or modify your content to a new version by following the same steps to
create a mix. Increment the mix version number for the next mix.
Examples
********
The following examples are designed to work together and in order. The examples
use:
* A stock installation of |CL|.
* A web server that comes with |CL| to host the content updates.
* A simple VM that will update against the locally produced content created in
Example 2.
Complete all `Prerequisites`_ before using these examples.
Example 1: Mix set up
======================
This example shows the basic steps for first time setup of mixer for a new mix.
#. Create an empty directory to use as a workspace for mixer:
.. code-block:: bash
mkdir ~/mixer
#. In your mixer workspace, generate an initial mix based on the latest upstream
|CL| version, with minimum bundles:
.. code-block:: bash
cd ~/mixer
mixer init
Note in the initialization output, that your initial mix version is set to
10 and that the minimum bundles have been added.
#. Edit :file:`builder.conf` to set the value of CONTENTURL and VERSIONURL to
the IP address of the nginx server you set up in the prerequisite
`Set up a nginx web server for mixer`_. For example:
.. code-block:: console
CONTENTURL="http://192.168.25.52"
VERSIONURL="http://192.168.25.52"
Example 2: Create a simple mix
==============================
This example shows how to create a simple custom mix using upstream content.
We'll create an image for a QEMU virtual machine which we can later use to test
our mix.
We can use the default bundles that were added during intialization, but these
include the :command:`native-kernel` bundle which is intended to be used on a
bare metal system instead of a VM. So we will modify the default bundle
set to get a smaller kernel image, which will also be faster to load.
#. Update bundles in mix:
.. code-block:: bash
mixer bundle remove kernel-native
mixer bundle add kernel-kvm
#. Build bundles:
.. code-block:: bash
mixer build bundles
Look in ~/mixer/update/image/<mix version>/full for the full chroot after the
:command:`build` command completes.
#. Build update content. Browse to your http://localhost site and you'll see
the web page is now up, but with no update content. Build the update content:
.. code-block:: bash
mixer build update
Refresh your http://localhost site and now you can see the update content for
mix version 10.
Look in ~/mixer/update/www/<mix version> to see the update content in your
workspace.
#. Configure image. Edit the ister configuration file for your image to include
all of the bundles you want preinstalled in the image. If this is the first
time creating an image, first get a copy of the
:file:`release-image-config.json` template file:
.. code-block:: bash
curl -O https://raw.githubusercontent.com/bryteise/ister/master/release-image-config.json
For this example, edit :file:`release-image-config.json` so that the root
partition size is "5G" and replace the "kernel-native" bundle with
"kernel-kvm".
.. code-block:: console
{
"DestinationType" : "virtual",
"PartitionLayout" : [ { "disk" : "release.img", "partition" : 1, "size" : "32M", "type" : "EFI" },
{ "disk" : "release.img", "partition" : 2, "size" : "16M", "type" : "swap" },
{ "disk" : "release.img", "partition" : 3, "size" : "5G", "type" : "linux" } ],
"FilesystemTypes" : [ { "disk" : "release.img", "partition" : 1, "type" : "vfat" },
{ "disk" : "release.img", "partition" : 2, "type" : "swap" },
{ "disk" : "release.img", "partition" : 3, "type" : "ext4" } ],
"PartitionMountPoints" : [ { "disk" : "release.img", "partition" : 1, "mount" : "/boot" },
{ "disk" : "release.img", "partition" : 3, "mount" : "/" } ],
"Version": "latest",
"Bundles": ["kernel-kvm", "os-core", "os-core-update"]
}
#. Build the image.
.. code-block:: bash
sudo mixer build image
The output from this step will be :file:`release.img`, which is a live image.
#. Make the next mix. Create a new version of your mix, for the live image to
update to. Increment your mix version by 10:
.. code-block:: bash
mixer versions update
Repeat steps 1-3 to add the upstream :command:`curl` bundle to the mix:
.. code-block:: bash
mixer bundle add curl
mixer build bundles
mixer build update
And build optional delta-packs, which will help reduce client update time:
.. code-block:: bash
mixer build delta-packs --from 10 --to 20
Refresh your http://localhost site and now you can see the update content for
mix version 20.
Look in ~/mixer/update/www/<mix version> to see the update content in your
workspace.
Example 3: Deploy updates to target
===================================
The image created in Example 2 is directly bootable in QEMU. In this example,
we'll boot the image from Example 2 to verify it, and update the image from mix
version 10 (which the image was built from), to mix version 20.
#. Set up the QEMU environment.
Install the :command:`kvm-host` bundle to your |CL|:
.. code-block:: bash
sudo swupd bundle-add kvm-host
Get the virtual EFI firmware, download the image launch script, and make it
executable:
.. code-block:: bash
curl -O https://download.clearlinux.org/image/OVMF.fd
curl -O https://download.clearlinux.org/image/start_qemu.sh
chmod +x start_qemu.sh
#. Start your VM image (created in Example 2):
.. code-block:: bash
sudo ./start_qemu.sh release.img
#. Log in as root and set a password
#. Try out your mix.
Take a look at the default bundles installed in your mix:
.. code-block:: bash
swupd info
swupd bundle-list
swupd bundle-list -a
Note that you cannot see the curl bundle that you added in Example 2 because
your mix is still on version 10.
Check for updates. You should see that version 20 is available. Use swupd to
update your mix:
.. code-block:: bash
swupd check-update
swupd update
swupd bundle-list -a
Now your mix should be at version 20 and curl is now available. Try using
curl. This will fail as curl is not yet installed:
.. code-block:: console
curl: command not found
To install curl use: swupd bundle-add curl
Add the new bundle from your update server to your VM. Retry curl. It works!
.. code-block:: bash
swupd bundle-add curl
curl -O https://download.clearlinux.org/image/start_qemu.sh
And shutdown your VM:
.. code-block:: bash
poweroff
.. Example: Create a mix with custom RPM
.. -------------------------------------
.. TODO future example to show copy into local-rpms...
References
**********
Reference the `mixer man page`_ for details regarding mixer commands and options.
.. contents::
:local:
:depth: 1
.. rst-class:: content-collapse
builder.conf
============
mixer initialization creates a :file:`builder.conf` that stores the basic
configuration for the mixer tool. The items of primary interest are CONTENTURL
and VERSIONURL, which will be used by systems updating against your custom
content.
.. code-block:: console
#builder.conf
#VERSION 1.0
[Builder]
CERT = "/home/clr/mix/Swupd_Root.pem"
SERVER_STATE_DIR = "/home/clr/mix/update"
VERSIONS_PATH = "/home/clr/mix"
YUM_CONF = "/home/clr/mix/.yum-mix.conf"
[Swupd]
BUNDLE = "os-core-update"
CONTENTURL = "<URL where the content will be hosted>"
VERSIONURL = "<URL where the version of the mix will be hosted>"
[Server]
DEBUG_INFO_BANNED = "true"
DEBUG_INFO_LIB = "/usr/lib/debug"
DEBUG_INFO_SRC = "/usr/src/debug"
[Mixer]
LOCAL_BUNDLE_DIR = "/home/clr/mix/local-bundles"
LOCAL_REPO_DIR = ""
LOCAL_RPM_DIR = ""
DOCKER_IMAGE_PATH = "clearlinux/mixer"
Additional explanation of variables in :file:`builder.conf` is provided in Table
1.
+-------------------------------+----------------------------------------------------------+
| **Variable** | **Explanation** |
+-------------------------------+----------------------------------------------------------+
| `CERT` | Sets the path where mixer stores the certificate file |
| | used to sign content for verification. mixer |
| | automatically generates the certificate if you do not |
| | provide the path to an existing one, and signs the |
| | :file:`Manifest.MoM` file to provide security for the |
| | updated content you create. |
| | |
| | The chroot-builder uses the certificate file to sign |
| | the root :file:`Manifest.MoM` file, to provide |
| | security for content verification. |
| | |
| | The swupd uses this certificate to verify the |
| | :file:`Manifest.MoM` file's signature. |
| | |
| | For now, we strongly recommend that you do not modify |
| | this variable, as swupd expects a certificate with a |
| | very specific configuration to sign and verify |
| | properly. |
+-------------------------------+----------------------------------------------------------+
| `CONTENTURL` and `VERSIONURL` | Set these variables to the IP address of the web server |
| | hosting the update content. |
| | |
| | VERSIONURL is the IP address where the swupd client |
| | looks to determine if a new version is available. |
| | |
| | CONTENTURL is the location where swupd will pull content |
| | updates from. |
| | |
| | If the web server is on the same machine as the |
| | SERVER_STATE_DIR directory, you can create a symlink to |
| | the directory in your web server's document root to |
| | easily host the content. |
| | |
| | These URLs are embedded in the images created by mixer. |
+-------------------------------+----------------------------------------------------------+
| `DOCKER_IMAGE_PATH` | Sets the base name of the docker image mixer will pull |
| | down in order to run builds in the proper container. |
+-------------------------------+----------------------------------------------------------+
| `LOCAL_BUNDLE_DIR` | Sets the path where mixer stores the local bundle |
| | definition files. The bundle definition files include |
| | any new, original bundles you create, along with any |
| | edited versions of upstream bundles. |
+-------------------------------+----------------------------------------------------------+
| `SERVER_STATE_DIR` | Sets the path for where mixer outputs content. By |
| | default, mixer will automatically set the path. |
+-------------------------------+----------------------------------------------------------+
| `VERSIONS_PATH` | Sets the path for the mix version and upstream version's |
| | two state files: :file:`mixversion` and |
| | :file:`upstreamversion`. mixer creates both files for |
| | you when you set up the workspace. |
+-------------------------------+----------------------------------------------------------+
| `YUM_CONF` | Sets the path where mixer automatically generates the |
| | :file:`.yum-mix.conf` file. |
| | |
| | The yum configuration file points the chroot-builder to |
| | where the RPMs are stored. |
+-------------------------------+----------------------------------------------------------+
| **Table 1**: *Variables in builder.conf* |
+-------------------------------+----------------------------------------------------------+
Format version
--------------
Compatible versions of an OS are tracked with an OS *compatibility epoch*.
Versions of an OS within an epoch are fully compatible and can update to any
other version within that epoch. The compatibility epoch is set as the `Format`
variable in the :file:`mixer.state` file. Variables in the :file:`mixer.state`
are used by mixer between executions and should not be manually changed.
If `Format` increments to a new epoch (a "format bump"), the OS has changed in
such a way that updating from build M in format X, to build N in format Y will
not work. Generally, this scenario occurs when the software updater/the software
has a change such that it is no longer compatible with the previous update
scheme, or when a package is removed from the update stream and the update
must ensure the files associated with that package are removed from the system.
Using a format increment, we make sure pre- and co-requisite changes flow out
with proper ordering. The updated client will only update to the latest
release in its respective format version, unless overridden by command line
flags. This way we can guarantee that all clients update to the final version
in their given format.
The given format *must* contain all the changes needed to understand the content
built in the next format. Only after reaching the final release in the old
format can a client continue to update to releases in the new format.
The format version is incremented only when a compatibility breakage is
introduced. Normal updates, like updating a software package, do not require a
format increment.
.. rst-class:: content-collapse
Bundles
=======
mixer stores information about the bundles included in a mix in a flat file
called :file:`mixbundles`, located in the path set by the VERSIONS_PATH
variable in :file:`builder.conf`. :file:`mixbundles` is automatically created
when the mix is initiated. mixer will refresh the file each time you change the
bundles in the mix.
Bundles can include other bundles. Nested bundles can themselves include other
bundles. If you see an unexpected bundle in your mix, it is likely a nested
bundle in one of the bundles you explicitley added.
A bundle will fill into one of two categoris: upstream or local. Upstream
bundles are those provided by |CL|. Local bundles are either modified upstream
bundles or new local bundles.
Upstream bundles
----------------
mixer automatically downloads and caches upstream bundle definition files. These
definition files are stored in the upstream-bundles directory in the workspace.
Do not modify the files in this directory. This directory is simply a mirror for
mixer to use. mixer will automatically delete the contents of this directory
before repopulating it on-the-fly if a new version must be downloaded.
The mixer tool automatically caches the bundles for the |CL| version configured
in the :file:`upstreamversion` file. mixer also cleans up old versions once they
are no longer needed.
Local bundles
-------------
Local bundles are bundles that you create, or are edited versions of upstream
bundles. Local bundle definition files are stored in the local-bundles directory
in the workspace. The LOCAL_BUNDLE_DIR variable sets the path of this directory
in the :file:`builder.conf` file.
*mixer always checks for local bundles first and the upstream bundles second.*
So bundles in the local-bundles directory will always take precedence over any
upstream bundles that have the same name. This precedence enables you to copy
upstream bundles locally, and edit into a local variation.
Bundle configuration
--------------------
mixer provides commands to configure the bundles for a mix, for example to add a
bundle to a mix, to create a new bundle for a mix, or to remove a bundle from a
mix. View the `mixer.bundle man page`_ for a full list of commands and more
information on configuring bundles in a mix.
Editing an existing local bundle is as simple as opening the bundle definition
file in your favorite editor, making the desired edits, and saving your changes.
A note on removing bundles from a mix: By default, removing a bundle will only
remove the bundle from the mix. The local bundle defintion file will still
remain. To completely remove a bundle, including its local bundle definition
file, use the :command:`--local` flag.
If you remove the bundle definition file for a local, edited version of an
upstream bundle in a mix, the mix will revert to reference the original upstream
version of the bundle.
.. rst-class:: content-collapse
Configure and enable Docker
===========================
Use these steps to enable Docker for the mixer tool. Make sure to
`Configure Docker proxy info`_ first if needed.
#. Start the Docker daemon:
.. code-block:: bash
sudo systemctl start docker
sudo chmod 777 /var/run/docker.sock
sudo docker info
#. Add user to the docker group
.. code-block:: bash
sudo usermod -G docker -a <username>
Pull Docker container manually (optional)
-----------------------------------------
By default, mixer will automatically pull a Docker container for mixing if one
does not already exist. If you need to troubleshoot the mixer container, it may
be useful to manually pull a mixer Docker container.
Versions of the mixer Docker container are available under the tags for the
`clearlinux/mixer repo <https://hub.docker.com/r/clearlinux/mixer/tags/>`_
on Docker Hub. Each version of the mixer Docker container is named after the
associated |CL| upstream format version. Refer to `Format version`_ for
additional information on upstream format versions.
Use the following steps to manually pull a mixer Docker container:
#. Find the version of the container you need by viewing the tags for the
`clearlinux/mixer repo <https://hub.docker.com/r/clearlinux/mixer/tags/>`_
on Docker Hub.
#. Pull the latest container version:
.. code-block:: bash
docker pull clearlinux/mixer:<upstream-format-version>
#. View local docker images:
.. code-block:: bash
docker images
.. rst-class:: content-collapse
Configure Docker proxy info
===========================
If needed, use these steps to configure the Docker proxy information.
Configure the Docker daemon proxies:
#. Create the Docker daemon proxy config directory:
.. code-block:: bash
sudo mkdir -p /etc/systemd/system/docker.service.d
Create :file:`/etc/systemd/system/docker.service.d/http-proxy.conf` and
add the following using your own proxy values:
.. code-block:: console
[Service]
Environment="HTTP_PROXY=<HTTP proxy URL>:<port number>"
Environment="HTTPS_PROXY=<HTTPS proxy URL>:<port number>"
#. Reload the Docker daemon:
.. code-block:: bash
sudo systemctl daemon-reload
Configure the Docker container proxies, in order to pass proxy
settings to containers:
#. Create a directory for your container config:
.. code-block:: bash
mkdir ~/.docker
#. Create the config file :file:`~/.docker/config.json` and add the following
entries, using your own proxy values:
.. code-block:: console
{
"proxies":
{
"default":
  {
   "httpProxy": "<proxy-url>:<port>",
    "httpsProxy": "<proxy-url>:<port>"
  }
}
}
#. Set ownership and permission on the docker config directory:
.. code-block:: bash
sudo chown "$USER":"$USER" /home/"$USER"/.docker -R
sudo chmod g+rwx "$HOME/.docker" -R
Lastly, configure proxies to allow mixer to access upstream content from behind
a firewall. For example:
#. Open your :file:`$HOME/.bashrc` file and add proxy and port values for the
following:
.. code-block:: console
export http_proxy="<proxy-url>:<port>"
export https_proxy="<proxy-url>:<port>"
export HTTP_PROXY="<proxy-url>:<port>"
export HTTPS_PROXY="<proxy-url>:<port>"
export no_proxy="<...>"
#. Log out and log back in for the proxies to take effect.
.. rst-class:: content-collapse
Set up a nginx web server for mixer
===================================
A web server is needed to host your update content. In this example, we use the
nginx web server, which comes with |CL|.
Set up a nginx web server for mixer with the following steps:
#. Install the :command:`nginx` bundle:
.. code-block:: bash
sudo swupd bundle-add nginx
#. Make the directory where mixer updates will reside:
.. code-block:: bash
sudo mkdir -p /var/www
#. Create a symbolic link between your workspace updates and the updates on the
local nginx web server. In this example, `$HOME/mixer` is the workspace for
the mix.
.. code-block:: bash
sudo ln -sf $HOME/mixer/update/www /var/www/mixer
#. Set up ``nginx`` configuration:
.. code-block:: bash
sudo mkdir -p /etc/nginx/conf.d
#. Copy the default example configuration file:
.. code-block:: bash
sudo cp -f /usr/share/nginx/conf/nginx.conf.example /etc/nginx/nginx.conf
#. Configure the mixer update server. Create and add the following server
configuration content to :file:`/etc/nginx/conf.d/mixer.conf` (sudo required):
.. code-block:: console
server {
server_name localhost;
location / {
root /var/www/mixer;
autoindex on;
}
}
#. Restart the daemon, enable nginx on boot, and start the service.
.. code-block:: bash
sudo systemctl daemon-reload
sudo systemctl enable nginx
sudo systemctl start nginx
#. Verify the web server is running at \http://localhost. At this point
you should no longer see a "404 Not Found" message.
Related topics
**************
* :ref:`About mixer <mixer-about>`
* :ref:`mixin`
* :ref:`autospec-about`
* :ref:`bundles-about`
* :ref:`swupd-about`
.. _Docker Hub: https://hub.docker.com/r/clearlinux/mixer/tags/
.. _mixer man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.1.rst
.. _mixer.init man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.init.1.rst
.. _mixer.bundle man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.bundle.1.rst
.. _mixer.build man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.build.1.rst
.. _suitable versions: https://github.com/clearlinux/clr-bundles/releases
.. _Install a bundle: https://clearlinux.org/documentation/clear-linux/guides/maintenance/swupd-guide#adding-a-bundle
source/clear-linux/guides/maintenance/mixin.rst → source/clear-linux/guides/clearlinux/mixin.rst
View File
source/clear-linux/guides/clearlinux/swupd-guide.rst
+355
View File
@@ -0,0 +1,355 @@
.. _swupd-guide:
swupd
#####
:command:`swupd` links a |CL-ATTR| installation with upstream updates and
software.
.. contents::
:local:
:depth: 2
Description
***********
:command:`swupd` has two main functions:
#. It manages software replacing APT or YUM, installing bundles
rather than packages.
#. It checks for system updates and installs them.
:ref:`Bundles <bundles>` are the smallest granularity component that is
managed by |CL| and contain everything needed to deliver a software
capability. Rather than downloading a cascade of package dependencies when
installing a piece of software, a bundle comes with all of its dependencies.
:command:`swupd` manages overlapping dependencies behind the scenes ensuring
that all software is compatible across the system.
Versioning
==========
In a traditional distribution, the process of describing current software
versioning usually involves:
- Listing and keeping track of the current OS release (generally
uninformative about any singular packages or functionality).
- Keeping track of packages and repositories being used, and updating them
individually.
- Listing and tracking every package available and installed on the
system, none of which are directly tied to the current OS release.
This can be done effectively, but given the nearly endless combinations of
packages and versions of packages a server may have, it quickly becomes
non-trivial to define what "version" the system is and what software it
is running without explicitly going through each system and inspecting
every package.
With |CL|, we need track:
- One number
A number representing the **current** release of the OS is sufficient to
describe the versions of all the software on the OS. Each build is
composed of a specific set of bundles made from a particular version of
packages. This matters on a daily basis to system administrators, who
need to determine which of their systems do not have the latest security
fixes, or which combinations of software have been tested. Every release
of the same number is guaranteed to contain the same versions of software,
so there's no ambiguity between two systems running the same version of |CL|.
Updating
========
|CL| enforces regular updating of the OS by default and will automatically
check for updates against a version server. The content server provides the
file and metadata content for all versions and can be the same as the
version server. The content url server provides metadata in the form of
manifests. These Manifest files list and describe file contents, symlinks,
and directories. Additionally, the actual content is
provided to clients in the form of archive files.
Software updates with |CL| are also efficient. Unlike package-based
distributions, :command:`swupd` only updates files that have changed rather
than entire packages. For example, it is quite common for an OS security
patch to be as small as 15 KB. Using binary deltas, the |CL| is able to
apply only what is needed.
To get a more detailed understanding of how to generate update content for
|CL| see the :ref:`mixer <mixer>` tool.
How it works
************
Prerequisites
=============
* The device is on a well-connected network.
* The device is able to connect to an update server. The default server is:
http://update.clearlinux.org
Updates
=======
|CL| updates are automatic by default but can be set to occur only on
demand. :command:`swupd` makes sure that regular updates are simple and
secure. It can also check the validity of currently installed files and
software and correct any problems.
Manifests
---------
The Clear Linux OS software update content consists of data and
metadata. The data is the files that end up in the OS. The metadata
contains relevant information to properly provision the data to the OS
file system, as well as update the system and add or remove additional
content to the OS.
The Manifests are mostly long lists of hashes that describe content.
Each bundle gets its own manifest file. There is a master manifest
file that describes all manifests to tie it all together.
Fullfiles, packs, and delta packs
---------------------------------
The data that an update provisions to a system can be obtained in
three different ways. There are three different methods, and they
exist to optimize the delivery of content and speed up updates.
Fullfiles are always generated for every file in every release. This
allows any Clear Linux OS to obtain the exact copy of the content
for each version directly. This would be used if the OS verification
needed to replace a single file, for instance.
Packs are available for some releases and combine many files to speed
up the creation of installation media and large updates. Delta packs
are an optimized version of packs that only contain updates (binary
diffs) and cannot be used without having the original file content.
Bundle Search
=============
:command:`swupd` search downloads manifest data and searches for
bundles that match the term. Enter only one term, or hyphenated term, per
search. Use the command :command:`man swupd` to learn more.
Only the base bundle is returned. Bundles can contain other bundles via
includes. For more details, see `Bundle Definition Files`_ and its
subdirectory bundles.
Bundles that are already installed, will be marked [installed] in search
results.
Optionally, you can review our `bundles`_ on GitHub.
Examples
********
Example 1: Disable and Enable automatic updates
===============================================
|CL| updates are automatic by default but can be set to occur only
on demand.
#. First verify your current auto-update setting.
.. code-block:: bash
sudo swupd autoupdate
.. code-block:: console
Enabled
#. Disable automatic updates.
.. code-block:: bash
sudo swupd autoupdate --disable
.. code-block:: console
Warning: disabling automatic updates may take you out of compliance with your IT policy
Running systemctl to disable updates
Created symlink /etc/systemd/system/swupd-update.service → /dev/null.
Created symlink /etc/systemd/system/swupd-update.timer → /dev/null.
#. Check manually for updates.
.. code-block:: bash
sudo swupd check-update
#. Install an update after identifying one that you need.
.. code-block:: bash
sudo swupd update -m <version number>
#. Re-enable automatic installs.
.. code-block:: bash
sudo swupd autoupdate --enable
.. _swupd-guide-example-install-bundle:
Example 2: Find and install Kata\* Containers
=============================================
Kata Containers is a popular container implementation. Unlike other
container implementations, each Kata Container has its own
kernel instance and runs on its own :abbr:`Virtual Machine (VM)` for
improved security.
|CL| makes it very easy to install, since you only need to add
`one bundle`_ to use `Kata Containers`_: `containers-virt`, despite a
number of dependencies. Also, check out our tutorial: :ref:`kata`.
#. Find the right bundle.
* To return all possible matches for the search string enter
:command:`swupd search`, followed by 'kata':
.. code-block:: bash
sudo swupd search kata
The output should be similar to:
.. code-block:: console
Bundle with the best search result:
containers-virt - Run container applications from Dockerhub in lightweight virtual machines
This bundle can be installed with:
swupd bundle-add containers-virt
Alternative bundle options are
cloud-native-basic - Contains ClearLinux native software for Cloud
.. note::
If your search does not produce results with a specific
term, shorten the search term. For example, use *kube* instead of
*kubernetes*.
#. Add the bundle.
.. code-block:: bash
sudo swupd bundle-add containers-virt
.. note::
To add multiple bundles simply add a space followed by the bundle name.
The output of a successful installation should be similar to:
.. code-block:: console
Downloading packs...
Extracting containers-virt pack for version 24430
...50%
Extracting kernel-container pack for version 24430
...100%
Starting download of remaining update content. This may take a while...
...100%
Finishing download of update content...
Installing bundle(s) files...
...100%
Calling post-update helper scripts.
Successfully installed 1 bundle
Example 3: Verify and correct sytem file mismatch
=================================================
:command:`swupd` can determine whether system directories and files have
been added to, overwritten, removed, or modified (e.g., permissions).
.. code-block:: bash
sudo swupd verify
All directories that are watched by :command:`swupd` are verified according
to the manifest data and hash mismatches are flagged as follows:
.. code-block:: console
Verifying version 23300
Verifying files
...0%
Hash mismatch for file: /usr/bin/chardetect
...
...
Hash mismatch for file: /usr/lib/python3.6/site-packages/urllib3/util/wait.py
...100%
Inspected 237180 files
423 files did not match
Verify successful
In this case, python packages that were installed on top of the default
install were flagged as mismatched. :command:`swupd` can be directed to
ignore or fix issues based on command line options.
:command:`swupd` can correct any issues it detects. Additional directives
can be added including a white list of directories that will be ignored.
The following command will repair issues, remove unknown items, and
ignore files or directories matching `/usr/lib/python`:
.. code-block:: bash
sudo swupd verify --fix --picky --picky-whitelist=/usr/lib/python
Quick Reference
***************
swupd info
To see the currently installed version and update servers.
swupd update <version number>
To update to a specific version or with no arguments to update to latest.
swupd bundle-list [--all]
To list installed bundles.
swupd bundle-add [-b] <search term>
To find a bundle that contains your search term.
swupd bundle-add <bundle name>
To add a bundle.
swupd bundle-remove <bundle name>
To remove a bundle.
swupd --help
For additional :command:`swupd` commands.
man swupd
To reference the :command:`swupd` man page, or see the
`source documentation`_ available on github.
Related topics
**************
* :ref:`autospec`
* :ref:`mixer`
* :ref:`bundles`
.. _source documentation: https://github.com/clearlinux/swupd-client/blob/master/docs/swupd.1.rst
.. _Kata Containers: https://clearlinux.org/containers
.. _one bundle: https://github.com/clearlinux/clr-bundles/blob/master/bundles/containers-virt
.. _Bundle Definition Files: https://github.com/clearlinux/clr-bundles
.. _bundles: https://github.com/clearlinux/clr-bundles/tree/master/bundles
source/clear-linux/guides/maintenance/developer-workstation.rst
+3 -2
View File
@@ -78,10 +78,11 @@ While your role may not neatly fit in one of these categories, consider using Ta
`swupd` search
**************
We recommend trying out :ref:`swupd search <swupd-search>`, to learn the
We recommend learning about :ref:`swupd <swupd-guide>`, to learn the
commands to search for and add bundles relevant to your project.
:ref:`swupd-search` shows you how to:
The guide provides an :ref:`example <swupd-guide-example-install-bundle>`
that shows you how to:
* Use `swupd` to search for bundles
* Use `swupd` to add bundles
source/clear-linux/guides/maintenance/kernel-modules.rst
+1 -1
View File
@@ -45,7 +45,7 @@ security practices, see the :ref:`security` page.
You can search for kernel module file names, which end with the :file:`.ko`
file extension, using the :command:`swupd search` command. For example:
:command:`sudo swupd search ${module_name}.ko`. See :ref:`swupd-search` for
:command:`sudo swupd search ${module_name}.ko`. See :ref:`swupd-guide` for
more information.
source/clear-linux/guides/maintenance/mixer.rst
-871
View File
@@ -1,871 +0,0 @@
.. _mixer:
Use the mixer tool
##################
*Mixing* refers to composing an operating system for specific use cases.
While the default |CL-ATTR| provides options to install bundles for various
server capabilities, some developers may wish to either augment the
operating system itself with functionality from their own packages or modify
the structure of current bundles to cater to their particular needs.
Prerequisites
*************
To start working with the mixer tools, you need a recent image of |CL| with
the `mixer` bundle installed. If the bundle is not yet installed, you can
add it with the :command:`swupd bundle-add` command as follows:
.. code-block:: bash
sudo swupd bundle-add mixer
Current mixing workflow
***********************
Mixer by default runs *all* build commands in a container to ensure the
correct version of the tooling is being used. This also allows custom mixes
to automatically perform downstream format bumps when upstream releases
a format bump. You can still run mixer natively by appending the *--native* flag to
any of the commands.
.. note::
You cannot run mixer if you are already in a container, unless you pass
*--native* to the command. Nested containerization is not supported, nor
is building images using the container mode.
There are two different workflows to create your own mix.
First, if your mix only uses |CL| content, *skip Create custom RPMs* below.
Second, if your mix includes your own
:abbr:`RPMs (RPM Package Manager files)`, follow all steps below.
.. contents::
:local:
:depth: 1
:backlinks: top
The following sections contain detailed information on every step of
these workflows.
.. _create-nginx-web-server:
Create nginx web server to host mixer updates
=============================================
Follow these steps to set up a HTTP service with ``nginx`` web
server, where you can host custom |CL| mixes:
#. Install ``web-server-basic``.
.. code-block:: bash
sudo swupd bundle-add web-server-basic
#. Make the directory where mixer updates will reside.
.. code-block:: bash
sudo mkdir -p /var/www
#. Create a symbolic link.
.. code-block:: bash
sudo ln -sf /home/clr/mix/update/www /var/www/mixer
.. note::
`/home/clr/mix` is the workspace for the mix (read `Create workspace`)
#. Set up ``nginx`` configuration.
.. code-block:: bash
sudo mkdir -p /etc/nginx/conf.d
#. Copy the default example configuration file.
.. code-block:: bash
sudo cp -f /usr/share/nginx/conf/nginx.conf.example /etc/nginx/nginx.conf
#. Open ``mixer.conf`` with an editor.
.. code-block:: bash
sudo nano /etc/nginx/conf.d/mixer.conf
#. Add the server configuration content below to ``mixer.conf``.
.. code-block:: console
server {
server_name localhost;
location / {
root /var/www/mixer;
autoindex on;
}
}
#. Restart the daemon, enable nginx on boot, and start the service.
.. code-block:: bash
sudo systemctl daemon-reload
sudo systemctl enable nginx
sudo systemctl start nginx
#. To verify the web server is running, check it in an Internet browser:
.. code-block:: bash
http://localhost
#. Alternatively, verify the server is running by entering:
.. code-block:: bash
ps aux | grep nginx
.. note::
If `nginx` is active, a list should appear showing one master process
and a few worker processes.
Connect the URL to mixer
------------------------
Add the URL of the `nginx` server to builder.conf. Your |CL| clients connect
to this URL to find the update content.
#. Get the IP address of your nginx server:
.. code-block:: bash
networkctl status
#. In the mixer workspace, edit `builder.conf` to set the value for
CONTENTURL and VERSIONURL to the IP `Address` of your `nginx` server.
.. code-block:: bash
nano builder.conf
.. note::
For example:
CONTNENTURL=http://192.168.25.52
VERSIONURL=http://192.168.25.52
Create a workspace
==================
Use the following command to create an empty directory in your |CL| image to
use as a **workspace** for mixing:
.. code-block:: bash
mkdir /home/clr/mix
This guide assumes your workspace location is :file:`/home/clr/mix`.
Generate the starting point for your mix
========================================
In your workspace, initialize mixer with the following command:
.. code-block:: bash
mixer init
This command initializes your workspace so you can make a mix at version 10
based on the latest released upstream |CL| version. If a :file:`builder.conf`
file is not already present in your workspace, mixer creates a default
configuration file. Additionally, mixer creates several version and tracking
files, and two bundle directories: :file:`local-bundles` and
:file:`upstream-bundles`.
If you wish to start with a different version of upstream |CL| or a
different initial mix version, you can specify these options as flags.
For example:
.. code-block:: bash
mixer init --upstream-version 21060 --mix-version 10
Additionally, to build a mix with your own custom RPMs, use the optional
*--local-rpms* flag, for example:
.. code-block:: bash
mixer init --local-rpms
This command creates the :file:`local-yum` and :file:`local-rpms`
directories in your mix workspace and adds their paths to the generated
:file:`builder.conf`. If the configuration file already exists, you must add
the paths manually. For more information on using these directories or
setting them up manually, see `Create or locate RPMs for the mix`_.
If all upstream |CL| bundles will be part of the mix, you can easily add
them all during initialization with the optional *--all-upstream* flag. For example:
.. code-block:: bash
mixer init --all-upstream
Finally, you may want to track the contents of your mixer workspace with a
git repository. This is a great way to track changes to your mix's content
or to revert to earlier versions if something goes wrong. Mixer can set this
up automatically with the optional *--git* flag, for example:
.. code-block:: bash
mixer init --git
.. note::
You can use any or all of the above optional flags at the same time, for example:
.. code-block:: bash
mixer init --upstream-version 21060 --mix-version 10 --local-rpms --all-upstream --git
Edit builder.conf
=================
To configure the mixer tool, edit the :file:`builder.conf` as needed.
The file :file:`builder.conf` is read automatically from the current
workspace directory. Use the *--config* flag during initialization
to specify an alternate path to the file as needed.
The :file:`builder.conf` file has different sections, for example:
* The `[Builder]` section provides the mixer tools with the required
configuration options. This section defines the path where the generated
bundles and update metadata are published.
* The `[swupd]` section contains specific update parameters. The
:abbr:`swupd-server (software update server)` creates an update using
said specific update parameters.
Edit the configuration file according to your needs with the command:
.. code-block:: bash
vim /home/clr/mix/builder.conf
Your version of the :file:`builder.conf` file should resemble the
following example:
.. code-block:: console
#VERSION 1.0
[Builder]
CERT = "/home/clr/mix/Swupd_Root.pem"
SERVER_STATE_DIR = "/home/clr/mix/update"
VERSIONS_PATH = "/home/clr/mix"
YUM_CONF = "/home/clr/mix/.yum-mix.conf"
[Swupd]
BUNDLE = "os-core-update"
CONTENTURL = "<URL where the content will be hosted>"
VERSIONURL = "<URL where the version of the mix will be hosted>"
[Server]
DEBUG_INFO_BANNED = "true"
DEBUG_INFO_LIB = "/usr/lib/debug"
DEBUG_INFO_SRC = "/usr/src/debug"
[Mixer]
LOCAL_BUNDLE_DIR = "/home/clr/mix/local-bundles"
LOCAL_REPO_DIR = ""
LOCAL_RPM_DIR = ""
DOCKER_IMAGE_PATH = "clearlinux/mixer"
The following variables require further explanation:
* The `LOCAL_BUNDLE_DIR` variable sets the path where mixer stores the local
bundle definition files. These bundle definition files include any new,
original bundles you create, along with any edited versions of upstream
|CL| bundles.
* The `SERVER_STATE_DIR` variable sets the path for the output of the mix
content. Mixer automatically creates the path for you, but the path can be
set to any location. In this example, we use the workspace directory.
* The `YUM_CONF` variable sets the path where mixer automatically generates
the :file:`.yum-mix.conf` yum configuration file. The yum configuration
file points the chroot-builder to the path where the RPMs are stored.
* The `CERT` variable sets the path where mixer stores the
:file:`Swupd_Root.pem` certificate file. The chroot-builder needs the
certificate file to sign the root :file:`Manifest.MoM` file to provide
security for content verification. The value of the `CERT` variable can
point to a different certificate. The chroot-builder inserts the
certificate specified in this value into the
:file:`/os-core-update/usr/share/clear/update-ca/` path. The software
update client uses this certificate to verify the :file:`Manifest.MoM`
file's signature. For now, we **strongly** recommend that you do not modify
this line, as the certificate that :abbr:`swupd (Software Updater)`
expects needs to have a very specific configuration to sign and verify
properly. Mixer automatically generates the certificate, if you do not
provide the path to an existing one, and signs the :file:`Manifest.MoM`
file to provide security for the updated content you create.
* The `CONTENTURL` and `VERSIONURL` variables set the domain or IP address
where swupd looks for your update content and the corresponding version.
You must set these variables to the domain or IP-address of the server
hosting the update content. You can use any web server to host your update
content. To learn how to install and configure web server using |CL|, see
:ref:`create-nginx-web-server`. For our example, the web update content
within the `SERVER_STATE_DIR` directory is located here:
:file:`/home/clr/mix/update/www`. If the web server is on the same machine
as this directory, you can create a symlink to the directory in your web
server's document root to easily host the content. These URLs are
embedded in images created for your mix. The `swupd-client` looks at
these URLs to determine if a new version is available and the location
from where to download the updated content. These links are equivalent
to the |CL| `update page`_ but for the mix.
* The `VERSIONS_PATH` variable sets the path for the mix version and upstream
|CL| version's two state files: :file:`mixversion` and
:file:`upstreamversion`. Mixer creates both files for you when you set up
the workspace.
* The `DOCKER_IMAGE_PATH` variable sets the base name of the docker image
mixer will pull down in order to run builds in the proper container.
.. note:: If you are working only with |CL| bundles, then
skip to `List, edit, create, add, remove, or validate bundles`_.
Create custom RPMs
==================
Create or locate RPMs for the mix
---------------------------------
If you create RPMs from scratch, you can use `autospec`, `mock`, `rpmbuild`,
or similar tools to build them. If the RPMs are not built on |CL|, ensure
your configuration and toolchain builds them correctly for |CL|, or else
there is no guarantee they will be compatible. For more information on
building the RPMs properly, refer to our `build RPMs instructions`_.
Import RPMs into workspace
--------------------------
#. Create a :file:`local-rpms` directory in your workspace, for example,
:file:`/home/clr/mix/local-rpms`.
#. Copy the RPMs into the directory you created.
#. Add the following line to your :file:`builder.conf` file:
.. code-block:: console
LOCAL_RPM_DIR=/home/clr/mix/local-rpms
Mixer uses this directory to find the RPMs to build a local RPM repo for
yum to use.
Create a local RPM repo
-----------------------
#. Create an empty directory in your workspace named :file:`local-yum`.
#. Add the path to your :file:`builder.conf` file:
.. code-block:: console
LOCAL_REPO_DIR=/home/clr/mix/local-yum
#. With these values configured, generate the yum repo with the following
command:
.. code-block:: bash
mixer add-rpms
After the tool exits, you should see the RPMs and a repository data
directory in :file:`/home/clr/mix/local-yum`. If the RPMs are not all in this
:file:`local-yum` directory, check to ensure that the RPM files are valid
and not corrupt.
List, edit, create, add, remove, or validate bundles
====================================================
The bundles in the mix are specified in the mix bundle list. Mixer stores
this list as a flat file called :file:`mixbundles` in the path set by the
`VERSIONS_PATH` variable of the :file:`builder.conf` file. Mixer
automatically generates the :file:`mixbundles` list file during
initialization. Mixer reads and writes the bundle list file when you change
the bundles of the mix.
List the bundles in the mix
---------------------------
To view the bundles already in the mix, enter the following command:
.. code-block:: bash
mixer bundle list
This command shows a list of every bundle in the mix. Bundles can include
other bundles. Those nested bundles can themselves include other
bundles. When listing bundles with this command, mixer automatically
recurses through the includes to show every single bundle in the mix.
If you see an unexpected bundle in the list, that bundle is probably included
in another bundle. Use the *--tree* flag to get a better view of how
a bundle ended up in the mix, for example:
.. code-block:: bash
mixer bundle list --tree
This command shows a visual representation of the inclusion relationships
between the bundles in the mix.
Bundles fall into two categories: **upstream** and **local**.
Upstream bundles are those provided by |CL|.
Mixer automatically downloads and caches upstream bundle definition files.
These definition files are stored in the :file:`upstream-bundles` directory
in the workspace. Do **not** modify the files in this directory. This
directory is simply a mirror for mixer to use.
The mixer tool automatically caches the bundles for the |CL| version
configured in the :file:`upstreamversion` file. Mixer also cleans up old
versions once they are no longer needed. You can see the available upstream
bundles with the following command:
.. code-block:: bash
mixer bundle list upstream
Local bundles are bundles that you create, or are edited versions of upstream
bundles.
Local bundle definition files live in the :file:`local-bundles` directory.
The `LOCAL_BUNDLE_DIR` variable sets the path of this directory in your
:file:`builder.conf` configuration file. For this example, the path is
:file:`/home/clr/mix/local-bundles`. You can see the available local bundles
with the following command:
.. code-block:: bash
mixer bundle list local
Both the local and upstream :command:`bundle list` commands accept the
*--tree* flag to show a visual representation of the inclusion relationships
between the bundles in the mix.
Edit the bundles in the mix
---------------------------
**Mixer always checks local bundles first and the upstream bundles second.**
Therefore, bundles in the :file:`local-bundles` directory always take
precedence over any upstream bundles that have the same name.
This precedence enables you to edit upstream bundles. The local, edited
version of the bundle overrides the bundle version found upstream.
For example, to edit the `bundle1` definition file, we use the following
command:
.. code-block:: bash
mixer bundle edit bundle1
If `bundle1` is found in your local bundles, mixer edits this bundle
definition file. If instead `bundle1` is only found upstream, mixer copies
the bundle definition file from upstream into your :file:`local-bundles`
directory first.
In both cases, mixer launches your default editor to edit the file. When the
editor closes, mixer automatically validates the edited bundle file and
reports any errors found. If mixer finds an error, you can edit the file
as-is, revert and edit, or skip and move on to the next bundle. If you skip a
file, mixer saves a backup of the original file with the ``.orig`` suffix.
Because mixer always checks your local bundles first, edited copies of an
upstream bundle always take precedence over their upstream counterpart. You
can edit multiple bundles with the following command:
.. code-block:: bash
mixer bundle edit bundle1 bundle2 [bundle3 ...]
Create bundles for the mix
--------------------------
To create a totally **new bundle**, the bundle name you specify cannot exist
upstream. If that is the case, create a :file:`new-bundle` with the following
command:
.. code-block:: bash
mixer bundle edit new-bundle
This command generates a blank template in :file:`local-bundles` with the
:file:`new-bundle` filename. Mixer launches the editor for you to fill out
the bundle and performs validation when you exit the editor. Add your package
or packages to the bundle definition file to define the packages to install
as part of the bundle.
.. note::
The :command:`mixer bundle edit` command accepts multiple bundles at once.
Thus, you can create multiple new bundles in a single command, for
example:
.. code-block:: bash
mixer bundle edit new-bundle1 new-bundle2 [new-bundle3 ...]
Add bundles to the mix
----------------------
Add `bundle1` to your mix with the following command:
.. code-block:: bash
mixer bundle add bundle1
This command adds the specified bundles to the mix bundles list stored in
your :file:`mixbundles` file. For each bundle you add, mixer checks your
local and upstream bundles to ensure the added bundles actually exist. If
mixer cannot find the bundle, it reports back an error. Additionally, when
mixer adds a bundle, it tells you whether the bundle is local or upstream.
Alternatively, you can learn this information with the
:command:`mixer bundle list` command. Refer to `List the bundles in the mix`_.
To add multiple bundles at once, use the following command:
.. code-block:: bash
mixer bundle add bundle1 bundle2 [bundle3 ...]
Remove bundles from the mix
---------------------------
Remove `bundle1` from your mix with the following command:
.. code-block:: bash
mixer bundle remove bundle1
This command removes `bundle1` from the mix bundle list stored in your
:file:`mixbundles` file. By default, this command does not remove the bundle
definition file from your local bundles. To completely remove a bundle,
including its local bundle definition file, use the following command with
the *--local* flag:
.. code-block:: bash
mixer bundle remove --local bundle1
By default, removing a local bundle file with this command removes the bundle
from the mix as well. To only remove the local bundle definition file, use
the following command with the *--mix=false* flag:
.. code-block:: bash
mixer bundle remove --local --mix=false bundle1
If you remove a local, edited version of an upstream bundle and keep the
bundle in the mix, the mix then references the original upstream version of
the bundle.
On the other hand, if you remove a bundle that is only found locally but
keep the bundle in the mix bundles list, mixer will not find a valid
bundle definition file and will produce an error.
Validate the bundles in the mix
-------------------------------
Mixer performs basic validation on all bundles when used throughout the
system.
Mixer checks the validity of the bundle's syntax and name. Optionally, you
can run this validation manually on `bundle1` with the following command:
.. code-block:: bash
mixer bundle validate bundle1
.. note:: This command can be useful in many circumstances. One example is
when importing already-existing local bundles from other projects.
If you use the optional *--strict* flag, the command additionally
checks if the rest of the bundle header fields can be parsed, if the bundle
header fields are non-empty, and if the bundle header ``Title`` field and
the bundle filename match. Perform a strict validation of `bundle1` with the
following command:
.. code-block:: bash
mixer bundle validate --strict bundle1
Validate multiple bundles with the following command:
.. code-block:: bash
mixer bundle validate bundle1 bundle2 [bundle3 ...]
Managing bundles with Git
-------------------------
If you initialized your workspace to be tracked as a git repository
with the :command:`mixer init --git` command, it might be useful to apply a
git commit after you modify the mix bundle list or edit a bundle definition
file.
All the :command:`mixer bundle` commands in the previous sections support an
optional *--git* flag. This flag automatically applies a git commit
when the command completes, for example:
.. code-block:: bash
mixer bundle remove --git bundle1
Build the bundle chroots
========================
To build all the ``chroots`` based on the defined bundles, use the following
command in your workspace:
.. code-block:: bash
mixer build bundles
If the mix has many bundles, this step might take some time.
Mixer automatically gathers the bundle definition files for the upstream
bundles into a :file:`upstream-bundles` directory, and user bundles should
be placed directly into :file:`local-bundles`. The local path is set in
the `LOCAL_BUNDLE_DIR` variable in the :file:`builder.conf`. **Do not edit
files in upstream-bundles.** Mixer automatically deletes the contents of
the :file:`upstream-bundles` directory before repopulating the directory
on-the-fly if a new version must be downloaded.
Create an update
================
Create an update with the following command:
.. code-block:: bash
mixer build update
When the build completes, you can find the mix update content under
:file:`/home/clr/mix/update/www/VER`. In our example, the update content is
found in :file:`/home/clr/mix/update/www/{<MIXVERSION>}`. `<MIXVERSION>`
is the defined mix version, which is 10 by default.
.. code-block:: bash
mixer build update
Mixer creates all the content needed to make a fully usable mix with this
step. However, only *zero packs* are automatically generated. Zero packs are
the content needed to go from nothing to the mix version for which you just
built the content.
You can create optional *delta packs*, which allow the transition from one
mix version to another, with the following command:
.. code-block:: bash
mixer build delta-packs --to <MIX_VERSION> --from <PAST_VERSION>
This command generates all delta packs for the bundles changed from
`PAST_VERSION` to `MIX_VERSION`. Mixer cannot create delta packs for the
first build because the update is from version 0. Version 0 implicitly has
no content, thus mixer can generate no deltas.
For subsequent builds, you can run :file:`mixer-pack-maker.sh` to generate
delta content between them, for example: 10 to 20.
Create an image
===============
Since mixer uses the `ister` tool to create a bootable image from your
updated content, we must first configure the `ister` tool. To configure the
image `ister` creates, we need the `ister` configuration file. Obtain a copy
with the default values from the `ister` package with the following command:
.. code-block:: bash
sudo cp /usr/share/defaults/ister/ister.json release-image-config.json
For reference, you can inspect the `Clear Linux ister configuration file`_
used for releases.
Edit the configuration file to include all bundles you want *preinstalled* in
the image. Users can install the bundles in the mix that are not included in
the configuration file with the following command:
.. code-block:: bash
sudo swupd bundle-add bundle1
Keeping the list of bundles in the configuration file small allows for a
smaller image size. For the minimal base image, the list is:
.. code-block:: console
"Bundles": ["os-core", "os-core-update", "kernel-native"]
Next, set the `Version` field to the mix version that you want the content
mixer to use to build the image. `ister` allows you to build an image from
any mix version that you have built, not just from the current version. In
our example so far, `Version` is set to 10.
With the `ister` tool configured, build the image with the following command:
.. code-block:: bash
sudo mixer build image
Mixer automatically looks for the :file:`release-image-config.json` file, but
you can freely choose the filename. To use a different name, simply pass the
*--template* flag when creating your image, for example:
.. code-block:: bash
sudo mixer build image --template path/to/file.config
By default, `ister` uses the format version of the build machine it runs on.
Therefore, if the format you are building differs from the format of the |CL|
OS you are building on, you must use the *--format <FORMAT_NUMBER>*
flag. Find the current format version of your OS with the following command:
.. code-block:: bash
sudo cat /usr/share/defaults/swupd/format
Update the next mix version information
=======================================
Increment the mix version number for the next mix with the following command:
.. code-block:: bash
mixer versions update
This command automatically updates the mix version stored in the
:file:`mixversion` file, incrementing it by 10. To increment by a different
amount, use the *--increment* flag, for example:
.. code-block:: bash
mixer versions update --increment 10
Alternatively, to set the mix version to a specific value, use the
*--mix-version* flag, for example:
.. code-block:: bash
mixer versions update --mix-version 20
The :command:`mixer versions update` command does not allow you to set the
mix version to a value less than its current value. The mix version is
expected to always increase, even if the new mix is undoing an earlier
change.
If you have been tracking your workspace with git, you can restore the mix to
an earlier state. However, be careful when "rewriting history" if you have
published the mix content to users already.
Use the following command with the the *--upstream-version* flag to
update the upstream version of |CL| used as a base for the mix:
.. code-block:: bash
mixer versions update --upstream-version 21070
This command also accepts the keyword "latest":
.. code-block:: bash
mixer versions update --upstream-version latest
This command sets the upstream version to the latest released version of
upstream |CL| within the same format version. The
:command:`mixer versions update` command does not allow you to set an
upstream version to a value that crosses an upstream format boundary.
Such values require a "format bump" build, which is currently a
manual process. Refer to :ref:`mixer-format` for more information.
Optionally, you can learn which mix version or upstream version you are
currently using with the following command:
.. code-block:: bash
mixer versions
At this point, you can continue to iterate through the workflows and make
modifications as needed, for example:
#. Add, remove, or modify bundles.
#. Build the chroots with:
.. code-block:: bash
mixer build chroots
#. Build and update with:
.. code-block:: bash
mixer build update
#. Optionally, you can create delta packs with:
.. code-block:: bash
mixer build delta-packs --to <NEWVERSION> --from <PREV_VERSION>
.. _mixer-format:
Format version
==============
The `Format` variable set in the :file:`builder.conf` file can be more
precisely referred to as an OS *compatibility epoch*. Versions of the OS
within a given epoch are fully compatible and can update to any other
version within that epoch. Across the `Format` boundary, the OS has changed
in such a way that updating from build M in format X, to build N in format Y
will not work. Generally, this scenario occurs when the software updater or
software manifests change in a way that they are no longer compatible with
the previous update scheme.
Using a format increment, we insure pre- and co-requisite changes flow out
with proper ordering. The updated client only ever updates to the latest
release in its respective format version, unless overridden by command line
flags. Thus, we can guarantee that all clients update to the final version
in their given format. The given format *must* contain all the changes
needed to understand the content built in the subsequent format. Only after
reaching the final release in the old format can a client continue to update
to releases in the new format.
When creating a custom mix, the format version should start at "1" or some
known number such as the host system format. The format version should
increment only when a compatibility breakage is introduced. Normal updates,
like updating a software package for example, do not require a format
increment.
.. _update page: https://cdn.download.clearlinux.org/update/
.. _format bumps wiki: https://github.com/clearlinux/swupd-server/wiki/Format-Bumps
.. _build RPMs instructions: https://github.com/clearlinux/common#build-rpms-for-a-package
.. _Clear Linux ister configuration file:
https://raw.githubusercontent.com/bryteise/ister/master/release-image-config.json
source/clear-linux/guides/maintenance/swupd-guide.rst
-185
View File
@@ -1,185 +0,0 @@
.. _swupd-guide:
Use swupd
#########
*swupd* manages the software update capability of |CL-ATTR|. It can check for
valid system updates and, if found, download and install them. It can also
perform verification of the system software.
|CL| uses :ref:`bundles <bundles-about>` as the base abstraction for
installing functionality on top of the core operating system. Use the `swupd`
tool to install and remove bundles.
This guide assumes:
* The device is on a well-connected network.
* The device is able to connect to the release infrastructure
http://update.clearlinux.org
.. contents::
:local:
:depth: 2
.. _man_swupd:
For a full listing of commands and options please see the man page found in
the `swupd` `source code`_ or on |CL| itself.
.. code-block:: bash
man swupd
OS update and verification
**************************
|CL| is designed to promote a regular update cadence. `swupd` helps to
make sure that process is simple and secure.
View OS info
============
Current OS version and update server info:
.. code-block:: bash
swupd info
.. code-block:: console
Installed version: 23330
Version URL: https://cdn.download.clearlinux.org/update/
Content URL: https://cdn.download.clearlinux.org/update/
Enable or disable automatic updates
===================================
|CL| updates are automatic by default but can be set to occur only
on demand. To verify your current auto-update setting:
.. code-block:: bash
sudo swupd autoupdate
Disable automatic updates:
.. code-block:: bash
sudo swupd autoupdate --disable
To re-enable automatic:
.. code-block:: bash
sudo swupd autoupdate --enable
Check for updates
=================
.. code-block:: bash
sudo swupd check-update
Perform a manual update
=======================
You can update to a specific version or accept the latest as the default with
no arguments. Initiate a manual update:
.. code-block:: bash
sudo swupd update -m 23330
.. note::
After a kernel update, reboot immediately for the enhancements to
take effect.
System Software Verification
============================
`swupd` can determine whether system directories and files have been added
to, overwritten, removed, or modified (e.g., permissions).
.. code-block:: bash
sudo swupd verify
All directories that are watched by `swupd` are verified according to
the manifest data and hash mismatches are flagged as follows:
.. code-block:: console
Verifying version 23300
Verifying files
...0%
Hash mismatch for file: /usr/bin/chardetect
...
...
Hash mismatch for file: /usr/lib/python3.6/site-packages/urllib3/util/wait.py
...100%
Inspected 237180 files
423 files did not match
Verify successful
In this case, python packages that were installed on top of the default
install were flagged as mismatched. `swupd` can be directed to ignore
or fix issues based on :ref:`command line options <man_swupd>`.
Fixing hash mismatches
======================
`swupd` can correct any issues it detects. Additional directives can be
added including a white list of directories that will be ignored, if
desired.
The following command will repair issues, remove unknown items, and
ignore files or directories matching `/usr/lib/python`:
.. code-block:: bash
sudo swupd verify --fix --picky --picky-whitelist=/usr/lib/python
Bundles
*******
Listing installed bundles
=========================
You can list all of the bundles currently installed on the system
.. code-block:: bash
sudo swupd bundle-list --all
Finding a bundle containing a binary
====================================
Run the following to display a list of bundles that contain a particular
binary. Note that it may be present in multiple bundles:
.. code-block:: bash
swupd search -b <binary you want>
Adding a bundle
===============
Start by selecting a bundle from the list of :ref:`bundles`. In
this example we're adding dev-utils-dev, which is useful for development.
.. code-block:: bash
sudo swupd bundle-add dev-utils-dev
Removing a bundle
=================
Dependencies common to other bundles will not be removed
.. code-block:: bash
sudo swupd bundle-remove dev-utils-dev
.. _source code: https://github.com/clearlinux/swupd-client/blob/master/docs/swupd.1.rst
source/clear-linux/guides/maintenance/swupd-search.rst
-140
View File
@@ -1,140 +0,0 @@
.. _swupd-search:
Use swupd search to find bundles
################################
This document explains how to use `swupd search` to find and add
bundles in |CL-ATTR|.
Assumptions
***********
This guide assumes you:
* Possess a basic knowledge of :ref:`swupd <swupd-guide>`
* Understand :ref:`how swupd differs <swupd-about>` from
other Linux\* distributions
* May :ref:`mixer` to produce a custom distribution/image
How do I search for a bundle?
*****************************
Use `swupd search` to locate the bundle where the application binary exists.
Example: Kata\* Containers
==========================
Containers have revolutionized the way we manage cloud infrastructure.
Traditional containers often share the same OS kernel, which raises
security concerns. Instead, with Kata Containers, each container has its own
kernel instance and runs on its own :abbr:`Virtual Machine (VM)`. Whether you're running 3 or 300 nodes on your cluster, Kata Containers provide a
lightweight, fast, and secure option for app/container management.
In |CL|, you only need to add `one bundle`_ to use `Kata Containers`_:
`containers-virt`. Also, check out our tutorial: :ref:`kata`.
We need to find *kata* containers in a bundle. How do we search for it?
#. Enter :command:`swupd search`, followed by 'kata' as the search term:
.. code-block:: bash
sudo swupd search kata
.. note::
`swupd search` downloads |CL| manifest data and searches for
matching paths. Enter only one term, or hyphenated term, per
search. Use the command :command:`man swupd` to learn more.
#. Alternatively, if you want to search binaries only, add the `-b`
flag:
.. code-block:: bash
sudo swupd search -b kata
.. note::
`-b` flag, or `--binary`, means: Restrict search to program binary paths. Omit this flag if you want a larger scope of search results.
Only the base bundle is returned. In |CL|, *bundles* can contain
other *bundles* via `includes`. For more details, see `Bundle Definition Files`_ and its subdirectory *bundles*.
If your search does not produce results on a specific term when using
the `-b` flag, abbreviate the search term. For example, if you search
for *kubernetes* and it does not show results, instead abbreviate the
term to *kube* to show results.
#. Optionally, you can review our `bundles`_ or individual `packages`_
#. Using `sudo swupd search -b kata` shows a match for our use case.
.. code-block:: console
Bundle containers-virt (834 MB to install)
/usr/bin/kata-virtfs-lite-proxy-helper
/usr/bin/kata-runtime
/usr/bin/kata-qemu-lite-system-x86_64
/usr/bin/kata-qemu-lite-pr-helper
/usr/bin/kata-qemu-lite-ga
/usr/bin/kata-collect-data.sh
.. note::
If the bundle is already installed, *[installed]* appears in search results. If this doesn't apppear, the bundle needs to be installed.
#. Add the bundle `containers-virt`:
.. code-block:: bash
sudo swupd bundle-add containers-virt
.. note::
To add multiple bundles simply add a space followed by the bundle name.
#. When prompted, enter your password.
#. Upon successful installation, your console should show similar data:
.. code-block:: console
Downloading packs...
Extracting containers-virt pack for version 24430
...50%
Extracting kernel-container pack for version 24430
...100%
Starting download of remaining update content. This may take a while...
...100%
Finishing download of update content...
Installing bundle(s) files...
...100%
Calling post-update helper scripts.
Successfully installed 1 bundle
Related Topics
==============
Learn how to:
* :ref:`kata`
* :ref:`kata_migration`
* :ref:`swupd-guide`
* :ref:`Show all available bundles <swupd-guide>`
* :ref:`Remove bundles<swupd-guide>`
.. _Kata Containers: https://clearlinux.org/blogs/clear-linux-os-announces-support-kata-containers
.. _one bundle: https://github.com/clearlinux/clr-bundles/blob/master/bundles/containers-virt
.. _Bundle Definition Files: https://github.com/clearlinux/clr-bundles
.. _bundles: https://github.com/clearlinux/clr-bundles/tree/master/bundles
.. _packages: https://github.com/clearlinux/clr-bundles/blob/master/packages
source/substitutions.txt
+5
View File
@@ -9,3 +9,8 @@
.. |CAUTION-BACKUP-USB| replace::
Burning an image formats the USB drive, and will destroy all pre-existing
content. Back up your data before proceeding.
.. |CAUTION-UNMOUNT-USB-PARTITIONS| replace::
Not fully unmounting the USB drive before burning an image could cause
file system checksum errors in it. If this happens, burn the image again
ensuring all the USB drive partitions are unmounted first.