diff --git a/source/clear-linux/get-started/bootable-usb/bootable-usb.rst b/source/clear-linux/get-started/bootable-usb/bootable-usb.rst index 3994b20f..e0ac2a2a 100644 --- a/source/clear-linux/get-started/bootable-usb/bootable-usb.rst +++ b/source/clear-linux/get-started/bootable-usb/bootable-usb.rst @@ -78,7 +78,7 @@ Burn the |CL| image onto a USB drive ├─sda8 8:8 0 30G 0 part / └─sda6 8:6 0 7.9G 0 part [SWAP] -#. You must unmount a USB drive before you can burn an image onto it. Note that +#. You must unmount a USB drive before burning an image onto it. Note that some Linux distros automatically mount a USB drive when it is plugged in. Unmount a USB drive with the :command:`umount` command followed by the device identifier/partition. For example: @@ -93,7 +93,11 @@ Burn the |CL| image onto a USB drive .. code-block:: bash - dd if=./clear-[version number]-[image type] of= bs=4M status=progress && sync + dd if=./clear-[version number]-[image type] of= oflag=sync bs=4M status=progress + +.. caution:: + + |CAUTION-UNMOUNT-USB-PARTITIONS| .. _bootable-usb-mac: diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/clearlinux/autospec.rst similarity index 100% rename from source/clear-linux/guides/maintenance/autospec.rst rename to source/clear-linux/guides/clearlinux/autospec.rst diff --git a/source/clear-linux/guides/maintenance/ister.rst b/source/clear-linux/guides/clearlinux/ister.rst similarity index 100% rename from source/clear-linux/guides/maintenance/ister.rst rename to source/clear-linux/guides/clearlinux/ister.rst diff --git a/source/clear-linux/guides/clearlinux/mixer.rst b/source/clear-linux/guides/clearlinux/mixer.rst new file mode 100644 index 00000000..a786a786 --- /dev/null +++ b/source/clear-linux/guides/clearlinux/mixer.rst @@ -0,0 +1,843 @@ +.. _mixer: + +mixer +##### + +mixer is the tool used by the |CL-ATTR| team to generate official update content +and releases. The update content generated by mixer is then consumed by swupd on +a downstream client. The same mixer tool is available as part of |CL| to create +your own customized update content and releases. + +.. contents:: + :local: + :depth: 1 + +Description +*********** + +mixer uses the following sources as inputs to generate update content: + +* Upstream |CL| bundles with their corresponding RPM packages +* Locally-defined bundles with their corresponding local RPM packages +* Locally-defined bundles with upstream RPM packages + +Using the mixer tool, you select which set of content from these sources will be +part of your update. You can select content from each of these sources to make a +unique combination of functionality for your custom update content (known as a +mix). + +The update content that mixer generates consists of various pieces of OS +content, update metadata, as well as a complete image. The OS content includes +all files in an update, as well as zero- and delta-packs for improved update +performance. The metadata, stored as manifests, describes all of the bundle +information for the update. Update content produced by mixer is then published +to a web server and consumed by clients via swupd. Refer to +:ref:`swupd ` for additional information regarding updates and +update content. + +How it works +************ + +Learn the mixer tool set up and workflow. + +.. contents:: + :local: + :depth: 1 + +Prerequisites +============= + +* :command:`mixer` bundle + + Add the mixer tool with the :command:`mixer` bundle. Refer to + `Install a bundle`_ for more details. + +* Docker container + + mixer by default runs all build commands in a Docker container to make sure + the correct tool versions are used. This also allows custom mixes to + automatically perform downstream format bumps when the upstream releases a + format bump. See `Format version`_ for additional information regarding + format bumps. + + Refer to `Configure and enable Docker`_ for instruction. + +* Docker proxy (optional) + + If you use a proxy server, you must set your proxy environment variables and + create a proxy configuration file for the Docker daemon and container. + + Consult your IT department if you are behind a corporate proxy for the correct + values. + + Refer to `Configure Docker proxy info`_ for instruction. + +* Location to host the update content and images + + In order for swupd to make use of your mix, the update content for your mix + must be hosted on a web server. Your mix will be configured with an update + location URL, which swupd will use to pull down updates. + + Refer to `Set up a nginx web server for mixer`_ for an simple example of + setting up an update location. + +Mix setup +========== + +Follow these steps to create and initialize the mixer workspace. Complete +setup before you create a mix. + +#. Create workspace. + + The mixer tool uses a simple workspace to contain all input and output in a + basic directory structure. The workspace is simply an empty folder that you + will execute the mixer commands from. Each mix will use its own separate + workspace. + +#. Initialize the workspace and mix. + + Before you create a mix, you must explicitly initialize the mixer workspace. + During initialization, the mixer workspace is configured and the base for + your mix is defined. By default, your mix will be based on the latest + upstream version and start with the minimum set of bundles. Your first custom + mix version number will start at 10. You can alternately select other + versions or bundle sets to start from. + + Initialization creates the directory structure within the workspace and adds + the :file:`builder.conf` file, which is used to configure the mixer tool. + + View the `mixer.init man page`_ for more information on mixer + initialization. + + View the list of `suitable versions`_ to mix from. + +#. Edit builder.conf. + + :file:`builder.conf` tells the mixer tool how to configure the mix. For + example, it allows you to configure where mixer output is located and where + swupd update content will be located. + + At minimum, set the URL of your update server so your custom OS knows where + to get update content. + + Refer to the `builder.conf`_ section for more information. + +Create a mix +============ + +A mix is created with the following steps: + +#. Add custom RPMs and set up local repo (optional). + + If you are adding custom RPMs to your mix, you will need to add the RPMs to + your mix workspace and set up a corresponding local repository. + + Go to the :ref:`autospec` guide to learn to build RPMs from + scratch. If the RPMs are not built on |CL|, make sure your + configuration and toolchain builds them correctly for |CL|. Otherwise there + is no guarantee they will be compatible. + + Refer to the :ref:`autospec` guide for more information on using autospec to + build RPMs. + +#. Update and build bundles. + + Add, edit, or remove bundles that will be part of your content and build + them. mixer will automatically update the :file:`mixbundles` file when you + update the bundles in your mix. + + View the `mixer.bundle man page`_ for more information on configuring bundles + in a mix. + + View the `mixer.build man page`_ for more information on building bundles. + + View the `Bundles`_ section for more information on how mixer manages + bundles. + +#. Create the update content. + + mixer creates update content with this step. Zero-packs are created + automatically, and delta-packs can be optionally created at the same time + (for all builds after version 0). + + A zero-pack is the full set of content needed to go from mix version 0 + (nothing) to the mix version you just built content for. + + A delta-pack provides the content *delta* between a `PAST_VERSION` to a + `MIX_VERSION` which allows the transition from one mix version to another. + + View :ref:`swupd-guide` for more information on update content. + +#. Create image. + + mixer creates a bootable image from your updated content using + the ister tool. In this step you can specify which bundles you want + *preinstalled* in the image. Users can later install other bundles available + in your mix. + +#. Make update available. + + Deploy update content and images to your update server. + + View the `Example 3: Deploy updates to target`_ for a simple deployment + scenario. + +Maintain or modify mix +====================== + +Update or modify your content to a new version by following the same steps to +create a mix. Increment the mix version number for the next mix. + +Examples +******** + +The following examples are designed to work together and in order. The examples +use: + +* A stock installation of |CL|. +* A web server that comes with |CL| to host the content updates. +* A simple VM that will update against the locally produced content created in + Example 2. + +Complete all `Prerequisites`_ before using these examples. + +Example 1: Mix set up +====================== + +This example shows the basic steps for first time setup of mixer for a new mix. + +#. Create an empty directory to use as a workspace for mixer: + + .. code-block:: bash + + mkdir ~/mixer + +#. In your mixer workspace, generate an initial mix based on the latest upstream + |CL| version, with minimum bundles: + + .. code-block:: bash + + cd ~/mixer + mixer init + + Note in the initialization output, that your initial mix version is set to + 10 and that the minimum bundles have been added. + +#. Edit :file:`builder.conf` to set the value of CONTENTURL and VERSIONURL to + the IP address of the nginx server you set up in the prerequisite + `Set up a nginx web server for mixer`_. For example: + + .. code-block:: console + + CONTENTURL="http://192.168.25.52" + VERSIONURL="http://192.168.25.52" + + + +Example 2: Create a simple mix +============================== + +This example shows how to create a simple custom mix using upstream content. +We'll create an image for a QEMU virtual machine which we can later use to test +our mix. + +We can use the default bundles that were added during intialization, but these +include the :command:`native-kernel` bundle which is intended to be used on a +bare metal system instead of a VM. So we will modify the default bundle +set to get a smaller kernel image, which will also be faster to load. + +#. Update bundles in mix: + + .. code-block:: bash + + mixer bundle remove kernel-native + mixer bundle add kernel-kvm + +#. Build bundles: + + .. code-block:: bash + + mixer build bundles + + Look in ~/mixer/update/image//full for the full chroot after the + :command:`build` command completes. + +#. Build update content. Browse to your http://localhost site and you'll see + the web page is now up, but with no update content. Build the update content: + + .. code-block:: bash + + mixer build update + + Refresh your http://localhost site and now you can see the update content for + mix version 10. + + Look in ~/mixer/update/www/ to see the update content in your + workspace. + +#. Configure image. Edit the ister configuration file for your image to include + all of the bundles you want preinstalled in the image. If this is the first + time creating an image, first get a copy of the + :file:`release-image-config.json` template file: + + .. code-block:: bash + + curl -O https://raw.githubusercontent.com/bryteise/ister/master/release-image-config.json + + For this example, edit :file:`release-image-config.json` so that the root + partition size is "5G" and replace the "kernel-native" bundle with + "kernel-kvm". + + .. code-block:: console + + { + "DestinationType" : "virtual", + "PartitionLayout" : [ { "disk" : "release.img", "partition" : 1, "size" : "32M", "type" : "EFI" }, + { "disk" : "release.img", "partition" : 2, "size" : "16M", "type" : "swap" }, + { "disk" : "release.img", "partition" : 3, "size" : "5G", "type" : "linux" } ], + "FilesystemTypes" : [ { "disk" : "release.img", "partition" : 1, "type" : "vfat" }, + { "disk" : "release.img", "partition" : 2, "type" : "swap" }, + { "disk" : "release.img", "partition" : 3, "type" : "ext4" } ], + "PartitionMountPoints" : [ { "disk" : "release.img", "partition" : 1, "mount" : "/boot" }, + { "disk" : "release.img", "partition" : 3, "mount" : "/" } ], + "Version": "latest", + "Bundles": ["kernel-kvm", "os-core", "os-core-update"] + } + +#. Build the image. + + .. code-block:: bash + + sudo mixer build image + + The output from this step will be :file:`release.img`, which is a live image. + +#. Make the next mix. Create a new version of your mix, for the live image to + update to. Increment your mix version by 10: + + .. code-block:: bash + + mixer versions update + + Repeat steps 1-3 to add the upstream :command:`curl` bundle to the mix: + + .. code-block:: bash + + mixer bundle add curl + mixer build bundles + mixer build update + + And build optional delta-packs, which will help reduce client update time: + + .. code-block:: bash + + mixer build delta-packs --from 10 --to 20 + + Refresh your http://localhost site and now you can see the update content for + mix version 20. + + Look in ~/mixer/update/www/ to see the update content in your + workspace. + +Example 3: Deploy updates to target +=================================== + +The image created in Example 2 is directly bootable in QEMU. In this example, +we'll boot the image from Example 2 to verify it, and update the image from mix +version 10 (which the image was built from), to mix version 20. + +#. Set up the QEMU environment. + + Install the :command:`kvm-host` bundle to your |CL|: + + .. code-block:: bash + + sudo swupd bundle-add kvm-host + + Get the virtual EFI firmware, download the image launch script, and make it + executable: + + .. code-block:: bash + + curl -O https://download.clearlinux.org/image/OVMF.fd + curl -O https://download.clearlinux.org/image/start_qemu.sh + chmod +x start_qemu.sh + +#. Start your VM image (created in Example 2): + + .. code-block:: bash + + sudo ./start_qemu.sh release.img + +#. Log in as root and set a password + +#. Try out your mix. + + Take a look at the default bundles installed in your mix: + + .. code-block:: bash + + swupd info + swupd bundle-list + swupd bundle-list -a + + Note that you cannot see the curl bundle that you added in Example 2 because + your mix is still on version 10. + + Check for updates. You should see that version 20 is available. Use swupd to + update your mix: + + .. code-block:: bash + + swupd check-update + swupd update + swupd bundle-list -a + + Now your mix should be at version 20 and curl is now available. Try using + curl. This will fail as curl is not yet installed: + + .. code-block:: console + + curl: command not found + To install curl use: swupd bundle-add curl + + Add the new bundle from your update server to your VM. Retry curl. It works! + + .. code-block:: bash + + swupd bundle-add curl + curl -O https://download.clearlinux.org/image/start_qemu.sh + + And shutdown your VM: + + .. code-block:: bash + + poweroff + +.. Example: Create a mix with custom RPM +.. ------------------------------------- +.. TODO future example to show copy into local-rpms... + +References +********** + +Reference the `mixer man page`_ for details regarding mixer commands and options. + +.. contents:: + :local: + :depth: 1 + +.. rst-class:: content-collapse + +builder.conf +============ + +mixer initialization creates a :file:`builder.conf` that stores the basic +configuration for the mixer tool. The items of primary interest are CONTENTURL +and VERSIONURL, which will be used by systems updating against your custom +content. + +.. code-block:: console + + #builder.conf + + #VERSION 1.0 + + [Builder] + CERT = "/home/clr/mix/Swupd_Root.pem" + SERVER_STATE_DIR = "/home/clr/mix/update" + VERSIONS_PATH = "/home/clr/mix" + YUM_CONF = "/home/clr/mix/.yum-mix.conf" + + [Swupd] + BUNDLE = "os-core-update" + CONTENTURL = "" + VERSIONURL = "" + + [Server] + DEBUG_INFO_BANNED = "true" + DEBUG_INFO_LIB = "/usr/lib/debug" + DEBUG_INFO_SRC = "/usr/src/debug" + + [Mixer] + LOCAL_BUNDLE_DIR = "/home/clr/mix/local-bundles" + LOCAL_REPO_DIR = "" + LOCAL_RPM_DIR = "" + DOCKER_IMAGE_PATH = "clearlinux/mixer" + +Additional explanation of variables in :file:`builder.conf` is provided in Table +1. + ++-------------------------------+----------------------------------------------------------+ +| **Variable** | **Explanation** | ++-------------------------------+----------------------------------------------------------+ +| `CERT` | Sets the path where mixer stores the certificate file | +| | used to sign content for verification. mixer | +| | automatically generates the certificate if you do not | +| | provide the path to an existing one, and signs the | +| | :file:`Manifest.MoM` file to provide security for the | +| | updated content you create. | +| | | +| | The chroot-builder uses the certificate file to sign | +| | the root :file:`Manifest.MoM` file, to provide | +| | security for content verification. | +| | | +| | The swupd uses this certificate to verify the | +| | :file:`Manifest.MoM` file's signature. | +| | | +| | For now, we strongly recommend that you do not modify | +| | this variable, as swupd expects a certificate with a | +| | very specific configuration to sign and verify | +| | properly. | ++-------------------------------+----------------------------------------------------------+ +| `CONTENTURL` and `VERSIONURL` | Set these variables to the IP address of the web server | +| | hosting the update content. | +| | | +| | VERSIONURL is the IP address where the swupd client | +| | looks to determine if a new version is available. | +| | | +| | CONTENTURL is the location where swupd will pull content | +| | updates from. | +| | | +| | If the web server is on the same machine as the | +| | SERVER_STATE_DIR directory, you can create a symlink to | +| | the directory in your web server's document root to | +| | easily host the content. | +| | | +| | These URLs are embedded in the images created by mixer. | ++-------------------------------+----------------------------------------------------------+ +| `DOCKER_IMAGE_PATH` | Sets the base name of the docker image mixer will pull | +| | down in order to run builds in the proper container. | ++-------------------------------+----------------------------------------------------------+ +| `LOCAL_BUNDLE_DIR` | Sets the path where mixer stores the local bundle | +| | definition files. The bundle definition files include | +| | any new, original bundles you create, along with any | +| | edited versions of upstream bundles. | ++-------------------------------+----------------------------------------------------------+ +| `SERVER_STATE_DIR` | Sets the path for where mixer outputs content. By | +| | default, mixer will automatically set the path. | ++-------------------------------+----------------------------------------------------------+ +| `VERSIONS_PATH` | Sets the path for the mix version and upstream version's | +| | two state files: :file:`mixversion` and | +| | :file:`upstreamversion`. mixer creates both files for | +| | you when you set up the workspace. | ++-------------------------------+----------------------------------------------------------+ +| `YUM_CONF` | Sets the path where mixer automatically generates the | +| | :file:`.yum-mix.conf` file. | +| | | +| | The yum configuration file points the chroot-builder to | +| | where the RPMs are stored. | ++-------------------------------+----------------------------------------------------------+ +| **Table 1**: *Variables in builder.conf* | ++-------------------------------+----------------------------------------------------------+ + +Format version +-------------- + +Compatible versions of an OS are tracked with an OS *compatibility epoch*. +Versions of an OS within an epoch are fully compatible and can update to any +other version within that epoch. The compatibility epoch is set as the `Format` +variable in the :file:`mixer.state` file. Variables in the :file:`mixer.state` +are used by mixer between executions and should not be manually changed. + +If `Format` increments to a new epoch (a "format bump"), the OS has changed in +such a way that updating from build M in format X, to build N in format Y will +not work. Generally, this scenario occurs when the software updater/the software +has a change such that it is no longer compatible with the previous update +scheme, or when a package is removed from the update stream and the update +must ensure the files associated with that package are removed from the system. + +Using a format increment, we make sure pre- and co-requisite changes flow out +with proper ordering. The updated client will only update to the latest +release in its respective format version, unless overridden by command line +flags. This way we can guarantee that all clients update to the final version +in their given format. + +The given format *must* contain all the changes needed to understand the content +built in the next format. Only after reaching the final release in the old +format can a client continue to update to releases in the new format. + +The format version is incremented only when a compatibility breakage is +introduced. Normal updates, like updating a software package, do not require a +format increment. + +.. rst-class:: content-collapse + +Bundles +======= + +mixer stores information about the bundles included in a mix in a flat file +called :file:`mixbundles`, located in the path set by the VERSIONS_PATH +variable in :file:`builder.conf`. :file:`mixbundles` is automatically created +when the mix is initiated. mixer will refresh the file each time you change the +bundles in the mix. + +Bundles can include other bundles. Nested bundles can themselves include other +bundles. If you see an unexpected bundle in your mix, it is likely a nested +bundle in one of the bundles you explicitley added. + +A bundle will fill into one of two categoris: upstream or local. Upstream +bundles are those provided by |CL|. Local bundles are either modified upstream +bundles or new local bundles. + +Upstream bundles +---------------- + +mixer automatically downloads and caches upstream bundle definition files. These +definition files are stored in the upstream-bundles directory in the workspace. +Do not modify the files in this directory. This directory is simply a mirror for +mixer to use. mixer will automatically delete the contents of this directory +before repopulating it on-the-fly if a new version must be downloaded. + +The mixer tool automatically caches the bundles for the |CL| version configured +in the :file:`upstreamversion` file. mixer also cleans up old versions once they +are no longer needed. + +Local bundles +------------- + +Local bundles are bundles that you create, or are edited versions of upstream +bundles. Local bundle definition files are stored in the local-bundles directory +in the workspace. The LOCAL_BUNDLE_DIR variable sets the path of this directory +in the :file:`builder.conf` file. + +*mixer always checks for local bundles first and the upstream bundles second.* +So bundles in the local-bundles directory will always take precedence over any +upstream bundles that have the same name. This precedence enables you to copy +upstream bundles locally, and edit into a local variation. + +Bundle configuration +-------------------- + +mixer provides commands to configure the bundles for a mix, for example to add a +bundle to a mix, to create a new bundle for a mix, or to remove a bundle from a +mix. View the `mixer.bundle man page`_ for a full list of commands and more +information on configuring bundles in a mix. + +Editing an existing local bundle is as simple as opening the bundle definition +file in your favorite editor, making the desired edits, and saving your changes. + +A note on removing bundles from a mix: By default, removing a bundle will only +remove the bundle from the mix. The local bundle defintion file will still +remain. To completely remove a bundle, including its local bundle definition +file, use the :command:`--local` flag. + +If you remove the bundle definition file for a local, edited version of an +upstream bundle in a mix, the mix will revert to reference the original upstream +version of the bundle. + +.. rst-class:: content-collapse + +Configure and enable Docker +=========================== + +Use these steps to enable Docker for the mixer tool. Make sure to +`Configure Docker proxy info`_ first if needed. + +#. Start the Docker daemon: + + .. code-block:: bash + + sudo systemctl start docker + sudo chmod 777 /var/run/docker.sock + sudo docker info + +#. Add user to the docker group + + .. code-block:: bash + + sudo usermod -G docker -a + +Pull Docker container manually (optional) +----------------------------------------- + +By default, mixer will automatically pull a Docker container for mixing if one +does not already exist. If you need to troubleshoot the mixer container, it may +be useful to manually pull a mixer Docker container. + +Versions of the mixer Docker container are available under the tags for the +`clearlinux/mixer repo `_ +on Docker Hub. Each version of the mixer Docker container is named after the +associated |CL| upstream format version. Refer to `Format version`_ for +additional information on upstream format versions. + +Use the following steps to manually pull a mixer Docker container: + +#. Find the version of the container you need by viewing the tags for the + `clearlinux/mixer repo `_ + on Docker Hub. + +#. Pull the latest container version: + + .. code-block:: bash + + docker pull clearlinux/mixer: + +#. View local docker images: + + .. code-block:: bash + + docker images + +.. rst-class:: content-collapse + +Configure Docker proxy info +=========================== + +If needed, use these steps to configure the Docker proxy information. + +Configure the Docker daemon proxies: + +#. Create the Docker daemon proxy config directory: + + .. code-block:: bash + + sudo mkdir -p /etc/systemd/system/docker.service.d + + Create :file:`/etc/systemd/system/docker.service.d/http-proxy.conf` and + add the following using your own proxy values: + + .. code-block:: console + + [Service] + Environment="HTTP_PROXY=:" + Environment="HTTPS_PROXY=:" + +#. Reload the Docker daemon: + + .. code-block:: bash + + sudo systemctl daemon-reload + +Configure the Docker container proxies, in order to pass proxy +settings to containers: + +#. Create a directory for your container config: + + .. code-block:: bash + + mkdir ~/.docker + +#. Create the config file :file:`~/.docker/config.json` and add the following + entries, using your own proxy values: + + .. code-block:: console + + { + "proxies": + { + "default": +   { +    "httpProxy": ":", +     "httpsProxy": ":" +   } + } + } + +#. Set ownership and permission on the docker config directory: + + .. code-block:: bash + + sudo chown "$USER":"$USER" /home/"$USER"/.docker -R + sudo chmod g+rwx "$HOME/.docker" -R + +Lastly, configure proxies to allow mixer to access upstream content from behind +a firewall. For example: + +#. Open your :file:`$HOME/.bashrc` file and add proxy and port values for the + following: + + .. code-block:: console + + export http_proxy=":" + export https_proxy=":" + export HTTP_PROXY=":" + export HTTPS_PROXY=":" + export no_proxy="<...>" + +#. Log out and log back in for the proxies to take effect. + +.. rst-class:: content-collapse + +Set up a nginx web server for mixer +=================================== + +A web server is needed to host your update content. In this example, we use the +nginx web server, which comes with |CL|. + +Set up a nginx web server for mixer with the following steps: + +#. Install the :command:`nginx` bundle: + + .. code-block:: bash + + sudo swupd bundle-add nginx + +#. Make the directory where mixer updates will reside: + + .. code-block:: bash + + sudo mkdir -p /var/www + +#. Create a symbolic link between your workspace updates and the updates on the + local nginx web server. In this example, `$HOME/mixer` is the workspace for + the mix. + + .. code-block:: bash + + sudo ln -sf $HOME/mixer/update/www /var/www/mixer + +#. Set up ``nginx`` configuration: + + .. code-block:: bash + + sudo mkdir -p /etc/nginx/conf.d + +#. Copy the default example configuration file: + + .. code-block:: bash + + sudo cp -f /usr/share/nginx/conf/nginx.conf.example /etc/nginx/nginx.conf + +#. Configure the mixer update server. Create and add the following server + configuration content to :file:`/etc/nginx/conf.d/mixer.conf` (sudo required): + + .. code-block:: console + + server { + server_name localhost; + location / { + root /var/www/mixer; + autoindex on; + } + } + +#. Restart the daemon, enable nginx on boot, and start the service. + + .. code-block:: bash + + sudo systemctl daemon-reload + + sudo systemctl enable nginx + + sudo systemctl start nginx + +#. Verify the web server is running at \http://localhost. At this point + you should no longer see a "404 Not Found" message. + +Related topics +************** + +* :ref:`About mixer ` +* :ref:`mixin` +* :ref:`autospec-about` +* :ref:`bundles-about` +* :ref:`swupd-about` + +.. _Docker Hub: https://hub.docker.com/r/clearlinux/mixer/tags/ +.. _mixer man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.1.rst +.. _mixer.init man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.init.1.rst +.. _mixer.bundle man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.bundle.1.rst +.. _mixer.build man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.build.1.rst +.. _suitable versions: https://github.com/clearlinux/clr-bundles/releases +.. _Install a bundle: https://clearlinux.org/documentation/clear-linux/guides/maintenance/swupd-guide#adding-a-bundle diff --git a/source/clear-linux/guides/maintenance/mixin.rst b/source/clear-linux/guides/clearlinux/mixin.rst similarity index 100% rename from source/clear-linux/guides/maintenance/mixin.rst rename to source/clear-linux/guides/clearlinux/mixin.rst diff --git a/source/clear-linux/guides/clearlinux/swupd-guide.rst b/source/clear-linux/guides/clearlinux/swupd-guide.rst new file mode 100644 index 00000000..adbe6a1b --- /dev/null +++ b/source/clear-linux/guides/clearlinux/swupd-guide.rst @@ -0,0 +1,355 @@ +.. _swupd-guide: + +swupd +##### + +:command:`swupd` links a |CL-ATTR| installation with upstream updates and +software. + +.. contents:: + :local: + :depth: 2 + +Description +*********** + +:command:`swupd` has two main functions: + +#. It manages software replacing APT or YUM, installing bundles + rather than packages. +#. It checks for system updates and installs them. + +:ref:`Bundles ` are the smallest granularity component that is +managed by |CL| and contain everything needed to deliver a software +capability. Rather than downloading a cascade of package dependencies when +installing a piece of software, a bundle comes with all of its dependencies. +:command:`swupd` manages overlapping dependencies behind the scenes ensuring +that all software is compatible across the system. + +Versioning +========== + +In a traditional distribution, the process of describing current software +versioning usually involves: + +- Listing and keeping track of the current OS release (generally + uninformative about any singular packages or functionality). + +- Keeping track of packages and repositories being used, and updating them + individually. + +- Listing and tracking every package available and installed on the + system, none of which are directly tied to the current OS release. + +This can be done effectively, but given the nearly endless combinations of +packages and versions of packages a server may have, it quickly becomes +non-trivial to define what "version" the system is and what software it +is running without explicitly going through each system and inspecting +every package. + +With |CL|, we need track: + +- One number + +A number representing the **current** release of the OS is sufficient to +describe the versions of all the software on the OS. Each build is +composed of a specific set of bundles made from a particular version of +packages. This matters on a daily basis to system administrators, who +need to determine which of their systems do not have the latest security +fixes, or which combinations of software have been tested. Every release +of the same number is guaranteed to contain the same versions of software, +so there's no ambiguity between two systems running the same version of |CL|. + +Updating +======== + +|CL| enforces regular updating of the OS by default and will automatically +check for updates against a version server. The content server provides the +file and metadata content for all versions and can be the same as the +version server. The content url server provides metadata in the form of +manifests. These Manifest files list and describe file contents, symlinks, +and directories. Additionally, the actual content is +provided to clients in the form of archive files. + +Software updates with |CL| are also efficient. Unlike package-based +distributions, :command:`swupd` only updates files that have changed rather +than entire packages. For example, it is quite common for an OS security +patch to be as small as 15 KB. Using binary deltas, the |CL| is able to +apply only what is needed. + +To get a more detailed understanding of how to generate update content for +|CL| see the :ref:`mixer ` tool. + +How it works +************ + +Prerequisites +============= + +* The device is on a well-connected network. +* The device is able to connect to an update server. The default server is: + http://update.clearlinux.org + +Updates +======= + +|CL| updates are automatic by default but can be set to occur only on +demand. :command:`swupd` makes sure that regular updates are simple and +secure. It can also check the validity of currently installed files and +software and correct any problems. + +Manifests +--------- + +The Clear Linux OS software update content consists of data and +metadata. The data is the files that end up in the OS. The metadata +contains relevant information to properly provision the data to the OS +file system, as well as update the system and add or remove additional +content to the OS. + +The Manifests are mostly long lists of hashes that describe content. +Each bundle gets its own manifest file. There is a master manifest +file that describes all manifests to tie it all together. + +Fullfiles, packs, and delta packs +--------------------------------- + +The data that an update provisions to a system can be obtained in +three different ways. There are three different methods, and they +exist to optimize the delivery of content and speed up updates. + +Fullfiles are always generated for every file in every release. This +allows any Clear Linux OS to obtain the exact copy of the content +for each version directly. This would be used if the OS verification +needed to replace a single file, for instance. + +Packs are available for some releases and combine many files to speed +up the creation of installation media and large updates. Delta packs +are an optimized version of packs that only contain updates (binary +diffs) and cannot be used without having the original file content. + +Bundle Search +============= + +:command:`swupd` search downloads manifest data and searches for +bundles that match the term. Enter only one term, or hyphenated term, per +search. Use the command :command:`man swupd` to learn more. + +Only the base bundle is returned. Bundles can contain other bundles via +includes. For more details, see `Bundle Definition Files`_ and its +subdirectory bundles. + +Bundles that are already installed, will be marked [installed] in search +results. + +Optionally, you can review our `bundles`_ on GitHub. + +Examples +******** + +Example 1: Disable and Enable automatic updates +=============================================== + +|CL| updates are automatic by default but can be set to occur only +on demand. + +#. First verify your current auto-update setting. + + .. code-block:: bash + + sudo swupd autoupdate + + .. code-block:: console + + Enabled + +#. Disable automatic updates. + + .. code-block:: bash + + sudo swupd autoupdate --disable + + .. code-block:: console + + Warning: disabling automatic updates may take you out of compliance with your IT policy + + Running systemctl to disable updates + Created symlink /etc/systemd/system/swupd-update.service → /dev/null. + Created symlink /etc/systemd/system/swupd-update.timer → /dev/null. + +#. Check manually for updates. + + .. code-block:: bash + + sudo swupd check-update + +#. Install an update after identifying one that you need. + + .. code-block:: bash + + sudo swupd update -m + +#. Re-enable automatic installs. + + .. code-block:: bash + + sudo swupd autoupdate --enable + +.. _swupd-guide-example-install-bundle: + +Example 2: Find and install Kata\* Containers +============================================= + +Kata Containers is a popular container implementation. Unlike other +container implementations, each Kata Container has its own +kernel instance and runs on its own :abbr:`Virtual Machine (VM)` for +improved security. + +|CL| makes it very easy to install, since you only need to add +`one bundle`_ to use `Kata Containers`_: `containers-virt`, despite a +number of dependencies. Also, check out our tutorial: :ref:`kata`. + +#. Find the right bundle. + + * To return all possible matches for the search string enter + :command:`swupd search`, followed by 'kata': + + .. code-block:: bash + + sudo swupd search kata + + The output should be similar to: + + .. code-block:: console + + Bundle with the best search result: + + containers-virt - Run container applications from Dockerhub in lightweight virtual machines + + This bundle can be installed with: + + swupd bundle-add containers-virt + + Alternative bundle options are + + cloud-native-basic - Contains ClearLinux native software for Cloud + + .. note:: + + If your search does not produce results with a specific + term, shorten the search term. For example, use *kube* instead of + *kubernetes*. + +#. Add the bundle. + + .. code-block:: bash + + sudo swupd bundle-add containers-virt + + .. note:: + + To add multiple bundles simply add a space followed by the bundle name. + + The output of a successful installation should be similar to: + + .. code-block:: console + + Downloading packs... + + Extracting containers-virt pack for version 24430 + ...50% + Extracting kernel-container pack for version 24430 + ...100% + Starting download of remaining update content. This may take a while... + ...100% + Finishing download of update content... + Installing bundle(s) files... + ...100% + Calling post-update helper scripts. + Successfully installed 1 bundle + +Example 3: Verify and correct sytem file mismatch +================================================= + +:command:`swupd` can determine whether system directories and files have +been added to, overwritten, removed, or modified (e.g., permissions). + +.. code-block:: bash + + sudo swupd verify + +All directories that are watched by :command:`swupd` are verified according +to the manifest data and hash mismatches are flagged as follows: + +.. code-block:: console + + Verifying version 23300 + Verifying files + ...0% + Hash mismatch for file: /usr/bin/chardetect + ... + ... + Hash mismatch for file: /usr/lib/python3.6/site-packages/urllib3/util/wait.py + ...100% + Inspected 237180 files + 423 files did not match + Verify successful + +In this case, python packages that were installed on top of the default +install were flagged as mismatched. :command:`swupd` can be directed to +ignore or fix issues based on command line options. + +:command:`swupd` can correct any issues it detects. Additional directives +can be added including a white list of directories that will be ignored. + +The following command will repair issues, remove unknown items, and +ignore files or directories matching `/usr/lib/python`: + +.. code-block:: bash + + sudo swupd verify --fix --picky --picky-whitelist=/usr/lib/python + +Quick Reference +*************** + +swupd info + To see the currently installed version and update servers. + +swupd update + To update to a specific version or with no arguments to update to latest. + +swupd bundle-list [--all] + To list installed bundles. + +swupd bundle-add [-b] + To find a bundle that contains your search term. + +swupd bundle-add + To add a bundle. + +swupd bundle-remove + To remove a bundle. + +swupd --help + For additional :command:`swupd` commands. + +man swupd + To reference the :command:`swupd` man page, or see the + `source documentation`_ available on github. + +Related topics +************** + +* :ref:`autospec` +* :ref:`mixer` +* :ref:`bundles` + +.. _source documentation: https://github.com/clearlinux/swupd-client/blob/master/docs/swupd.1.rst + +.. _Kata Containers: https://clearlinux.org/containers + +.. _one bundle: https://github.com/clearlinux/clr-bundles/blob/master/bundles/containers-virt + +.. _Bundle Definition Files: https://github.com/clearlinux/clr-bundles + +.. _bundles: https://github.com/clearlinux/clr-bundles/tree/master/bundles \ No newline at end of file diff --git a/source/clear-linux/guides/maintenance/developer-workstation.rst b/source/clear-linux/guides/maintenance/developer-workstation.rst index fc0582d9..765c44c3 100644 --- a/source/clear-linux/guides/maintenance/developer-workstation.rst +++ b/source/clear-linux/guides/maintenance/developer-workstation.rst @@ -78,10 +78,11 @@ While your role may not neatly fit in one of these categories, consider using Ta `swupd` search ************** -We recommend trying out :ref:`swupd search `, to learn the +We recommend learning about :ref:`swupd `, to learn the commands to search for and add bundles relevant to your project. -:ref:`swupd-search` shows you how to: +The guide provides an :ref:`example ` +that shows you how to: * Use `swupd` to search for bundles * Use `swupd` to add bundles diff --git a/source/clear-linux/guides/maintenance/kernel-modules.rst b/source/clear-linux/guides/maintenance/kernel-modules.rst index 2339453a..94daac09 100644 --- a/source/clear-linux/guides/maintenance/kernel-modules.rst +++ b/source/clear-linux/guides/maintenance/kernel-modules.rst @@ -45,7 +45,7 @@ security practices, see the :ref:`security` page. You can search for kernel module file names, which end with the :file:`.ko` file extension, using the :command:`swupd search` command. For example: -:command:`sudo swupd search ${module_name}.ko`. See :ref:`swupd-search` for +:command:`sudo swupd search ${module_name}.ko`. See :ref:`swupd-guide` for more information. diff --git a/source/clear-linux/guides/maintenance/mixer.rst b/source/clear-linux/guides/maintenance/mixer.rst deleted file mode 100644 index f826ab4f..00000000 --- a/source/clear-linux/guides/maintenance/mixer.rst +++ /dev/null @@ -1,871 +0,0 @@ -.. _mixer: - -Use the mixer tool -################## - -*Mixing* refers to composing an operating system for specific use cases. -While the default |CL-ATTR| provides options to install bundles for various -server capabilities, some developers may wish to either augment the -operating system itself with functionality from their own packages or modify -the structure of current bundles to cater to their particular needs. - -Prerequisites -************* - -To start working with the mixer tools, you need a recent image of |CL| with -the `mixer` bundle installed. If the bundle is not yet installed, you can -add it with the :command:`swupd bundle-add` command as follows: - -.. code-block:: bash - - sudo swupd bundle-add mixer - -Current mixing workflow -*********************** -Mixer by default runs *all* build commands in a container to ensure the -correct version of the tooling is being used. This also allows custom mixes -to automatically perform downstream format bumps when upstream releases -a format bump. You can still run mixer natively by appending the *--native* flag to -any of the commands. - -.. note:: - You cannot run mixer if you are already in a container, unless you pass - *--native* to the command. Nested containerization is not supported, nor - is building images using the container mode. - -There are two different workflows to create your own mix. -First, if your mix only uses |CL| content, *skip Create custom RPMs* below. -Second, if your mix includes your own -:abbr:`RPMs (RPM Package Manager files)`, follow all steps below. - -.. contents:: - :local: - :depth: 1 - :backlinks: top - -The following sections contain detailed information on every step of -these workflows. - -.. _create-nginx-web-server: - -Create nginx web server to host mixer updates -============================================= - -Follow these steps to set up a HTTP service with ``nginx`` web -server, where you can host custom |CL| mixes: - -#. Install ``web-server-basic``. - - .. code-block:: bash - - sudo swupd bundle-add web-server-basic - -#. Make the directory where mixer updates will reside. - - .. code-block:: bash - - sudo mkdir -p /var/www - -#. Create a symbolic link. - - .. code-block:: bash - - sudo ln -sf /home/clr/mix/update/www /var/www/mixer - - .. note:: - - `/home/clr/mix` is the workspace for the mix (read `Create workspace`) - - -#. Set up ``nginx`` configuration. - - .. code-block:: bash - - sudo mkdir -p /etc/nginx/conf.d - -#. Copy the default example configuration file. - - .. code-block:: bash - - sudo cp -f /usr/share/nginx/conf/nginx.conf.example /etc/nginx/nginx.conf - -#. Open ``mixer.conf`` with an editor. - - .. code-block:: bash - - sudo nano /etc/nginx/conf.d/mixer.conf - -#. Add the server configuration content below to ``mixer.conf``. - - .. code-block:: console - - server { - server_name localhost; - location / { - root /var/www/mixer; - autoindex on; - } - } - -#. Restart the daemon, enable nginx on boot, and start the service. - - .. code-block:: bash - - sudo systemctl daemon-reload - - sudo systemctl enable nginx - - sudo systemctl start nginx - -#. To verify the web server is running, check it in an Internet browser: - - .. code-block:: bash - - http://localhost - -#. Alternatively, verify the server is running by entering: - - .. code-block:: bash - - ps aux | grep nginx - - .. note:: - - If `nginx` is active, a list should appear showing one master process - and a few worker processes. - -Connect the URL to mixer ------------------------- - -Add the URL of the `nginx` server to builder.conf. Your |CL| clients connect -to this URL to find the update content. - -#. Get the IP address of your nginx server: - - .. code-block:: bash - - networkctl status - -#. In the mixer workspace, edit `builder.conf` to set the value for - CONTENTURL and VERSIONURL to the IP `Address` of your `nginx` server. - - .. code-block:: bash - - nano builder.conf - - .. note:: - - For example: - CONTNENTURL=http://192.168.25.52 - VERSIONURL=http://192.168.25.52 - -Create a workspace -================== - -Use the following command to create an empty directory in your |CL| image to -use as a **workspace** for mixing: - -.. code-block:: bash - - mkdir /home/clr/mix - -This guide assumes your workspace location is :file:`/home/clr/mix`. - -Generate the starting point for your mix -======================================== - -In your workspace, initialize mixer with the following command: - -.. code-block:: bash - - mixer init - -This command initializes your workspace so you can make a mix at version 10 -based on the latest released upstream |CL| version. If a :file:`builder.conf` -file is not already present in your workspace, mixer creates a default -configuration file. Additionally, mixer creates several version and tracking -files, and two bundle directories: :file:`local-bundles` and -:file:`upstream-bundles`. - -If you wish to start with a different version of upstream |CL| or a -different initial mix version, you can specify these options as flags. -For example: - -.. code-block:: bash - - mixer init --upstream-version 21060 --mix-version 10 - - -Additionally, to build a mix with your own custom RPMs, use the optional -*--local-rpms* flag, for example: - -.. code-block:: bash - - mixer init --local-rpms - -This command creates the :file:`local-yum` and :file:`local-rpms` -directories in your mix workspace and adds their paths to the generated -:file:`builder.conf`. If the configuration file already exists, you must add -the paths manually. For more information on using these directories or -setting them up manually, see `Create or locate RPMs for the mix`_. - -If all upstream |CL| bundles will be part of the mix, you can easily add -them all during initialization with the optional *--all-upstream* flag. For example: - -.. code-block:: bash - - mixer init --all-upstream - -Finally, you may want to track the contents of your mixer workspace with a -git repository. This is a great way to track changes to your mix's content -or to revert to earlier versions if something goes wrong. Mixer can set this -up automatically with the optional *--git* flag, for example: - -.. code-block:: bash - - mixer init --git - -.. note:: - You can use any or all of the above optional flags at the same time, for example: - - .. code-block:: bash - - mixer init --upstream-version 21060 --mix-version 10 --local-rpms --all-upstream --git - -Edit builder.conf -================= - -To configure the mixer tool, edit the :file:`builder.conf` as needed. - -The file :file:`builder.conf` is read automatically from the current -workspace directory. Use the *--config* flag during initialization -to specify an alternate path to the file as needed. - -The :file:`builder.conf` file has different sections, for example: - -* The `[Builder]` section provides the mixer tools with the required - configuration options. This section defines the path where the generated - bundles and update metadata are published. - -* The `[swupd]` section contains specific update parameters. The - :abbr:`swupd-server (software update server)` creates an update using - said specific update parameters. - -Edit the configuration file according to your needs with the command: - -.. code-block:: bash - - vim /home/clr/mix/builder.conf - -Your version of the :file:`builder.conf` file should resemble the -following example: - -.. code-block:: console - - #VERSION 1.0 - - [Builder] - CERT = "/home/clr/mix/Swupd_Root.pem" - SERVER_STATE_DIR = "/home/clr/mix/update" - VERSIONS_PATH = "/home/clr/mix" - YUM_CONF = "/home/clr/mix/.yum-mix.conf" - - [Swupd] - BUNDLE = "os-core-update" - CONTENTURL = "" - VERSIONURL = "" - - [Server] - DEBUG_INFO_BANNED = "true" - DEBUG_INFO_LIB = "/usr/lib/debug" - DEBUG_INFO_SRC = "/usr/src/debug" - - [Mixer] - LOCAL_BUNDLE_DIR = "/home/clr/mix/local-bundles" - LOCAL_REPO_DIR = "" - LOCAL_RPM_DIR = "" - DOCKER_IMAGE_PATH = "clearlinux/mixer" - -The following variables require further explanation: - -* The `LOCAL_BUNDLE_DIR` variable sets the path where mixer stores the local - bundle definition files. These bundle definition files include any new, - original bundles you create, along with any edited versions of upstream - |CL| bundles. - -* The `SERVER_STATE_DIR` variable sets the path for the output of the mix - content. Mixer automatically creates the path for you, but the path can be - set to any location. In this example, we use the workspace directory. - -* The `YUM_CONF` variable sets the path where mixer automatically generates - the :file:`.yum-mix.conf` yum configuration file. The yum configuration - file points the chroot-builder to the path where the RPMs are stored. - -* The `CERT` variable sets the path where mixer stores the - :file:`Swupd_Root.pem` certificate file. The chroot-builder needs the - certificate file to sign the root :file:`Manifest.MoM` file to provide - security for content verification. The value of the `CERT` variable can - point to a different certificate. The chroot-builder inserts the - certificate specified in this value into the - :file:`/os-core-update/usr/share/clear/update-ca/` path. The software - update client uses this certificate to verify the :file:`Manifest.MoM` - file's signature. For now, we **strongly** recommend that you do not modify - this line, as the certificate that :abbr:`swupd (Software Updater)` - expects needs to have a very specific configuration to sign and verify - properly. Mixer automatically generates the certificate, if you do not - provide the path to an existing one, and signs the :file:`Manifest.MoM` - file to provide security for the updated content you create. - -* The `CONTENTURL` and `VERSIONURL` variables set the domain or IP address - where swupd looks for your update content and the corresponding version. - You must set these variables to the domain or IP-address of the server - hosting the update content. You can use any web server to host your update - content. To learn how to install and configure web server using |CL|, see - :ref:`create-nginx-web-server`. For our example, the web update content - within the `SERVER_STATE_DIR` directory is located here: - :file:`/home/clr/mix/update/www`. If the web server is on the same machine - as this directory, you can create a symlink to the directory in your web - server's document root to easily host the content. These URLs are - embedded in images created for your mix. The `swupd-client` looks at - these URLs to determine if a new version is available and the location - from where to download the updated content. These links are equivalent - to the |CL| `update page`_ but for the mix. - -* The `VERSIONS_PATH` variable sets the path for the mix version and upstream - |CL| version's two state files: :file:`mixversion` and - :file:`upstreamversion`. Mixer creates both files for you when you set up - the workspace. - -* The `DOCKER_IMAGE_PATH` variable sets the base name of the docker image - mixer will pull down in order to run builds in the proper container. - -.. note:: If you are working only with |CL| bundles, then - skip to `List, edit, create, add, remove, or validate bundles`_. - - -Create custom RPMs -================== - -Create or locate RPMs for the mix ---------------------------------- - -If you create RPMs from scratch, you can use `autospec`, `mock`, `rpmbuild`, -or similar tools to build them. If the RPMs are not built on |CL|, ensure -your configuration and toolchain builds them correctly for |CL|, or else -there is no guarantee they will be compatible. For more information on -building the RPMs properly, refer to our `build RPMs instructions`_. - -Import RPMs into workspace --------------------------- - -#. Create a :file:`local-rpms` directory in your workspace, for example, - :file:`/home/clr/mix/local-rpms`. - -#. Copy the RPMs into the directory you created. - -#. Add the following line to your :file:`builder.conf` file: - - .. code-block:: console - - LOCAL_RPM_DIR=/home/clr/mix/local-rpms - -Mixer uses this directory to find the RPMs to build a local RPM repo for -yum to use. - -Create a local RPM repo ------------------------ - -#. Create an empty directory in your workspace named :file:`local-yum`. -#. Add the path to your :file:`builder.conf` file: - - .. code-block:: console - - LOCAL_REPO_DIR=/home/clr/mix/local-yum - -#. With these values configured, generate the yum repo with the following - command: - - .. code-block:: bash - - mixer add-rpms - -After the tool exits, you should see the RPMs and a repository data -directory in :file:`/home/clr/mix/local-yum`. If the RPMs are not all in this -:file:`local-yum` directory, check to ensure that the RPM files are valid -and not corrupt. - -List, edit, create, add, remove, or validate bundles -==================================================== - -The bundles in the mix are specified in the mix bundle list. Mixer stores -this list as a flat file called :file:`mixbundles` in the path set by the -`VERSIONS_PATH` variable of the :file:`builder.conf` file. Mixer -automatically generates the :file:`mixbundles` list file during -initialization. Mixer reads and writes the bundle list file when you change -the bundles of the mix. - -List the bundles in the mix ---------------------------- - -To view the bundles already in the mix, enter the following command: - -.. code-block:: bash - - mixer bundle list - -This command shows a list of every bundle in the mix. Bundles can include -other bundles. Those nested bundles can themselves include other -bundles. When listing bundles with this command, mixer automatically -recurses through the includes to show every single bundle in the mix. - -If you see an unexpected bundle in the list, that bundle is probably included -in another bundle. Use the *--tree* flag to get a better view of how -a bundle ended up in the mix, for example: - -.. code-block:: bash - - mixer bundle list --tree - -This command shows a visual representation of the inclusion relationships -between the bundles in the mix. - -Bundles fall into two categories: **upstream** and **local**. - -Upstream bundles are those provided by |CL|. - -Mixer automatically downloads and caches upstream bundle definition files. -These definition files are stored in the :file:`upstream-bundles` directory -in the workspace. Do **not** modify the files in this directory. This -directory is simply a mirror for mixer to use. - -The mixer tool automatically caches the bundles for the |CL| version -configured in the :file:`upstreamversion` file. Mixer also cleans up old -versions once they are no longer needed. You can see the available upstream -bundles with the following command: - -.. code-block:: bash - - mixer bundle list upstream - -Local bundles are bundles that you create, or are edited versions of upstream -bundles. - -Local bundle definition files live in the :file:`local-bundles` directory. -The `LOCAL_BUNDLE_DIR` variable sets the path of this directory in your -:file:`builder.conf` configuration file. For this example, the path is -:file:`/home/clr/mix/local-bundles`. You can see the available local bundles -with the following command: - -.. code-block:: bash - - mixer bundle list local - -Both the local and upstream :command:`bundle list` commands accept the -*--tree* flag to show a visual representation of the inclusion relationships -between the bundles in the mix. - -Edit the bundles in the mix ---------------------------- - -**Mixer always checks local bundles first and the upstream bundles second.** - -Therefore, bundles in the :file:`local-bundles` directory always take -precedence over any upstream bundles that have the same name. - -This precedence enables you to edit upstream bundles. The local, edited -version of the bundle overrides the bundle version found upstream. - -For example, to edit the `bundle1` definition file, we use the following -command: - -.. code-block:: bash - - mixer bundle edit bundle1 - -If `bundle1` is found in your local bundles, mixer edits this bundle -definition file. If instead `bundle1` is only found upstream, mixer copies -the bundle definition file from upstream into your :file:`local-bundles` -directory first. - -In both cases, mixer launches your default editor to edit the file. When the -editor closes, mixer automatically validates the edited bundle file and -reports any errors found. If mixer finds an error, you can edit the file -as-is, revert and edit, or skip and move on to the next bundle. If you skip a -file, mixer saves a backup of the original file with the ``.orig`` suffix. -Because mixer always checks your local bundles first, edited copies of an -upstream bundle always take precedence over their upstream counterpart. You -can edit multiple bundles with the following command: - -.. code-block:: bash - - mixer bundle edit bundle1 bundle2 [bundle3 ...] - -Create bundles for the mix --------------------------- - -To create a totally **new bundle**, the bundle name you specify cannot exist -upstream. If that is the case, create a :file:`new-bundle` with the following -command: - -.. code-block:: bash - - mixer bundle edit new-bundle - -This command generates a blank template in :file:`local-bundles` with the -:file:`new-bundle` filename. Mixer launches the editor for you to fill out -the bundle and performs validation when you exit the editor. Add your package -or packages to the bundle definition file to define the packages to install -as part of the bundle. - -.. note:: - - The :command:`mixer bundle edit` command accepts multiple bundles at once. - Thus, you can create multiple new bundles in a single command, for - example: - - .. code-block:: bash - - mixer bundle edit new-bundle1 new-bundle2 [new-bundle3 ...] - -Add bundles to the mix ----------------------- - -Add `bundle1` to your mix with the following command: - -.. code-block:: bash - - mixer bundle add bundle1 - -This command adds the specified bundles to the mix bundles list stored in -your :file:`mixbundles` file. For each bundle you add, mixer checks your -local and upstream bundles to ensure the added bundles actually exist. If -mixer cannot find the bundle, it reports back an error. Additionally, when -mixer adds a bundle, it tells you whether the bundle is local or upstream. -Alternatively, you can learn this information with the -:command:`mixer bundle list` command. Refer to `List the bundles in the mix`_. - -To add multiple bundles at once, use the following command: - -.. code-block:: bash - - mixer bundle add bundle1 bundle2 [bundle3 ...] - -Remove bundles from the mix ---------------------------- - -Remove `bundle1` from your mix with the following command: - -.. code-block:: bash - - mixer bundle remove bundle1 - -This command removes `bundle1` from the mix bundle list stored in your -:file:`mixbundles` file. By default, this command does not remove the bundle -definition file from your local bundles. To completely remove a bundle, -including its local bundle definition file, use the following command with -the *--local* flag: - -.. code-block:: bash - - mixer bundle remove --local bundle1 - -By default, removing a local bundle file with this command removes the bundle -from the mix as well. To only remove the local bundle definition file, use -the following command with the *--mix=false* flag: - -.. code-block:: bash - - mixer bundle remove --local --mix=false bundle1 - -If you remove a local, edited version of an upstream bundle and keep the -bundle in the mix, the mix then references the original upstream version of -the bundle. - -On the other hand, if you remove a bundle that is only found locally but -keep the bundle in the mix bundles list, mixer will not find a valid -bundle definition file and will produce an error. - -Validate the bundles in the mix -------------------------------- - -Mixer performs basic validation on all bundles when used throughout the -system. - -Mixer checks the validity of the bundle's syntax and name. Optionally, you -can run this validation manually on `bundle1` with the following command: - -.. code-block:: bash - - mixer bundle validate bundle1 - -.. note:: This command can be useful in many circumstances. One example is - when importing already-existing local bundles from other projects. - -If you use the optional *--strict* flag, the command additionally -checks if the rest of the bundle header fields can be parsed, if the bundle -header fields are non-empty, and if the bundle header ``Title`` field and -the bundle filename match. Perform a strict validation of `bundle1` with the -following command: - -.. code-block:: bash - - mixer bundle validate --strict bundle1 - -Validate multiple bundles with the following command: - -.. code-block:: bash - - mixer bundle validate bundle1 bundle2 [bundle3 ...] - -Managing bundles with Git -------------------------- - -If you initialized your workspace to be tracked as a git repository -with the :command:`mixer init --git` command, it might be useful to apply a -git commit after you modify the mix bundle list or edit a bundle definition -file. - -All the :command:`mixer bundle` commands in the previous sections support an -optional *--git* flag. This flag automatically applies a git commit -when the command completes, for example: - -.. code-block:: bash - - mixer bundle remove --git bundle1 - -Build the bundle chroots -======================== - -To build all the ``chroots`` based on the defined bundles, use the following -command in your workspace: - -.. code-block:: bash - - mixer build bundles - -If the mix has many bundles, this step might take some time. - -Mixer automatically gathers the bundle definition files for the upstream -bundles into a :file:`upstream-bundles` directory, and user bundles should -be placed directly into :file:`local-bundles`. The local path is set in -the `LOCAL_BUNDLE_DIR` variable in the :file:`builder.conf`. **Do not edit -files in upstream-bundles.** Mixer automatically deletes the contents of -the :file:`upstream-bundles` directory before repopulating the directory -on-the-fly if a new version must be downloaded. - - -Create an update -================ - -Create an update with the following command: - -.. code-block:: bash - - mixer build update - -When the build completes, you can find the mix update content under -:file:`/home/clr/mix/update/www/VER`. In our example, the update content is -found in :file:`/home/clr/mix/update/www/{}`. `` -is the defined mix version, which is 10 by default. - -.. code-block:: bash - - mixer build update - -Mixer creates all the content needed to make a fully usable mix with this -step. However, only *zero packs* are automatically generated. Zero packs are -the content needed to go from nothing to the mix version for which you just -built the content. - -You can create optional *delta packs*, which allow the transition from one -mix version to another, with the following command: - -.. code-block:: bash - - mixer build delta-packs --to --from - -This command generates all delta packs for the bundles changed from -`PAST_VERSION` to `MIX_VERSION`. Mixer cannot create delta packs for the -first build because the update is from version 0. Version 0 implicitly has -no content, thus mixer can generate no deltas. - -For subsequent builds, you can run :file:`mixer-pack-maker.sh` to generate -delta content between them, for example: 10 to 20. - -Create an image -=============== - -Since mixer uses the `ister` tool to create a bootable image from your -updated content, we must first configure the `ister` tool. To configure the -image `ister` creates, we need the `ister` configuration file. Obtain a copy -with the default values from the `ister` package with the following command: - -.. code-block:: bash - - sudo cp /usr/share/defaults/ister/ister.json release-image-config.json - -For reference, you can inspect the `Clear Linux ister configuration file`_ -used for releases. - -Edit the configuration file to include all bundles you want *preinstalled* in -the image. Users can install the bundles in the mix that are not included in -the configuration file with the following command: - -.. code-block:: bash - - sudo swupd bundle-add bundle1 - -Keeping the list of bundles in the configuration file small allows for a -smaller image size. For the minimal base image, the list is: - -.. code-block:: console - - "Bundles": ["os-core", "os-core-update", "kernel-native"] - -Next, set the `Version` field to the mix version that you want the content -mixer to use to build the image. `ister` allows you to build an image from -any mix version that you have built, not just from the current version. In -our example so far, `Version` is set to 10. - -With the `ister` tool configured, build the image with the following command: - -.. code-block:: bash - - sudo mixer build image - -Mixer automatically looks for the :file:`release-image-config.json` file, but -you can freely choose the filename. To use a different name, simply pass the -*--template* flag when creating your image, for example: - -.. code-block:: bash - - sudo mixer build image --template path/to/file.config - -By default, `ister` uses the format version of the build machine it runs on. -Therefore, if the format you are building differs from the format of the |CL| -OS you are building on, you must use the *--format * -flag. Find the current format version of your OS with the following command: - -.. code-block:: bash - - sudo cat /usr/share/defaults/swupd/format - -Update the next mix version information -======================================= - -Increment the mix version number for the next mix with the following command: - -.. code-block:: bash - - mixer versions update - -This command automatically updates the mix version stored in the -:file:`mixversion` file, incrementing it by 10. To increment by a different -amount, use the *--increment* flag, for example: - -.. code-block:: bash - - mixer versions update --increment 10 - -Alternatively, to set the mix version to a specific value, use the -*--mix-version* flag, for example: - -.. code-block:: bash - - mixer versions update --mix-version 20 - -The :command:`mixer versions update` command does not allow you to set the -mix version to a value less than its current value. The mix version is -expected to always increase, even if the new mix is undoing an earlier -change. - -If you have been tracking your workspace with git, you can restore the mix to -an earlier state. However, be careful when "rewriting history" if you have -published the mix content to users already. - -Use the following command with the the *--upstream-version* flag to -update the upstream version of |CL| used as a base for the mix: - -.. code-block:: bash - - mixer versions update --upstream-version 21070 - -This command also accepts the keyword "latest": - -.. code-block:: bash - - mixer versions update --upstream-version latest - -This command sets the upstream version to the latest released version of -upstream |CL| within the same format version. The -:command:`mixer versions update` command does not allow you to set an -upstream version to a value that crosses an upstream format boundary. -Such values require a "format bump" build, which is currently a -manual process. Refer to :ref:`mixer-format` for more information. - -Optionally, you can learn which mix version or upstream version you are -currently using with the following command: - -.. code-block:: bash - - mixer versions - -At this point, you can continue to iterate through the workflows and make -modifications as needed, for example: - -#. Add, remove, or modify bundles. -#. Build the chroots with: - - .. code-block:: bash - - mixer build chroots - -#. Build and update with: - - .. code-block:: bash - - mixer build update - -#. Optionally, you can create delta packs with: - - .. code-block:: bash - - mixer build delta-packs --to --from - -.. _mixer-format: - -Format version -============== - -The `Format` variable set in the :file:`builder.conf` file can be more -precisely referred to as an OS *compatibility epoch*. Versions of the OS -within a given epoch are fully compatible and can update to any other -version within that epoch. Across the `Format` boundary, the OS has changed -in such a way that updating from build M in format X, to build N in format Y -will not work. Generally, this scenario occurs when the software updater or -software manifests change in a way that they are no longer compatible with -the previous update scheme. - -Using a format increment, we insure pre- and co-requisite changes flow out -with proper ordering. The updated client only ever updates to the latest -release in its respective format version, unless overridden by command line -flags. Thus, we can guarantee that all clients update to the final version -in their given format. The given format *must* contain all the changes -needed to understand the content built in the subsequent format. Only after -reaching the final release in the old format can a client continue to update -to releases in the new format. - -When creating a custom mix, the format version should start at "1" or some -known number such as the host system format. The format version should -increment only when a compatibility breakage is introduced. Normal updates, -like updating a software package for example, do not require a format -increment. - -.. _update page: https://cdn.download.clearlinux.org/update/ - -.. _format bumps wiki: https://github.com/clearlinux/swupd-server/wiki/Format-Bumps - -.. _build RPMs instructions: https://github.com/clearlinux/common#build-rpms-for-a-package - -.. _Clear Linux ister configuration file: - https://raw.githubusercontent.com/bryteise/ister/master/release-image-config.json diff --git a/source/clear-linux/guides/maintenance/swupd-guide.rst b/source/clear-linux/guides/maintenance/swupd-guide.rst deleted file mode 100644 index 686c7a1f..00000000 --- a/source/clear-linux/guides/maintenance/swupd-guide.rst +++ /dev/null @@ -1,185 +0,0 @@ -.. _swupd-guide: - -Use swupd -######### - -*swupd* manages the software update capability of |CL-ATTR|. It can check for -valid system updates and, if found, download and install them. It can also -perform verification of the system software. - -|CL| uses :ref:`bundles ` as the base abstraction for -installing functionality on top of the core operating system. Use the `swupd` -tool to install and remove bundles. - -This guide assumes: - -* The device is on a well-connected network. - -* The device is able to connect to the release infrastructure - http://update.clearlinux.org - -.. contents:: - :local: - :depth: 2 - -.. _man_swupd: - -For a full listing of commands and options please see the man page found in -the `swupd` `source code`_ or on |CL| itself. - -.. code-block:: bash - - man swupd - -OS update and verification -************************** - -|CL| is designed to promote a regular update cadence. `swupd` helps to -make sure that process is simple and secure. - -View OS info -============ - -Current OS version and update server info: - -.. code-block:: bash - - swupd info - -.. code-block:: console - - Installed version: 23330 - Version URL: https://cdn.download.clearlinux.org/update/ - Content URL: https://cdn.download.clearlinux.org/update/ - -Enable or disable automatic updates -=================================== - -|CL| updates are automatic by default but can be set to occur only -on demand. To verify your current auto-update setting: - -.. code-block:: bash - - sudo swupd autoupdate - -Disable automatic updates: - -.. code-block:: bash - - sudo swupd autoupdate --disable - -To re-enable automatic: - -.. code-block:: bash - - sudo swupd autoupdate --enable - -Check for updates -================= - -.. code-block:: bash - - sudo swupd check-update - -Perform a manual update -======================= - -You can update to a specific version or accept the latest as the default with -no arguments. Initiate a manual update: - -.. code-block:: bash - - sudo swupd update -m 23330 - -.. note:: - - After a kernel update, reboot immediately for the enhancements to - take effect. - -System Software Verification -============================ - -`swupd` can determine whether system directories and files have been added -to, overwritten, removed, or modified (e.g., permissions). - -.. code-block:: bash - - sudo swupd verify - -All directories that are watched by `swupd` are verified according to -the manifest data and hash mismatches are flagged as follows: - -.. code-block:: console - - Verifying version 23300 - Verifying files - ...0% - Hash mismatch for file: /usr/bin/chardetect - ... - ... - Hash mismatch for file: /usr/lib/python3.6/site-packages/urllib3/util/wait.py - ...100% - Inspected 237180 files - 423 files did not match - Verify successful - -In this case, python packages that were installed on top of the default -install were flagged as mismatched. `swupd` can be directed to ignore -or fix issues based on :ref:`command line options `. - -Fixing hash mismatches -====================== - -`swupd` can correct any issues it detects. Additional directives can be -added including a white list of directories that will be ignored, if -desired. - -The following command will repair issues, remove unknown items, and -ignore files or directories matching `/usr/lib/python`: - -.. code-block:: bash - - sudo swupd verify --fix --picky --picky-whitelist=/usr/lib/python - -Bundles -******* - -Listing installed bundles -========================= - -You can list all of the bundles currently installed on the system - -.. code-block:: bash - - sudo swupd bundle-list --all - -Finding a bundle containing a binary -==================================== - -Run the following to display a list of bundles that contain a particular -binary. Note that it may be present in multiple bundles: - -.. code-block:: bash - - swupd search -b - -Adding a bundle -=============== - -Start by selecting a bundle from the list of :ref:`bundles`. In -this example we're adding dev-utils-dev, which is useful for development. - -.. code-block:: bash - - sudo swupd bundle-add dev-utils-dev - -Removing a bundle -================= - -Dependencies common to other bundles will not be removed - -.. code-block:: bash - - sudo swupd bundle-remove dev-utils-dev - -.. _source code: https://github.com/clearlinux/swupd-client/blob/master/docs/swupd.1.rst \ No newline at end of file diff --git a/source/clear-linux/guides/maintenance/swupd-search.rst b/source/clear-linux/guides/maintenance/swupd-search.rst deleted file mode 100644 index 45764a65..00000000 --- a/source/clear-linux/guides/maintenance/swupd-search.rst +++ /dev/null @@ -1,140 +0,0 @@ -.. _swupd-search: - -Use swupd search to find bundles -################################ - -This document explains how to use `swupd search` to find and add -bundles in |CL-ATTR|. - -Assumptions -*********** - -This guide assumes you: - -* Possess a basic knowledge of :ref:`swupd ` -* Understand :ref:`how swupd differs ` from - other Linux\* distributions -* May :ref:`mixer` to produce a custom distribution/image - -How do I search for a bundle? -***************************** - -Use `swupd search` to locate the bundle where the application binary exists. - -Example: Kata\* Containers -========================== - -Containers have revolutionized the way we manage cloud infrastructure. -Traditional containers often share the same OS kernel, which raises -security concerns. Instead, with Kata Containers, each container has its own -kernel instance and runs on its own :abbr:`Virtual Machine (VM)`. Whether you're running 3 or 300 nodes on your cluster, Kata Containers provide a -lightweight, fast, and secure option for app/container management. - -In |CL|, you only need to add `one bundle`_ to use `Kata Containers`_: -`containers-virt`. Also, check out our tutorial: :ref:`kata`. - -We need to find *kata* containers in a bundle. How do we search for it? - -#. Enter :command:`swupd search`, followed by 'kata' as the search term: - - .. code-block:: bash - - sudo swupd search kata - - .. note:: - - `swupd search` downloads |CL| manifest data and searches for - matching paths. Enter only one term, or hyphenated term, per - search. Use the command :command:`man swupd` to learn more. - -#. Alternatively, if you want to search binaries only, add the `-b` - flag: - - .. code-block:: bash - - sudo swupd search -b kata - - .. note:: - - `-b` flag, or `--binary`, means: Restrict search to program binary paths. Omit this flag if you want a larger scope of search results. - - Only the base bundle is returned. In |CL|, *bundles* can contain - other *bundles* via `includes`. For more details, see `Bundle Definition Files`_ and its subdirectory *bundles*. - - If your search does not produce results on a specific term when using - the `-b` flag, abbreviate the search term. For example, if you search - for *kubernetes* and it does not show results, instead abbreviate the - term to *kube* to show results. - -#. Optionally, you can review our `bundles`_ or individual `packages`_ - -#. Using `sudo swupd search -b kata` shows a match for our use case. - - .. code-block:: console - - Bundle containers-virt (834 MB to install) - /usr/bin/kata-virtfs-lite-proxy-helper - /usr/bin/kata-runtime - /usr/bin/kata-qemu-lite-system-x86_64 - /usr/bin/kata-qemu-lite-pr-helper - /usr/bin/kata-qemu-lite-ga - /usr/bin/kata-collect-data.sh - - .. note:: - - If the bundle is already installed, *[installed]* appears in search results. If this doesn't apppear, the bundle needs to be installed. - -#. Add the bundle `containers-virt`: - - .. code-block:: bash - - sudo swupd bundle-add containers-virt - - .. note:: - - To add multiple bundles simply add a space followed by the bundle name. - -#. When prompted, enter your password. - -#. Upon successful installation, your console should show similar data: - - .. code-block:: console - - Downloading packs... - - Extracting containers-virt pack for version 24430 - ...50% - Extracting kernel-container pack for version 24430 - ...100% - Starting download of remaining update content. This may take a while... - ...100% - Finishing download of update content... - Installing bundle(s) files... - ...100% - Calling post-update helper scripts. - Successfully installed 1 bundle - -Related Topics -============== - -Learn how to: - -* :ref:`kata` - -* :ref:`kata_migration` - -* :ref:`swupd-guide` - -* :ref:`Show all available bundles ` - -* :ref:`Remove bundles` - -.. _Kata Containers: https://clearlinux.org/blogs/clear-linux-os-announces-support-kata-containers - -.. _one bundle: https://github.com/clearlinux/clr-bundles/blob/master/bundles/containers-virt - -.. _Bundle Definition Files: https://github.com/clearlinux/clr-bundles - - .. _bundles: https://github.com/clearlinux/clr-bundles/tree/master/bundles - - .. _packages: https://github.com/clearlinux/clr-bundles/blob/master/packages diff --git a/source/substitutions.txt b/source/substitutions.txt index 6790a2c3..20c6b780 100644 --- a/source/substitutions.txt +++ b/source/substitutions.txt @@ -9,3 +9,8 @@ .. |CAUTION-BACKUP-USB| replace:: Burning an image formats the USB drive, and will destroy all pre-existing content. Back up your data before proceeding. + +.. |CAUTION-UNMOUNT-USB-PARTITIONS| replace:: + Not fully unmounting the USB drive before burning an image could cause + file system checksum errors in it. If this happens, burn the image again + ensuring all the USB drive partitions are unmounted first.