From 99f9289a67fdd65016f7d0d07cb61751819c8202 Mon Sep 17 00:00:00 2001 From: Kevin Putnam Date: Wed, 27 Mar 2019 20:42:55 -0700 Subject: [PATCH 1/4] Update to the swupd Guide (#409) * partial commit to save work * latest changes * replaced out of date swupd-guide.rst with contents of swupd-temp.rst; removed swupd-temp.rst * changed page ref to swupd-guide from swupd; too many existing dependencies * Added descriptions of manifests and delta packs from How to Clear * fixed typo * changed header organization slightly and removed unnecessary text formatting * updated a couple headers to make more sense * numerous cosmetic and readability changes based on full render review * changes to description section based on updated swupd concept page * updates based on feedback * updates based on technical review * removed swupd-search.rst; updated maintenance.rst toc tree after removal; updated references in developer-workstation.rst and kernel-module.rst that linked to swupd-search.rst --- .../maintenance/developer-workstation.rst | 5 +- .../guides/maintenance/kernel-modules.rst | 2 +- .../guides/maintenance/maintenance.rst | 1 - .../guides/maintenance/swupd-guide.rst | 368 +++++++++++++----- .../guides/maintenance/swupd-search.rst | 140 ------- 5 files changed, 273 insertions(+), 243 deletions(-) delete mode 100644 source/clear-linux/guides/maintenance/swupd-search.rst diff --git a/source/clear-linux/guides/maintenance/developer-workstation.rst b/source/clear-linux/guides/maintenance/developer-workstation.rst index fc0582d9..765c44c3 100644 --- a/source/clear-linux/guides/maintenance/developer-workstation.rst +++ b/source/clear-linux/guides/maintenance/developer-workstation.rst @@ -78,10 +78,11 @@ While your role may not neatly fit in one of these categories, consider using Ta `swupd` search ************** -We recommend trying out :ref:`swupd search `, to learn the +We recommend learning about :ref:`swupd `, to learn the commands to search for and add bundles relevant to your project. -:ref:`swupd-search` shows you how to: +The guide provides an :ref:`example ` +that shows you how to: * Use `swupd` to search for bundles * Use `swupd` to add bundles diff --git a/source/clear-linux/guides/maintenance/kernel-modules.rst b/source/clear-linux/guides/maintenance/kernel-modules.rst index 2339453a..94daac09 100644 --- a/source/clear-linux/guides/maintenance/kernel-modules.rst +++ b/source/clear-linux/guides/maintenance/kernel-modules.rst @@ -45,7 +45,7 @@ security practices, see the :ref:`security` page. You can search for kernel module file names, which end with the :file:`.ko` file extension, using the :command:`swupd search` command. For example: -:command:`sudo swupd search ${module_name}.ko`. See :ref:`swupd-search` for +:command:`sudo swupd search ${module_name}.ko`. See :ref:`swupd-guide` for more information. diff --git a/source/clear-linux/guides/maintenance/maintenance.rst b/source/clear-linux/guides/maintenance/maintenance.rst index 9e93ce9a..9e8f7785 100644 --- a/source/clear-linux/guides/maintenance/maintenance.rst +++ b/source/clear-linux/guides/maintenance/maintenance.rst @@ -12,7 +12,6 @@ completed. autoproxy developer-workstation - swupd-search enable-user-space swupd-guide bulk-provision diff --git a/source/clear-linux/guides/maintenance/swupd-guide.rst b/source/clear-linux/guides/maintenance/swupd-guide.rst index 686c7a1f..adbe6a1b 100644 --- a/source/clear-linux/guides/maintenance/swupd-guide.rst +++ b/source/clear-linux/guides/maintenance/swupd-guide.rst @@ -1,113 +1,285 @@ .. _swupd-guide: -Use swupd -######### +swupd +##### -*swupd* manages the software update capability of |CL-ATTR|. It can check for -valid system updates and, if found, download and install them. It can also -perform verification of the system software. - -|CL| uses :ref:`bundles ` as the base abstraction for -installing functionality on top of the core operating system. Use the `swupd` -tool to install and remove bundles. - -This guide assumes: - -* The device is on a well-connected network. - -* The device is able to connect to the release infrastructure - http://update.clearlinux.org +:command:`swupd` links a |CL-ATTR| installation with upstream updates and +software. .. contents:: :local: :depth: 2 -.. _man_swupd: +Description +*********** -For a full listing of commands and options please see the man page found in -the `swupd` `source code`_ or on |CL| itself. +:command:`swupd` has two main functions: -.. code-block:: bash +#. It manages software replacing APT or YUM, installing bundles + rather than packages. +#. It checks for system updates and installs them. - man swupd +:ref:`Bundles ` are the smallest granularity component that is +managed by |CL| and contain everything needed to deliver a software +capability. Rather than downloading a cascade of package dependencies when +installing a piece of software, a bundle comes with all of its dependencies. +:command:`swupd` manages overlapping dependencies behind the scenes ensuring +that all software is compatible across the system. -OS update and verification -************************** +Versioning +========== -|CL| is designed to promote a regular update cadence. `swupd` helps to -make sure that process is simple and secure. +In a traditional distribution, the process of describing current software +versioning usually involves: -View OS info -============ +- Listing and keeping track of the current OS release (generally + uninformative about any singular packages or functionality). -Current OS version and update server info: +- Keeping track of packages and repositories being used, and updating them + individually. -.. code-block:: bash +- Listing and tracking every package available and installed on the + system, none of which are directly tied to the current OS release. - swupd info +This can be done effectively, but given the nearly endless combinations of +packages and versions of packages a server may have, it quickly becomes +non-trivial to define what "version" the system is and what software it +is running without explicitly going through each system and inspecting +every package. -.. code-block:: console +With |CL|, we need track: - Installed version: 23330 - Version URL: https://cdn.download.clearlinux.org/update/ - Content URL: https://cdn.download.clearlinux.org/update/ +- One number -Enable or disable automatic updates -=================================== +A number representing the **current** release of the OS is sufficient to +describe the versions of all the software on the OS. Each build is +composed of a specific set of bundles made from a particular version of +packages. This matters on a daily basis to system administrators, who +need to determine which of their systems do not have the latest security +fixes, or which combinations of software have been tested. Every release +of the same number is guaranteed to contain the same versions of software, +so there's no ambiguity between two systems running the same version of |CL|. + +Updating +======== + +|CL| enforces regular updating of the OS by default and will automatically +check for updates against a version server. The content server provides the +file and metadata content for all versions and can be the same as the +version server. The content url server provides metadata in the form of +manifests. These Manifest files list and describe file contents, symlinks, +and directories. Additionally, the actual content is +provided to clients in the form of archive files. + +Software updates with |CL| are also efficient. Unlike package-based +distributions, :command:`swupd` only updates files that have changed rather +than entire packages. For example, it is quite common for an OS security +patch to be as small as 15 KB. Using binary deltas, the |CL| is able to +apply only what is needed. + +To get a more detailed understanding of how to generate update content for +|CL| see the :ref:`mixer ` tool. + +How it works +************ + +Prerequisites +============= + +* The device is on a well-connected network. +* The device is able to connect to an update server. The default server is: + http://update.clearlinux.org + +Updates +======= + +|CL| updates are automatic by default but can be set to occur only on +demand. :command:`swupd` makes sure that regular updates are simple and +secure. It can also check the validity of currently installed files and +software and correct any problems. + +Manifests +--------- + +The Clear Linux OS software update content consists of data and +metadata. The data is the files that end up in the OS. The metadata +contains relevant information to properly provision the data to the OS +file system, as well as update the system and add or remove additional +content to the OS. + +The Manifests are mostly long lists of hashes that describe content. +Each bundle gets its own manifest file. There is a master manifest +file that describes all manifests to tie it all together. + +Fullfiles, packs, and delta packs +--------------------------------- + +The data that an update provisions to a system can be obtained in +three different ways. There are three different methods, and they +exist to optimize the delivery of content and speed up updates. + +Fullfiles are always generated for every file in every release. This +allows any Clear Linux OS to obtain the exact copy of the content +for each version directly. This would be used if the OS verification +needed to replace a single file, for instance. + +Packs are available for some releases and combine many files to speed +up the creation of installation media and large updates. Delta packs +are an optimized version of packs that only contain updates (binary +diffs) and cannot be used without having the original file content. + +Bundle Search +============= + +:command:`swupd` search downloads manifest data and searches for +bundles that match the term. Enter only one term, or hyphenated term, per +search. Use the command :command:`man swupd` to learn more. + +Only the base bundle is returned. Bundles can contain other bundles via +includes. For more details, see `Bundle Definition Files`_ and its +subdirectory bundles. + +Bundles that are already installed, will be marked [installed] in search +results. + +Optionally, you can review our `bundles`_ on GitHub. + +Examples +******** + +Example 1: Disable and Enable automatic updates +=============================================== |CL| updates are automatic by default but can be set to occur only -on demand. To verify your current auto-update setting: +on demand. -.. code-block:: bash +#. First verify your current auto-update setting. - sudo swupd autoupdate + .. code-block:: bash -Disable automatic updates: + sudo swupd autoupdate -.. code-block:: bash + .. code-block:: console - sudo swupd autoupdate --disable + Enabled -To re-enable automatic: +#. Disable automatic updates. -.. code-block:: bash + .. code-block:: bash - sudo swupd autoupdate --enable + sudo swupd autoupdate --disable -Check for updates -================= + .. code-block:: console -.. code-block:: bash + Warning: disabling automatic updates may take you out of compliance with your IT policy - sudo swupd check-update + Running systemctl to disable updates + Created symlink /etc/systemd/system/swupd-update.service → /dev/null. + Created symlink /etc/systemd/system/swupd-update.timer → /dev/null. -Perform a manual update -======================= +#. Check manually for updates. -You can update to a specific version or accept the latest as the default with -no arguments. Initiate a manual update: + .. code-block:: bash -.. code-block:: bash + sudo swupd check-update - sudo swupd update -m 23330 +#. Install an update after identifying one that you need. -.. note:: + .. code-block:: bash - After a kernel update, reboot immediately for the enhancements to - take effect. + sudo swupd update -m -System Software Verification -============================ +#. Re-enable automatic installs. -`swupd` can determine whether system directories and files have been added -to, overwritten, removed, or modified (e.g., permissions). + .. code-block:: bash + + sudo swupd autoupdate --enable + +.. _swupd-guide-example-install-bundle: + +Example 2: Find and install Kata\* Containers +============================================= + +Kata Containers is a popular container implementation. Unlike other +container implementations, each Kata Container has its own +kernel instance and runs on its own :abbr:`Virtual Machine (VM)` for +improved security. + +|CL| makes it very easy to install, since you only need to add +`one bundle`_ to use `Kata Containers`_: `containers-virt`, despite a +number of dependencies. Also, check out our tutorial: :ref:`kata`. + +#. Find the right bundle. + + * To return all possible matches for the search string enter + :command:`swupd search`, followed by 'kata': + + .. code-block:: bash + + sudo swupd search kata + + The output should be similar to: + + .. code-block:: console + + Bundle with the best search result: + + containers-virt - Run container applications from Dockerhub in lightweight virtual machines + + This bundle can be installed with: + + swupd bundle-add containers-virt + + Alternative bundle options are + + cloud-native-basic - Contains ClearLinux native software for Cloud + + .. note:: + + If your search does not produce results with a specific + term, shorten the search term. For example, use *kube* instead of + *kubernetes*. + +#. Add the bundle. + + .. code-block:: bash + + sudo swupd bundle-add containers-virt + + .. note:: + + To add multiple bundles simply add a space followed by the bundle name. + + The output of a successful installation should be similar to: + + .. code-block:: console + + Downloading packs... + + Extracting containers-virt pack for version 24430 + ...50% + Extracting kernel-container pack for version 24430 + ...100% + Starting download of remaining update content. This may take a while... + ...100% + Finishing download of update content... + Installing bundle(s) files... + ...100% + Calling post-update helper scripts. + Successfully installed 1 bundle + +Example 3: Verify and correct sytem file mismatch +================================================= + +:command:`swupd` can determine whether system directories and files have +been added to, overwritten, removed, or modified (e.g., permissions). .. code-block:: bash sudo swupd verify -All directories that are watched by `swupd` are verified according to -the manifest data and hash mismatches are flagged as follows: +All directories that are watched by :command:`swupd` are verified according +to the manifest data and hash mismatches are flagged as follows: .. code-block:: console @@ -124,15 +296,11 @@ the manifest data and hash mismatches are flagged as follows: Verify successful In this case, python packages that were installed on top of the default -install were flagged as mismatched. `swupd` can be directed to ignore -or fix issues based on :ref:`command line options `. +install were flagged as mismatched. :command:`swupd` can be directed to +ignore or fix issues based on command line options. -Fixing hash mismatches -====================== - -`swupd` can correct any issues it detects. Additional directives can be -added including a white list of directories that will be ignored, if -desired. +:command:`swupd` can correct any issues it detects. Additional directives +can be added including a white list of directories that will be ignored. The following command will repair issues, remove unknown items, and ignore files or directories matching `/usr/lib/python`: @@ -141,45 +309,47 @@ ignore files or directories matching `/usr/lib/python`: sudo swupd verify --fix --picky --picky-whitelist=/usr/lib/python -Bundles -******* +Quick Reference +*************** -Listing installed bundles -========================= +swupd info + To see the currently installed version and update servers. -You can list all of the bundles currently installed on the system +swupd update + To update to a specific version or with no arguments to update to latest. -.. code-block:: bash +swupd bundle-list [--all] + To list installed bundles. - sudo swupd bundle-list --all +swupd bundle-add [-b] + To find a bundle that contains your search term. -Finding a bundle containing a binary -==================================== +swupd bundle-add + To add a bundle. -Run the following to display a list of bundles that contain a particular -binary. Note that it may be present in multiple bundles: +swupd bundle-remove + To remove a bundle. -.. code-block:: bash +swupd --help + For additional :command:`swupd` commands. - swupd search -b +man swupd + To reference the :command:`swupd` man page, or see the + `source documentation`_ available on github. -Adding a bundle -=============== +Related topics +************** -Start by selecting a bundle from the list of :ref:`bundles`. In -this example we're adding dev-utils-dev, which is useful for development. +* :ref:`autospec` +* :ref:`mixer` +* :ref:`bundles` -.. code-block:: bash +.. _source documentation: https://github.com/clearlinux/swupd-client/blob/master/docs/swupd.1.rst - sudo swupd bundle-add dev-utils-dev +.. _Kata Containers: https://clearlinux.org/containers -Removing a bundle -================= +.. _one bundle: https://github.com/clearlinux/clr-bundles/blob/master/bundles/containers-virt -Dependencies common to other bundles will not be removed +.. _Bundle Definition Files: https://github.com/clearlinux/clr-bundles -.. code-block:: bash - - sudo swupd bundle-remove dev-utils-dev - -.. _source code: https://github.com/clearlinux/swupd-client/blob/master/docs/swupd.1.rst \ No newline at end of file +.. _bundles: https://github.com/clearlinux/clr-bundles/tree/master/bundles \ No newline at end of file diff --git a/source/clear-linux/guides/maintenance/swupd-search.rst b/source/clear-linux/guides/maintenance/swupd-search.rst deleted file mode 100644 index 45764a65..00000000 --- a/source/clear-linux/guides/maintenance/swupd-search.rst +++ /dev/null @@ -1,140 +0,0 @@ -.. _swupd-search: - -Use swupd search to find bundles -################################ - -This document explains how to use `swupd search` to find and add -bundles in |CL-ATTR|. - -Assumptions -*********** - -This guide assumes you: - -* Possess a basic knowledge of :ref:`swupd ` -* Understand :ref:`how swupd differs ` from - other Linux\* distributions -* May :ref:`mixer` to produce a custom distribution/image - -How do I search for a bundle? -***************************** - -Use `swupd search` to locate the bundle where the application binary exists. - -Example: Kata\* Containers -========================== - -Containers have revolutionized the way we manage cloud infrastructure. -Traditional containers often share the same OS kernel, which raises -security concerns. Instead, with Kata Containers, each container has its own -kernel instance and runs on its own :abbr:`Virtual Machine (VM)`. Whether you're running 3 or 300 nodes on your cluster, Kata Containers provide a -lightweight, fast, and secure option for app/container management. - -In |CL|, you only need to add `one bundle`_ to use `Kata Containers`_: -`containers-virt`. Also, check out our tutorial: :ref:`kata`. - -We need to find *kata* containers in a bundle. How do we search for it? - -#. Enter :command:`swupd search`, followed by 'kata' as the search term: - - .. code-block:: bash - - sudo swupd search kata - - .. note:: - - `swupd search` downloads |CL| manifest data and searches for - matching paths. Enter only one term, or hyphenated term, per - search. Use the command :command:`man swupd` to learn more. - -#. Alternatively, if you want to search binaries only, add the `-b` - flag: - - .. code-block:: bash - - sudo swupd search -b kata - - .. note:: - - `-b` flag, or `--binary`, means: Restrict search to program binary paths. Omit this flag if you want a larger scope of search results. - - Only the base bundle is returned. In |CL|, *bundles* can contain - other *bundles* via `includes`. For more details, see `Bundle Definition Files`_ and its subdirectory *bundles*. - - If your search does not produce results on a specific term when using - the `-b` flag, abbreviate the search term. For example, if you search - for *kubernetes* and it does not show results, instead abbreviate the - term to *kube* to show results. - -#. Optionally, you can review our `bundles`_ or individual `packages`_ - -#. Using `sudo swupd search -b kata` shows a match for our use case. - - .. code-block:: console - - Bundle containers-virt (834 MB to install) - /usr/bin/kata-virtfs-lite-proxy-helper - /usr/bin/kata-runtime - /usr/bin/kata-qemu-lite-system-x86_64 - /usr/bin/kata-qemu-lite-pr-helper - /usr/bin/kata-qemu-lite-ga - /usr/bin/kata-collect-data.sh - - .. note:: - - If the bundle is already installed, *[installed]* appears in search results. If this doesn't apppear, the bundle needs to be installed. - -#. Add the bundle `containers-virt`: - - .. code-block:: bash - - sudo swupd bundle-add containers-virt - - .. note:: - - To add multiple bundles simply add a space followed by the bundle name. - -#. When prompted, enter your password. - -#. Upon successful installation, your console should show similar data: - - .. code-block:: console - - Downloading packs... - - Extracting containers-virt pack for version 24430 - ...50% - Extracting kernel-container pack for version 24430 - ...100% - Starting download of remaining update content. This may take a while... - ...100% - Finishing download of update content... - Installing bundle(s) files... - ...100% - Calling post-update helper scripts. - Successfully installed 1 bundle - -Related Topics -============== - -Learn how to: - -* :ref:`kata` - -* :ref:`kata_migration` - -* :ref:`swupd-guide` - -* :ref:`Show all available bundles ` - -* :ref:`Remove bundles` - -.. _Kata Containers: https://clearlinux.org/blogs/clear-linux-os-announces-support-kata-containers - -.. _one bundle: https://github.com/clearlinux/clr-bundles/blob/master/bundles/containers-virt - -.. _Bundle Definition Files: https://github.com/clearlinux/clr-bundles - - .. _bundles: https://github.com/clearlinux/clr-bundles/tree/master/bundles - - .. _packages: https://github.com/clearlinux/clr-bundles/blob/master/packages From 73e154d0727d4051d20b91a215467ddba49ab4f0 Mon Sep 17 00:00:00 2001 From: Jair Gonzalez Date: Thu, 28 Mar 2019 19:45:16 -0600 Subject: [PATCH 2/4] Bootable USB (#436) * bootable-usb: Add extra caution about unmount Not unmounting an USB partition before an image is burned could cause file system checksum errors leading to other issues (including kernel panics) when trying to boot from the USB drive. This commit adds an extra caution note to warn the users about the possible issues they could find when this step is not followed correctly. Signed-off-by: Jair Gonzalez * bootable-usb: Add o_sync to the dd command Update the example 'dd' command to burn an image, by adding the 'oflag=sync' option. This allows to reduce the flashing time in conjunction with the recommended 'bs=4M', enables a more accurate progress indicator, and allows to remove the additional sync command, while preserving some cache optimizations. Signed-off-by: Jair Gonzalez --- .../clear-linux/get-started/bootable-usb/bootable-usb.rst | 8 ++++++-- source/substitutions.txt | 5 +++++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/source/clear-linux/get-started/bootable-usb/bootable-usb.rst b/source/clear-linux/get-started/bootable-usb/bootable-usb.rst index 3994b20f..e0ac2a2a 100644 --- a/source/clear-linux/get-started/bootable-usb/bootable-usb.rst +++ b/source/clear-linux/get-started/bootable-usb/bootable-usb.rst @@ -78,7 +78,7 @@ Burn the |CL| image onto a USB drive ├─sda8 8:8 0 30G 0 part / └─sda6 8:6 0 7.9G 0 part [SWAP] -#. You must unmount a USB drive before you can burn an image onto it. Note that +#. You must unmount a USB drive before burning an image onto it. Note that some Linux distros automatically mount a USB drive when it is plugged in. Unmount a USB drive with the :command:`umount` command followed by the device identifier/partition. For example: @@ -93,7 +93,11 @@ Burn the |CL| image onto a USB drive .. code-block:: bash - dd if=./clear-[version number]-[image type] of= bs=4M status=progress && sync + dd if=./clear-[version number]-[image type] of= oflag=sync bs=4M status=progress + +.. caution:: + + |CAUTION-UNMOUNT-USB-PARTITIONS| .. _bootable-usb-mac: diff --git a/source/substitutions.txt b/source/substitutions.txt index 6790a2c3..20c6b780 100644 --- a/source/substitutions.txt +++ b/source/substitutions.txt @@ -9,3 +9,8 @@ .. |CAUTION-BACKUP-USB| replace:: Burning an image formats the USB drive, and will destroy all pre-existing content. Back up your data before proceeding. + +.. |CAUTION-UNMOUNT-USB-PARTITIONS| replace:: + Not fully unmounting the USB drive before burning an image could cause + file system checksum errors in it. If this happens, burn the image again + ensuring all the USB drive partitions are unmounted first. From a7ce92acdf626eed3edbc588a461434d30bb1838 Mon Sep 17 00:00:00 2001 From: Kristal Dale Date: Tue, 2 Apr 2019 15:20:55 -0700 Subject: [PATCH 3/4] Mixer guide rework (#372) * Initial commit of rework - Restructure organization of content to be task and background oriented - Include references to available man pages Signed-off-by: Kristal Dale * - Updated bundle section: updates in tooling, simplified and added ref to man page Signed-off-by: Kristal Dale * Tested all examples except Docker proxy info. Expanded examples and included setup of VM to test image and update content. Signed-off-by: Kristal Dale * Comment removal Signed-off-by: Kristal Dale * Clean up comments and TODOs in preparation for initial review. Signed-off-by: Kristal Dale * Updates based on @intelkevinputnam feedback Signed-off-by: Kristal Dale * Update bundle section based on new info from engineering. Signed-off-by: Kristal Dale * Correct capitalization of mixer name. Signed-off-by: Kristal Dale * Edits per feedback from @bktan8 Signed-off-by: Kristal Dale * Update prereq formatting and numbered list formatting (remove bold). Signed-off-by: Kristal Dale * Minor edit to section heading for standardization Signed-off-by: Kristal Dale * Minor edits around prerequirements formatting to bring inline with template. Signed-off-by: Kristal Dale * Revise example intro to be more concise. Signed-off-by: Kristal Dale * Minor revisions to standardize CL tooling guides Signed-off-by: Kristal Dale * Revisions based on feedback from @rchiossi. Signed-off-by: Kristal Dale * updates based on feedback * slight wording fix * fixed lack of indent after line break which was causing a problem with numbered list; fixed console code block syntax error * fixed final instruction to read 'no longer'; added local bundles and upstream RMPs to bullet list * fixed json proxy urls --- .../clear-linux/guides/maintenance/mixer.rst | 1518 ++++++++--------- 1 file changed, 745 insertions(+), 773 deletions(-) diff --git a/source/clear-linux/guides/maintenance/mixer.rst b/source/clear-linux/guides/maintenance/mixer.rst index f826ab4f..a786a786 100644 --- a/source/clear-linux/guides/maintenance/mixer.rst +++ b/source/clear-linux/guides/maintenance/mixer.rst @@ -1,267 +1,446 @@ .. _mixer: -Use the mixer tool -################## +mixer +##### -*Mixing* refers to composing an operating system for specific use cases. -While the default |CL-ATTR| provides options to install bundles for various -server capabilities, some developers may wish to either augment the -operating system itself with functionality from their own packages or modify -the structure of current bundles to cater to their particular needs. +mixer is the tool used by the |CL-ATTR| team to generate official update content +and releases. The update content generated by mixer is then consumed by swupd on +a downstream client. The same mixer tool is available as part of |CL| to create +your own customized update content and releases. -Prerequisites -************* - -To start working with the mixer tools, you need a recent image of |CL| with -the `mixer` bundle installed. If the bundle is not yet installed, you can -add it with the :command:`swupd bundle-add` command as follows: - -.. code-block:: bash - - sudo swupd bundle-add mixer - -Current mixing workflow -*********************** -Mixer by default runs *all* build commands in a container to ensure the -correct version of the tooling is being used. This also allows custom mixes -to automatically perform downstream format bumps when upstream releases -a format bump. You can still run mixer natively by appending the *--native* flag to -any of the commands. - -.. note:: - You cannot run mixer if you are already in a container, unless you pass - *--native* to the command. Nested containerization is not supported, nor - is building images using the container mode. - -There are two different workflows to create your own mix. -First, if your mix only uses |CL| content, *skip Create custom RPMs* below. -Second, if your mix includes your own -:abbr:`RPMs (RPM Package Manager files)`, follow all steps below. - -.. contents:: +.. contents:: :local: :depth: 1 - :backlinks: top -The following sections contain detailed information on every step of -these workflows. +Description +*********** -.. _create-nginx-web-server: +mixer uses the following sources as inputs to generate update content: -Create nginx web server to host mixer updates -============================================= +* Upstream |CL| bundles with their corresponding RPM packages +* Locally-defined bundles with their corresponding local RPM packages +* Locally-defined bundles with upstream RPM packages -Follow these steps to set up a HTTP service with ``nginx`` web -server, where you can host custom |CL| mixes: +Using the mixer tool, you select which set of content from these sources will be +part of your update. You can select content from each of these sources to make a +unique combination of functionality for your custom update content (known as a +mix). -#. Install ``web-server-basic``. - - .. code-block:: bash +The update content that mixer generates consists of various pieces of OS +content, update metadata, as well as a complete image. The OS content includes +all files in an update, as well as zero- and delta-packs for improved update +performance. The metadata, stored as manifests, describes all of the bundle +information for the update. Update content produced by mixer is then published +to a web server and consumed by clients via swupd. Refer to +:ref:`swupd ` for additional information regarding updates and +update content. - sudo swupd bundle-add web-server-basic +How it works +************ -#. Make the directory where mixer updates will reside. +Learn the mixer tool set up and workflow. + +.. contents:: + :local: + :depth: 1 + +Prerequisites +============= + +* :command:`mixer` bundle + + Add the mixer tool with the :command:`mixer` bundle. Refer to + `Install a bundle`_ for more details. + +* Docker container + + mixer by default runs all build commands in a Docker container to make sure + the correct tool versions are used. This also allows custom mixes to + automatically perform downstream format bumps when the upstream releases a + format bump. See `Format version`_ for additional information regarding + format bumps. + + Refer to `Configure and enable Docker`_ for instruction. + +* Docker proxy (optional) + + If you use a proxy server, you must set your proxy environment variables and + create a proxy configuration file for the Docker daemon and container. + + Consult your IT department if you are behind a corporate proxy for the correct + values. + + Refer to `Configure Docker proxy info`_ for instruction. + +* Location to host the update content and images + + In order for swupd to make use of your mix, the update content for your mix + must be hosted on a web server. Your mix will be configured with an update + location URL, which swupd will use to pull down updates. + + Refer to `Set up a nginx web server for mixer`_ for an simple example of + setting up an update location. + +Mix setup +========== + +Follow these steps to create and initialize the mixer workspace. Complete +setup before you create a mix. + +#. Create workspace. + + The mixer tool uses a simple workspace to contain all input and output in a + basic directory structure. The workspace is simply an empty folder that you + will execute the mixer commands from. Each mix will use its own separate + workspace. + +#. Initialize the workspace and mix. + + Before you create a mix, you must explicitly initialize the mixer workspace. + During initialization, the mixer workspace is configured and the base for + your mix is defined. By default, your mix will be based on the latest + upstream version and start with the minimum set of bundles. Your first custom + mix version number will start at 10. You can alternately select other + versions or bundle sets to start from. + + Initialization creates the directory structure within the workspace and adds + the :file:`builder.conf` file, which is used to configure the mixer tool. + + View the `mixer.init man page`_ for more information on mixer + initialization. + + View the list of `suitable versions`_ to mix from. + +#. Edit builder.conf. + + :file:`builder.conf` tells the mixer tool how to configure the mix. For + example, it allows you to configure where mixer output is located and where + swupd update content will be located. + + At minimum, set the URL of your update server so your custom OS knows where + to get update content. + + Refer to the `builder.conf`_ section for more information. + +Create a mix +============ + +A mix is created with the following steps: + +#. Add custom RPMs and set up local repo (optional). + + If you are adding custom RPMs to your mix, you will need to add the RPMs to + your mix workspace and set up a corresponding local repository. + + Go to the :ref:`autospec` guide to learn to build RPMs from + scratch. If the RPMs are not built on |CL|, make sure your + configuration and toolchain builds them correctly for |CL|. Otherwise there + is no guarantee they will be compatible. + + Refer to the :ref:`autospec` guide for more information on using autospec to + build RPMs. + +#. Update and build bundles. + + Add, edit, or remove bundles that will be part of your content and build + them. mixer will automatically update the :file:`mixbundles` file when you + update the bundles in your mix. + + View the `mixer.bundle man page`_ for more information on configuring bundles + in a mix. + + View the `mixer.build man page`_ for more information on building bundles. + + View the `Bundles`_ section for more information on how mixer manages + bundles. + +#. Create the update content. + + mixer creates update content with this step. Zero-packs are created + automatically, and delta-packs can be optionally created at the same time + (for all builds after version 0). + + A zero-pack is the full set of content needed to go from mix version 0 + (nothing) to the mix version you just built content for. + + A delta-pack provides the content *delta* between a `PAST_VERSION` to a + `MIX_VERSION` which allows the transition from one mix version to another. + + View :ref:`swupd-guide` for more information on update content. + +#. Create image. + + mixer creates a bootable image from your updated content using + the ister tool. In this step you can specify which bundles you want + *preinstalled* in the image. Users can later install other bundles available + in your mix. + +#. Make update available. + + Deploy update content and images to your update server. + + View the `Example 3: Deploy updates to target`_ for a simple deployment + scenario. + +Maintain or modify mix +====================== + +Update or modify your content to a new version by following the same steps to +create a mix. Increment the mix version number for the next mix. + +Examples +******** + +The following examples are designed to work together and in order. The examples +use: + +* A stock installation of |CL|. +* A web server that comes with |CL| to host the content updates. +* A simple VM that will update against the locally produced content created in + Example 2. + +Complete all `Prerequisites`_ before using these examples. + +Example 1: Mix set up +====================== + +This example shows the basic steps for first time setup of mixer for a new mix. + +#. Create an empty directory to use as a workspace for mixer: .. code-block:: bash - sudo mkdir -p /var/www + mkdir ~/mixer -#. Create a symbolic link. +#. In your mixer workspace, generate an initial mix based on the latest upstream + |CL| version, with minimum bundles: .. code-block:: bash - sudo ln -sf /home/clr/mix/update/www /var/www/mixer + cd ~/mixer + mixer init - .. note:: + Note in the initialization output, that your initial mix version is set to + 10 and that the minimum bundles have been added. - `/home/clr/mix` is the workspace for the mix (read `Create workspace`) - - -#. Set up ``nginx`` configuration. - - .. code-block:: bash - - sudo mkdir -p /etc/nginx/conf.d - -#. Copy the default example configuration file. - - .. code-block:: bash - - sudo cp -f /usr/share/nginx/conf/nginx.conf.example /etc/nginx/nginx.conf - -#. Open ``mixer.conf`` with an editor. - - .. code-block:: bash - - sudo nano /etc/nginx/conf.d/mixer.conf - -#. Add the server configuration content below to ``mixer.conf``. +#. Edit :file:`builder.conf` to set the value of CONTENTURL and VERSIONURL to + the IP address of the nginx server you set up in the prerequisite + `Set up a nginx web server for mixer`_. For example: .. code-block:: console - server { - server_name localhost; - location / { - root /var/www/mixer; - autoindex on; - } + CONTENTURL="http://192.168.25.52" + VERSIONURL="http://192.168.25.52" + + + +Example 2: Create a simple mix +============================== + +This example shows how to create a simple custom mix using upstream content. +We'll create an image for a QEMU virtual machine which we can later use to test +our mix. + +We can use the default bundles that were added during intialization, but these +include the :command:`native-kernel` bundle which is intended to be used on a +bare metal system instead of a VM. So we will modify the default bundle +set to get a smaller kernel image, which will also be faster to load. + +#. Update bundles in mix: + + .. code-block:: bash + + mixer bundle remove kernel-native + mixer bundle add kernel-kvm + +#. Build bundles: + + .. code-block:: bash + + mixer build bundles + + Look in ~/mixer/update/image//full for the full chroot after the + :command:`build` command completes. + +#. Build update content. Browse to your http://localhost site and you'll see + the web page is now up, but with no update content. Build the update content: + + .. code-block:: bash + + mixer build update + + Refresh your http://localhost site and now you can see the update content for + mix version 10. + + Look in ~/mixer/update/www/ to see the update content in your + workspace. + +#. Configure image. Edit the ister configuration file for your image to include + all of the bundles you want preinstalled in the image. If this is the first + time creating an image, first get a copy of the + :file:`release-image-config.json` template file: + + .. code-block:: bash + + curl -O https://raw.githubusercontent.com/bryteise/ister/master/release-image-config.json + + For this example, edit :file:`release-image-config.json` so that the root + partition size is "5G" and replace the "kernel-native" bundle with + "kernel-kvm". + + .. code-block:: console + + { + "DestinationType" : "virtual", + "PartitionLayout" : [ { "disk" : "release.img", "partition" : 1, "size" : "32M", "type" : "EFI" }, + { "disk" : "release.img", "partition" : 2, "size" : "16M", "type" : "swap" }, + { "disk" : "release.img", "partition" : 3, "size" : "5G", "type" : "linux" } ], + "FilesystemTypes" : [ { "disk" : "release.img", "partition" : 1, "type" : "vfat" }, + { "disk" : "release.img", "partition" : 2, "type" : "swap" }, + { "disk" : "release.img", "partition" : 3, "type" : "ext4" } ], + "PartitionMountPoints" : [ { "disk" : "release.img", "partition" : 1, "mount" : "/boot" }, + { "disk" : "release.img", "partition" : 3, "mount" : "/" } ], + "Version": "latest", + "Bundles": ["kernel-kvm", "os-core", "os-core-update"] } -#. Restart the daemon, enable nginx on boot, and start the service. +#. Build the image. .. code-block:: bash - sudo systemctl daemon-reload + sudo mixer build image - sudo systemctl enable nginx + The output from this step will be :file:`release.img`, which is a live image. - sudo systemctl start nginx - -#. To verify the web server is running, check it in an Internet browser: +#. Make the next mix. Create a new version of your mix, for the live image to + update to. Increment your mix version by 10: .. code-block:: bash - http://localhost + mixer versions update -#. Alternatively, verify the server is running by entering: + Repeat steps 1-3 to add the upstream :command:`curl` bundle to the mix: .. code-block:: bash - ps aux | grep nginx + mixer bundle add curl + mixer build bundles + mixer build update - .. note:: - - If `nginx` is active, a list should appear showing one master process - and a few worker processes. - -Connect the URL to mixer ------------------------- - -Add the URL of the `nginx` server to builder.conf. Your |CL| clients connect -to this URL to find the update content. - -#. Get the IP address of your nginx server: + And build optional delta-packs, which will help reduce client update time: .. code-block:: bash - networkctl status + mixer build delta-packs --from 10 --to 20 -#. In the mixer workspace, edit `builder.conf` to set the value for - CONTENTURL and VERSIONURL to the IP `Address` of your `nginx` server. + Refresh your http://localhost site and now you can see the update content for + mix version 20. + + Look in ~/mixer/update/www/ to see the update content in your + workspace. + +Example 3: Deploy updates to target +=================================== + +The image created in Example 2 is directly bootable in QEMU. In this example, +we'll boot the image from Example 2 to verify it, and update the image from mix +version 10 (which the image was built from), to mix version 20. + +#. Set up the QEMU environment. + + Install the :command:`kvm-host` bundle to your |CL|: .. code-block:: bash - nano builder.conf + sudo swupd bundle-add kvm-host - .. note:: - - For example: - CONTNENTURL=http://192.168.25.52 - VERSIONURL=http://192.168.25.52 - -Create a workspace -================== - -Use the following command to create an empty directory in your |CL| image to -use as a **workspace** for mixing: - -.. code-block:: bash - - mkdir /home/clr/mix - -This guide assumes your workspace location is :file:`/home/clr/mix`. - -Generate the starting point for your mix -======================================== - -In your workspace, initialize mixer with the following command: - -.. code-block:: bash - - mixer init - -This command initializes your workspace so you can make a mix at version 10 -based on the latest released upstream |CL| version. If a :file:`builder.conf` -file is not already present in your workspace, mixer creates a default -configuration file. Additionally, mixer creates several version and tracking -files, and two bundle directories: :file:`local-bundles` and -:file:`upstream-bundles`. - -If you wish to start with a different version of upstream |CL| or a -different initial mix version, you can specify these options as flags. -For example: - -.. code-block:: bash - - mixer init --upstream-version 21060 --mix-version 10 - - -Additionally, to build a mix with your own custom RPMs, use the optional -*--local-rpms* flag, for example: - -.. code-block:: bash - - mixer init --local-rpms - -This command creates the :file:`local-yum` and :file:`local-rpms` -directories in your mix workspace and adds their paths to the generated -:file:`builder.conf`. If the configuration file already exists, you must add -the paths manually. For more information on using these directories or -setting them up manually, see `Create or locate RPMs for the mix`_. - -If all upstream |CL| bundles will be part of the mix, you can easily add -them all during initialization with the optional *--all-upstream* flag. For example: - -.. code-block:: bash - - mixer init --all-upstream - -Finally, you may want to track the contents of your mixer workspace with a -git repository. This is a great way to track changes to your mix's content -or to revert to earlier versions if something goes wrong. Mixer can set this -up automatically with the optional *--git* flag, for example: - -.. code-block:: bash - - mixer init --git - -.. note:: - You can use any or all of the above optional flags at the same time, for example: + Get the virtual EFI firmware, download the image launch script, and make it + executable: .. code-block:: bash - mixer init --upstream-version 21060 --mix-version 10 --local-rpms --all-upstream --git + curl -O https://download.clearlinux.org/image/OVMF.fd + curl -O https://download.clearlinux.org/image/start_qemu.sh + chmod +x start_qemu.sh -Edit builder.conf -================= +#. Start your VM image (created in Example 2): -To configure the mixer tool, edit the :file:`builder.conf` as needed. + .. code-block:: bash -The file :file:`builder.conf` is read automatically from the current -workspace directory. Use the *--config* flag during initialization -to specify an alternate path to the file as needed. + sudo ./start_qemu.sh release.img -The :file:`builder.conf` file has different sections, for example: +#. Log in as root and set a password -* The `[Builder]` section provides the mixer tools with the required - configuration options. This section defines the path where the generated - bundles and update metadata are published. +#. Try out your mix. -* The `[swupd]` section contains specific update parameters. The - :abbr:`swupd-server (software update server)` creates an update using - said specific update parameters. + Take a look at the default bundles installed in your mix: -Edit the configuration file according to your needs with the command: + .. code-block:: bash -.. code-block:: bash + swupd info + swupd bundle-list + swupd bundle-list -a - vim /home/clr/mix/builder.conf + Note that you cannot see the curl bundle that you added in Example 2 because + your mix is still on version 10. -Your version of the :file:`builder.conf` file should resemble the -following example: + Check for updates. You should see that version 20 is available. Use swupd to + update your mix: + + .. code-block:: bash + + swupd check-update + swupd update + swupd bundle-list -a + + Now your mix should be at version 20 and curl is now available. Try using + curl. This will fail as curl is not yet installed: + + .. code-block:: console + + curl: command not found + To install curl use: swupd bundle-add curl + + Add the new bundle from your update server to your VM. Retry curl. It works! + + .. code-block:: bash + + swupd bundle-add curl + curl -O https://download.clearlinux.org/image/start_qemu.sh + + And shutdown your VM: + + .. code-block:: bash + + poweroff + +.. Example: Create a mix with custom RPM +.. ------------------------------------- +.. TODO future example to show copy into local-rpms... + +References +********** + +Reference the `mixer man page`_ for details regarding mixer commands and options. + +.. contents:: + :local: + :depth: 1 + +.. rst-class:: content-collapse + +builder.conf +============ + +mixer initialization creates a :file:`builder.conf` that stores the basic +configuration for the mixer tool. The items of primary interest are CONTENTURL +and VERSIONURL, which will be used by systems updating against your custom +content. .. code-block:: console + #builder.conf + #VERSION 1.0 [Builder] @@ -286,586 +465,379 @@ following example: LOCAL_RPM_DIR = "" DOCKER_IMAGE_PATH = "clearlinux/mixer" -The following variables require further explanation: - -* The `LOCAL_BUNDLE_DIR` variable sets the path where mixer stores the local - bundle definition files. These bundle definition files include any new, - original bundles you create, along with any edited versions of upstream - |CL| bundles. - -* The `SERVER_STATE_DIR` variable sets the path for the output of the mix - content. Mixer automatically creates the path for you, but the path can be - set to any location. In this example, we use the workspace directory. - -* The `YUM_CONF` variable sets the path where mixer automatically generates - the :file:`.yum-mix.conf` yum configuration file. The yum configuration - file points the chroot-builder to the path where the RPMs are stored. - -* The `CERT` variable sets the path where mixer stores the - :file:`Swupd_Root.pem` certificate file. The chroot-builder needs the - certificate file to sign the root :file:`Manifest.MoM` file to provide - security for content verification. The value of the `CERT` variable can - point to a different certificate. The chroot-builder inserts the - certificate specified in this value into the - :file:`/os-core-update/usr/share/clear/update-ca/` path. The software - update client uses this certificate to verify the :file:`Manifest.MoM` - file's signature. For now, we **strongly** recommend that you do not modify - this line, as the certificate that :abbr:`swupd (Software Updater)` - expects needs to have a very specific configuration to sign and verify - properly. Mixer automatically generates the certificate, if you do not - provide the path to an existing one, and signs the :file:`Manifest.MoM` - file to provide security for the updated content you create. - -* The `CONTENTURL` and `VERSIONURL` variables set the domain or IP address - where swupd looks for your update content and the corresponding version. - You must set these variables to the domain or IP-address of the server - hosting the update content. You can use any web server to host your update - content. To learn how to install and configure web server using |CL|, see - :ref:`create-nginx-web-server`. For our example, the web update content - within the `SERVER_STATE_DIR` directory is located here: - :file:`/home/clr/mix/update/www`. If the web server is on the same machine - as this directory, you can create a symlink to the directory in your web - server's document root to easily host the content. These URLs are - embedded in images created for your mix. The `swupd-client` looks at - these URLs to determine if a new version is available and the location - from where to download the updated content. These links are equivalent - to the |CL| `update page`_ but for the mix. - -* The `VERSIONS_PATH` variable sets the path for the mix version and upstream - |CL| version's two state files: :file:`mixversion` and - :file:`upstreamversion`. Mixer creates both files for you when you set up - the workspace. - -* The `DOCKER_IMAGE_PATH` variable sets the base name of the docker image - mixer will pull down in order to run builds in the proper container. - -.. note:: If you are working only with |CL| bundles, then - skip to `List, edit, create, add, remove, or validate bundles`_. - - -Create custom RPMs -================== - -Create or locate RPMs for the mix ---------------------------------- - -If you create RPMs from scratch, you can use `autospec`, `mock`, `rpmbuild`, -or similar tools to build them. If the RPMs are not built on |CL|, ensure -your configuration and toolchain builds them correctly for |CL|, or else -there is no guarantee they will be compatible. For more information on -building the RPMs properly, refer to our `build RPMs instructions`_. - -Import RPMs into workspace --------------------------- - -#. Create a :file:`local-rpms` directory in your workspace, for example, - :file:`/home/clr/mix/local-rpms`. - -#. Copy the RPMs into the directory you created. - -#. Add the following line to your :file:`builder.conf` file: - - .. code-block:: console - - LOCAL_RPM_DIR=/home/clr/mix/local-rpms - -Mixer uses this directory to find the RPMs to build a local RPM repo for -yum to use. - -Create a local RPM repo ------------------------ - -#. Create an empty directory in your workspace named :file:`local-yum`. -#. Add the path to your :file:`builder.conf` file: - - .. code-block:: console - - LOCAL_REPO_DIR=/home/clr/mix/local-yum - -#. With these values configured, generate the yum repo with the following - command: - - .. code-block:: bash - - mixer add-rpms - -After the tool exits, you should see the RPMs and a repository data -directory in :file:`/home/clr/mix/local-yum`. If the RPMs are not all in this -:file:`local-yum` directory, check to ensure that the RPM files are valid -and not corrupt. - -List, edit, create, add, remove, or validate bundles -==================================================== - -The bundles in the mix are specified in the mix bundle list. Mixer stores -this list as a flat file called :file:`mixbundles` in the path set by the -`VERSIONS_PATH` variable of the :file:`builder.conf` file. Mixer -automatically generates the :file:`mixbundles` list file during -initialization. Mixer reads and writes the bundle list file when you change -the bundles of the mix. - -List the bundles in the mix ---------------------------- - -To view the bundles already in the mix, enter the following command: - -.. code-block:: bash - - mixer bundle list - -This command shows a list of every bundle in the mix. Bundles can include -other bundles. Those nested bundles can themselves include other -bundles. When listing bundles with this command, mixer automatically -recurses through the includes to show every single bundle in the mix. - -If you see an unexpected bundle in the list, that bundle is probably included -in another bundle. Use the *--tree* flag to get a better view of how -a bundle ended up in the mix, for example: - -.. code-block:: bash - - mixer bundle list --tree - -This command shows a visual representation of the inclusion relationships -between the bundles in the mix. - -Bundles fall into two categories: **upstream** and **local**. - -Upstream bundles are those provided by |CL|. - -Mixer automatically downloads and caches upstream bundle definition files. -These definition files are stored in the :file:`upstream-bundles` directory -in the workspace. Do **not** modify the files in this directory. This -directory is simply a mirror for mixer to use. - -The mixer tool automatically caches the bundles for the |CL| version -configured in the :file:`upstreamversion` file. Mixer also cleans up old -versions once they are no longer needed. You can see the available upstream -bundles with the following command: - -.. code-block:: bash - - mixer bundle list upstream - -Local bundles are bundles that you create, or are edited versions of upstream -bundles. - -Local bundle definition files live in the :file:`local-bundles` directory. -The `LOCAL_BUNDLE_DIR` variable sets the path of this directory in your -:file:`builder.conf` configuration file. For this example, the path is -:file:`/home/clr/mix/local-bundles`. You can see the available local bundles -with the following command: - -.. code-block:: bash - - mixer bundle list local - -Both the local and upstream :command:`bundle list` commands accept the -*--tree* flag to show a visual representation of the inclusion relationships -between the bundles in the mix. - -Edit the bundles in the mix ---------------------------- - -**Mixer always checks local bundles first and the upstream bundles second.** - -Therefore, bundles in the :file:`local-bundles` directory always take -precedence over any upstream bundles that have the same name. - -This precedence enables you to edit upstream bundles. The local, edited -version of the bundle overrides the bundle version found upstream. - -For example, to edit the `bundle1` definition file, we use the following -command: - -.. code-block:: bash - - mixer bundle edit bundle1 - -If `bundle1` is found in your local bundles, mixer edits this bundle -definition file. If instead `bundle1` is only found upstream, mixer copies -the bundle definition file from upstream into your :file:`local-bundles` -directory first. - -In both cases, mixer launches your default editor to edit the file. When the -editor closes, mixer automatically validates the edited bundle file and -reports any errors found. If mixer finds an error, you can edit the file -as-is, revert and edit, or skip and move on to the next bundle. If you skip a -file, mixer saves a backup of the original file with the ``.orig`` suffix. -Because mixer always checks your local bundles first, edited copies of an -upstream bundle always take precedence over their upstream counterpart. You -can edit multiple bundles with the following command: - -.. code-block:: bash - - mixer bundle edit bundle1 bundle2 [bundle3 ...] - -Create bundles for the mix --------------------------- - -To create a totally **new bundle**, the bundle name you specify cannot exist -upstream. If that is the case, create a :file:`new-bundle` with the following -command: - -.. code-block:: bash - - mixer bundle edit new-bundle - -This command generates a blank template in :file:`local-bundles` with the -:file:`new-bundle` filename. Mixer launches the editor for you to fill out -the bundle and performs validation when you exit the editor. Add your package -or packages to the bundle definition file to define the packages to install -as part of the bundle. - -.. note:: - - The :command:`mixer bundle edit` command accepts multiple bundles at once. - Thus, you can create multiple new bundles in a single command, for - example: - - .. code-block:: bash - - mixer bundle edit new-bundle1 new-bundle2 [new-bundle3 ...] - -Add bundles to the mix ----------------------- - -Add `bundle1` to your mix with the following command: - -.. code-block:: bash - - mixer bundle add bundle1 - -This command adds the specified bundles to the mix bundles list stored in -your :file:`mixbundles` file. For each bundle you add, mixer checks your -local and upstream bundles to ensure the added bundles actually exist. If -mixer cannot find the bundle, it reports back an error. Additionally, when -mixer adds a bundle, it tells you whether the bundle is local or upstream. -Alternatively, you can learn this information with the -:command:`mixer bundle list` command. Refer to `List the bundles in the mix`_. - -To add multiple bundles at once, use the following command: - -.. code-block:: bash - - mixer bundle add bundle1 bundle2 [bundle3 ...] - -Remove bundles from the mix ---------------------------- - -Remove `bundle1` from your mix with the following command: - -.. code-block:: bash - - mixer bundle remove bundle1 - -This command removes `bundle1` from the mix bundle list stored in your -:file:`mixbundles` file. By default, this command does not remove the bundle -definition file from your local bundles. To completely remove a bundle, -including its local bundle definition file, use the following command with -the *--local* flag: - -.. code-block:: bash - - mixer bundle remove --local bundle1 - -By default, removing a local bundle file with this command removes the bundle -from the mix as well. To only remove the local bundle definition file, use -the following command with the *--mix=false* flag: - -.. code-block:: bash - - mixer bundle remove --local --mix=false bundle1 - -If you remove a local, edited version of an upstream bundle and keep the -bundle in the mix, the mix then references the original upstream version of -the bundle. - -On the other hand, if you remove a bundle that is only found locally but -keep the bundle in the mix bundles list, mixer will not find a valid -bundle definition file and will produce an error. - -Validate the bundles in the mix -------------------------------- - -Mixer performs basic validation on all bundles when used throughout the -system. - -Mixer checks the validity of the bundle's syntax and name. Optionally, you -can run this validation manually on `bundle1` with the following command: - -.. code-block:: bash - - mixer bundle validate bundle1 - -.. note:: This command can be useful in many circumstances. One example is - when importing already-existing local bundles from other projects. - -If you use the optional *--strict* flag, the command additionally -checks if the rest of the bundle header fields can be parsed, if the bundle -header fields are non-empty, and if the bundle header ``Title`` field and -the bundle filename match. Perform a strict validation of `bundle1` with the -following command: - -.. code-block:: bash - - mixer bundle validate --strict bundle1 - -Validate multiple bundles with the following command: - -.. code-block:: bash - - mixer bundle validate bundle1 bundle2 [bundle3 ...] - -Managing bundles with Git -------------------------- - -If you initialized your workspace to be tracked as a git repository -with the :command:`mixer init --git` command, it might be useful to apply a -git commit after you modify the mix bundle list or edit a bundle definition -file. - -All the :command:`mixer bundle` commands in the previous sections support an -optional *--git* flag. This flag automatically applies a git commit -when the command completes, for example: - -.. code-block:: bash - - mixer bundle remove --git bundle1 - -Build the bundle chroots -======================== - -To build all the ``chroots`` based on the defined bundles, use the following -command in your workspace: - -.. code-block:: bash - - mixer build bundles - -If the mix has many bundles, this step might take some time. - -Mixer automatically gathers the bundle definition files for the upstream -bundles into a :file:`upstream-bundles` directory, and user bundles should -be placed directly into :file:`local-bundles`. The local path is set in -the `LOCAL_BUNDLE_DIR` variable in the :file:`builder.conf`. **Do not edit -files in upstream-bundles.** Mixer automatically deletes the contents of -the :file:`upstream-bundles` directory before repopulating the directory -on-the-fly if a new version must be downloaded. - - -Create an update -================ - -Create an update with the following command: - -.. code-block:: bash - - mixer build update - -When the build completes, you can find the mix update content under -:file:`/home/clr/mix/update/www/VER`. In our example, the update content is -found in :file:`/home/clr/mix/update/www/{}`. `` -is the defined mix version, which is 10 by default. - -.. code-block:: bash - - mixer build update - -Mixer creates all the content needed to make a fully usable mix with this -step. However, only *zero packs* are automatically generated. Zero packs are -the content needed to go from nothing to the mix version for which you just -built the content. - -You can create optional *delta packs*, which allow the transition from one -mix version to another, with the following command: - -.. code-block:: bash - - mixer build delta-packs --to --from - -This command generates all delta packs for the bundles changed from -`PAST_VERSION` to `MIX_VERSION`. Mixer cannot create delta packs for the -first build because the update is from version 0. Version 0 implicitly has -no content, thus mixer can generate no deltas. - -For subsequent builds, you can run :file:`mixer-pack-maker.sh` to generate -delta content between them, for example: 10 to 20. - -Create an image -=============== - -Since mixer uses the `ister` tool to create a bootable image from your -updated content, we must first configure the `ister` tool. To configure the -image `ister` creates, we need the `ister` configuration file. Obtain a copy -with the default values from the `ister` package with the following command: - -.. code-block:: bash - - sudo cp /usr/share/defaults/ister/ister.json release-image-config.json - -For reference, you can inspect the `Clear Linux ister configuration file`_ -used for releases. - -Edit the configuration file to include all bundles you want *preinstalled* in -the image. Users can install the bundles in the mix that are not included in -the configuration file with the following command: - -.. code-block:: bash - - sudo swupd bundle-add bundle1 - -Keeping the list of bundles in the configuration file small allows for a -smaller image size. For the minimal base image, the list is: - -.. code-block:: console - - "Bundles": ["os-core", "os-core-update", "kernel-native"] - -Next, set the `Version` field to the mix version that you want the content -mixer to use to build the image. `ister` allows you to build an image from -any mix version that you have built, not just from the current version. In -our example so far, `Version` is set to 10. - -With the `ister` tool configured, build the image with the following command: - -.. code-block:: bash - - sudo mixer build image - -Mixer automatically looks for the :file:`release-image-config.json` file, but -you can freely choose the filename. To use a different name, simply pass the -*--template* flag when creating your image, for example: - -.. code-block:: bash - - sudo mixer build image --template path/to/file.config - -By default, `ister` uses the format version of the build machine it runs on. -Therefore, if the format you are building differs from the format of the |CL| -OS you are building on, you must use the *--format * -flag. Find the current format version of your OS with the following command: - -.. code-block:: bash - - sudo cat /usr/share/defaults/swupd/format - -Update the next mix version information -======================================= - -Increment the mix version number for the next mix with the following command: - -.. code-block:: bash - - mixer versions update - -This command automatically updates the mix version stored in the -:file:`mixversion` file, incrementing it by 10. To increment by a different -amount, use the *--increment* flag, for example: - -.. code-block:: bash - - mixer versions update --increment 10 - -Alternatively, to set the mix version to a specific value, use the -*--mix-version* flag, for example: - -.. code-block:: bash - - mixer versions update --mix-version 20 - -The :command:`mixer versions update` command does not allow you to set the -mix version to a value less than its current value. The mix version is -expected to always increase, even if the new mix is undoing an earlier -change. - -If you have been tracking your workspace with git, you can restore the mix to -an earlier state. However, be careful when "rewriting history" if you have -published the mix content to users already. - -Use the following command with the the *--upstream-version* flag to -update the upstream version of |CL| used as a base for the mix: - -.. code-block:: bash - - mixer versions update --upstream-version 21070 - -This command also accepts the keyword "latest": - -.. code-block:: bash - - mixer versions update --upstream-version latest - -This command sets the upstream version to the latest released version of -upstream |CL| within the same format version. The -:command:`mixer versions update` command does not allow you to set an -upstream version to a value that crosses an upstream format boundary. -Such values require a "format bump" build, which is currently a -manual process. Refer to :ref:`mixer-format` for more information. - -Optionally, you can learn which mix version or upstream version you are -currently using with the following command: - -.. code-block:: bash - - mixer versions - -At this point, you can continue to iterate through the workflows and make -modifications as needed, for example: - -#. Add, remove, or modify bundles. -#. Build the chroots with: - - .. code-block:: bash - - mixer build chroots - -#. Build and update with: - - .. code-block:: bash - - mixer build update - -#. Optionally, you can create delta packs with: - - .. code-block:: bash - - mixer build delta-packs --to --from - -.. _mixer-format: +Additional explanation of variables in :file:`builder.conf` is provided in Table +1. + ++-------------------------------+----------------------------------------------------------+ +| **Variable** | **Explanation** | ++-------------------------------+----------------------------------------------------------+ +| `CERT` | Sets the path where mixer stores the certificate file | +| | used to sign content for verification. mixer | +| | automatically generates the certificate if you do not | +| | provide the path to an existing one, and signs the | +| | :file:`Manifest.MoM` file to provide security for the | +| | updated content you create. | +| | | +| | The chroot-builder uses the certificate file to sign | +| | the root :file:`Manifest.MoM` file, to provide | +| | security for content verification. | +| | | +| | The swupd uses this certificate to verify the | +| | :file:`Manifest.MoM` file's signature. | +| | | +| | For now, we strongly recommend that you do not modify | +| | this variable, as swupd expects a certificate with a | +| | very specific configuration to sign and verify | +| | properly. | ++-------------------------------+----------------------------------------------------------+ +| `CONTENTURL` and `VERSIONURL` | Set these variables to the IP address of the web server | +| | hosting the update content. | +| | | +| | VERSIONURL is the IP address where the swupd client | +| | looks to determine if a new version is available. | +| | | +| | CONTENTURL is the location where swupd will pull content | +| | updates from. | +| | | +| | If the web server is on the same machine as the | +| | SERVER_STATE_DIR directory, you can create a symlink to | +| | the directory in your web server's document root to | +| | easily host the content. | +| | | +| | These URLs are embedded in the images created by mixer. | ++-------------------------------+----------------------------------------------------------+ +| `DOCKER_IMAGE_PATH` | Sets the base name of the docker image mixer will pull | +| | down in order to run builds in the proper container. | ++-------------------------------+----------------------------------------------------------+ +| `LOCAL_BUNDLE_DIR` | Sets the path where mixer stores the local bundle | +| | definition files. The bundle definition files include | +| | any new, original bundles you create, along with any | +| | edited versions of upstream bundles. | ++-------------------------------+----------------------------------------------------------+ +| `SERVER_STATE_DIR` | Sets the path for where mixer outputs content. By | +| | default, mixer will automatically set the path. | ++-------------------------------+----------------------------------------------------------+ +| `VERSIONS_PATH` | Sets the path for the mix version and upstream version's | +| | two state files: :file:`mixversion` and | +| | :file:`upstreamversion`. mixer creates both files for | +| | you when you set up the workspace. | ++-------------------------------+----------------------------------------------------------+ +| `YUM_CONF` | Sets the path where mixer automatically generates the | +| | :file:`.yum-mix.conf` file. | +| | | +| | The yum configuration file points the chroot-builder to | +| | where the RPMs are stored. | ++-------------------------------+----------------------------------------------------------+ +| **Table 1**: *Variables in builder.conf* | ++-------------------------------+----------------------------------------------------------+ Format version -============== +-------------- -The `Format` variable set in the :file:`builder.conf` file can be more -precisely referred to as an OS *compatibility epoch*. Versions of the OS -within a given epoch are fully compatible and can update to any other -version within that epoch. Across the `Format` boundary, the OS has changed -in such a way that updating from build M in format X, to build N in format Y -will not work. Generally, this scenario occurs when the software updater or -software manifests change in a way that they are no longer compatible with -the previous update scheme. +Compatible versions of an OS are tracked with an OS *compatibility epoch*. +Versions of an OS within an epoch are fully compatible and can update to any +other version within that epoch. The compatibility epoch is set as the `Format` +variable in the :file:`mixer.state` file. Variables in the :file:`mixer.state` +are used by mixer between executions and should not be manually changed. -Using a format increment, we insure pre- and co-requisite changes flow out -with proper ordering. The updated client only ever updates to the latest +If `Format` increments to a new epoch (a "format bump"), the OS has changed in +such a way that updating from build M in format X, to build N in format Y will +not work. Generally, this scenario occurs when the software updater/the software +has a change such that it is no longer compatible with the previous update +scheme, or when a package is removed from the update stream and the update +must ensure the files associated with that package are removed from the system. + +Using a format increment, we make sure pre- and co-requisite changes flow out +with proper ordering. The updated client will only update to the latest release in its respective format version, unless overridden by command line -flags. Thus, we can guarantee that all clients update to the final version -in their given format. The given format *must* contain all the changes -needed to understand the content built in the subsequent format. Only after -reaching the final release in the old format can a client continue to update -to releases in the new format. +flags. This way we can guarantee that all clients update to the final version +in their given format. -When creating a custom mix, the format version should start at "1" or some -known number such as the host system format. The format version should -increment only when a compatibility breakage is introduced. Normal updates, -like updating a software package for example, do not require a format -increment. +The given format *must* contain all the changes needed to understand the content +built in the next format. Only after reaching the final release in the old +format can a client continue to update to releases in the new format. -.. _update page: https://cdn.download.clearlinux.org/update/ +The format version is incremented only when a compatibility breakage is +introduced. Normal updates, like updating a software package, do not require a +format increment. -.. _format bumps wiki: https://github.com/clearlinux/swupd-server/wiki/Format-Bumps +.. rst-class:: content-collapse -.. _build RPMs instructions: https://github.com/clearlinux/common#build-rpms-for-a-package +Bundles +======= -.. _Clear Linux ister configuration file: - https://raw.githubusercontent.com/bryteise/ister/master/release-image-config.json +mixer stores information about the bundles included in a mix in a flat file +called :file:`mixbundles`, located in the path set by the VERSIONS_PATH +variable in :file:`builder.conf`. :file:`mixbundles` is automatically created +when the mix is initiated. mixer will refresh the file each time you change the +bundles in the mix. + +Bundles can include other bundles. Nested bundles can themselves include other +bundles. If you see an unexpected bundle in your mix, it is likely a nested +bundle in one of the bundles you explicitley added. + +A bundle will fill into one of two categoris: upstream or local. Upstream +bundles are those provided by |CL|. Local bundles are either modified upstream +bundles or new local bundles. + +Upstream bundles +---------------- + +mixer automatically downloads and caches upstream bundle definition files. These +definition files are stored in the upstream-bundles directory in the workspace. +Do not modify the files in this directory. This directory is simply a mirror for +mixer to use. mixer will automatically delete the contents of this directory +before repopulating it on-the-fly if a new version must be downloaded. + +The mixer tool automatically caches the bundles for the |CL| version configured +in the :file:`upstreamversion` file. mixer also cleans up old versions once they +are no longer needed. + +Local bundles +------------- + +Local bundles are bundles that you create, or are edited versions of upstream +bundles. Local bundle definition files are stored in the local-bundles directory +in the workspace. The LOCAL_BUNDLE_DIR variable sets the path of this directory +in the :file:`builder.conf` file. + +*mixer always checks for local bundles first and the upstream bundles second.* +So bundles in the local-bundles directory will always take precedence over any +upstream bundles that have the same name. This precedence enables you to copy +upstream bundles locally, and edit into a local variation. + +Bundle configuration +-------------------- + +mixer provides commands to configure the bundles for a mix, for example to add a +bundle to a mix, to create a new bundle for a mix, or to remove a bundle from a +mix. View the `mixer.bundle man page`_ for a full list of commands and more +information on configuring bundles in a mix. + +Editing an existing local bundle is as simple as opening the bundle definition +file in your favorite editor, making the desired edits, and saving your changes. + +A note on removing bundles from a mix: By default, removing a bundle will only +remove the bundle from the mix. The local bundle defintion file will still +remain. To completely remove a bundle, including its local bundle definition +file, use the :command:`--local` flag. + +If you remove the bundle definition file for a local, edited version of an +upstream bundle in a mix, the mix will revert to reference the original upstream +version of the bundle. + +.. rst-class:: content-collapse + +Configure and enable Docker +=========================== + +Use these steps to enable Docker for the mixer tool. Make sure to +`Configure Docker proxy info`_ first if needed. + +#. Start the Docker daemon: + + .. code-block:: bash + + sudo systemctl start docker + sudo chmod 777 /var/run/docker.sock + sudo docker info + +#. Add user to the docker group + + .. code-block:: bash + + sudo usermod -G docker -a + +Pull Docker container manually (optional) +----------------------------------------- + +By default, mixer will automatically pull a Docker container for mixing if one +does not already exist. If you need to troubleshoot the mixer container, it may +be useful to manually pull a mixer Docker container. + +Versions of the mixer Docker container are available under the tags for the +`clearlinux/mixer repo `_ +on Docker Hub. Each version of the mixer Docker container is named after the +associated |CL| upstream format version. Refer to `Format version`_ for +additional information on upstream format versions. + +Use the following steps to manually pull a mixer Docker container: + +#. Find the version of the container you need by viewing the tags for the + `clearlinux/mixer repo `_ + on Docker Hub. + +#. Pull the latest container version: + + .. code-block:: bash + + docker pull clearlinux/mixer: + +#. View local docker images: + + .. code-block:: bash + + docker images + +.. rst-class:: content-collapse + +Configure Docker proxy info +=========================== + +If needed, use these steps to configure the Docker proxy information. + +Configure the Docker daemon proxies: + +#. Create the Docker daemon proxy config directory: + + .. code-block:: bash + + sudo mkdir -p /etc/systemd/system/docker.service.d + + Create :file:`/etc/systemd/system/docker.service.d/http-proxy.conf` and + add the following using your own proxy values: + + .. code-block:: console + + [Service] + Environment="HTTP_PROXY=:" + Environment="HTTPS_PROXY=:" + +#. Reload the Docker daemon: + + .. code-block:: bash + + sudo systemctl daemon-reload + +Configure the Docker container proxies, in order to pass proxy +settings to containers: + +#. Create a directory for your container config: + + .. code-block:: bash + + mkdir ~/.docker + +#. Create the config file :file:`~/.docker/config.json` and add the following + entries, using your own proxy values: + + .. code-block:: console + + { + "proxies": + { + "default": +   { +    "httpProxy": ":", +     "httpsProxy": ":" +   } + } + } + +#. Set ownership and permission on the docker config directory: + + .. code-block:: bash + + sudo chown "$USER":"$USER" /home/"$USER"/.docker -R + sudo chmod g+rwx "$HOME/.docker" -R + +Lastly, configure proxies to allow mixer to access upstream content from behind +a firewall. For example: + +#. Open your :file:`$HOME/.bashrc` file and add proxy and port values for the + following: + + .. code-block:: console + + export http_proxy=":" + export https_proxy=":" + export HTTP_PROXY=":" + export HTTPS_PROXY=":" + export no_proxy="<...>" + +#. Log out and log back in for the proxies to take effect. + +.. rst-class:: content-collapse + +Set up a nginx web server for mixer +=================================== + +A web server is needed to host your update content. In this example, we use the +nginx web server, which comes with |CL|. + +Set up a nginx web server for mixer with the following steps: + +#. Install the :command:`nginx` bundle: + + .. code-block:: bash + + sudo swupd bundle-add nginx + +#. Make the directory where mixer updates will reside: + + .. code-block:: bash + + sudo mkdir -p /var/www + +#. Create a symbolic link between your workspace updates and the updates on the + local nginx web server. In this example, `$HOME/mixer` is the workspace for + the mix. + + .. code-block:: bash + + sudo ln -sf $HOME/mixer/update/www /var/www/mixer + +#. Set up ``nginx`` configuration: + + .. code-block:: bash + + sudo mkdir -p /etc/nginx/conf.d + +#. Copy the default example configuration file: + + .. code-block:: bash + + sudo cp -f /usr/share/nginx/conf/nginx.conf.example /etc/nginx/nginx.conf + +#. Configure the mixer update server. Create and add the following server + configuration content to :file:`/etc/nginx/conf.d/mixer.conf` (sudo required): + + .. code-block:: console + + server { + server_name localhost; + location / { + root /var/www/mixer; + autoindex on; + } + } + +#. Restart the daemon, enable nginx on boot, and start the service. + + .. code-block:: bash + + sudo systemctl daemon-reload + + sudo systemctl enable nginx + + sudo systemctl start nginx + +#. Verify the web server is running at \http://localhost. At this point + you should no longer see a "404 Not Found" message. + +Related topics +************** + +* :ref:`About mixer ` +* :ref:`mixin` +* :ref:`autospec-about` +* :ref:`bundles-about` +* :ref:`swupd-about` + +.. _Docker Hub: https://hub.docker.com/r/clearlinux/mixer/tags/ +.. _mixer man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.1.rst +.. _mixer.init man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.init.1.rst +.. _mixer.bundle man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.bundle.1.rst +.. _mixer.build man page: https://github.com/clearlinux/mixer-tools/blob/master/docs/mixer.build.1.rst +.. _suitable versions: https://github.com/clearlinux/clr-bundles/releases +.. _Install a bundle: https://clearlinux.org/documentation/clear-linux/guides/maintenance/swupd-guide#adding-a-bundle From 101020bd843a6714d65f743b8aaf0a2728b7d1f7 Mon Sep 17 00:00:00 2001 From: Kevin Putnam Date: Tue, 2 Apr 2019 15:52:04 -0700 Subject: [PATCH 4/4] moved tooling items in guides to clear linux tooling section --- .../clear-linux/guides/{maintenance => clearlinux}/autospec.rst | 0 source/clear-linux/guides/{maintenance => clearlinux}/ister.rst | 0 source/clear-linux/guides/{maintenance => clearlinux}/mixer.rst | 0 source/clear-linux/guides/{maintenance => clearlinux}/mixin.rst | 0 .../guides/{maintenance => clearlinux}/swupd-guide.rst | 0 5 files changed, 0 insertions(+), 0 deletions(-) rename source/clear-linux/guides/{maintenance => clearlinux}/autospec.rst (100%) rename source/clear-linux/guides/{maintenance => clearlinux}/ister.rst (100%) rename source/clear-linux/guides/{maintenance => clearlinux}/mixer.rst (100%) rename source/clear-linux/guides/{maintenance => clearlinux}/mixin.rst (100%) rename source/clear-linux/guides/{maintenance => clearlinux}/swupd-guide.rst (100%) diff --git a/source/clear-linux/guides/maintenance/autospec.rst b/source/clear-linux/guides/clearlinux/autospec.rst similarity index 100% rename from source/clear-linux/guides/maintenance/autospec.rst rename to source/clear-linux/guides/clearlinux/autospec.rst diff --git a/source/clear-linux/guides/maintenance/ister.rst b/source/clear-linux/guides/clearlinux/ister.rst similarity index 100% rename from source/clear-linux/guides/maintenance/ister.rst rename to source/clear-linux/guides/clearlinux/ister.rst diff --git a/source/clear-linux/guides/maintenance/mixer.rst b/source/clear-linux/guides/clearlinux/mixer.rst similarity index 100% rename from source/clear-linux/guides/maintenance/mixer.rst rename to source/clear-linux/guides/clearlinux/mixer.rst diff --git a/source/clear-linux/guides/maintenance/mixin.rst b/source/clear-linux/guides/clearlinux/mixin.rst similarity index 100% rename from source/clear-linux/guides/maintenance/mixin.rst rename to source/clear-linux/guides/clearlinux/mixin.rst diff --git a/source/clear-linux/guides/maintenance/swupd-guide.rst b/source/clear-linux/guides/clearlinux/swupd-guide.rst similarity index 100% rename from source/clear-linux/guides/maintenance/swupd-guide.rst rename to source/clear-linux/guides/clearlinux/swupd-guide.rst