diff --git a/source/clear-linux/FAQ/faq.rst b/source/clear-linux/FAQ/faq.rst new file mode 100644 index 00000000..bc1de24d --- /dev/null +++ b/source/clear-linux/FAQ/faq.rst @@ -0,0 +1,189 @@ +.. _faq: + +Frequently Asked Questions (FAQ) +################################ + +Below is a list of commonly asked questions with answers sourced from the +|CL-ATTR| team and `Clear Linux community forums`_. + +.. contents:: :local: + :depth: 2 + + +General +******* + +Why did you make another distro? +================================ + +The |CL| team felt that performance was left on the table with Linux software. +|CL| takes a holistic approach to improving performance across the stack. We +also wanted to take more modern approaches with OS updates and tooling. + +| + +Can other distros copy |CL| improvements? +========================================= + +Yes, we absolutely love open source reuse and upstreaming improvements. + +| + +How often do you update? +======================== + +The |CL| team puts out multiple releases a week, often releasing 2 or more +times a day. This rolling release approach allows |CL| to remain agile to +upstream changes and security patches. + +| + +Is telemetry required? +====================== + +The telemetry solution provided by |CL| is entirely optional and customizable. +It is disabled by default. If you do choose to enable telemetry, the data +helps the |CL| team proactively identify and resolve bugs. See the +:ref:`telemetry ` page for more information. + +| + +What is the default firewall? +============================= + +|CL| packages :command:`iptables` as a bundle, however, there are no default +firewall rules. All network traffic is allowed by default. + +| + +Where are the files that I usually see under /etc like fstab? +============================================================= + +|CL| has a stateless design that maintains a separation between system files +and user files. Default values are stored under :file:`/usr/share/defaults/`. +Files under :file:`/etc/` are not created unless you create one. + +A blog post explaining how this is accomplished with :file:`/etc/fstab/` +specifically is available here: +https://clearlinux.org/news-blogs/where-etcfstab-clear-linux + +| + +Software packages +***************** + +How is software installed and updated? +====================================== + +|CL| provides software in the form of :ref:`bundles ` and +updates software with :ref:`swupd `. + +:ref:`FlatPak\* ` is an application virtualization solution that allows +more software to be available to |CL| users by augmenting the software |CL| +packages natively with software available through FlatPak. + +Our goal is to have software packaged natively and made available through +bundles whenever possible. + +| + +Does |CL| use RPMs like other distros? +====================================== + +|CL| provides software in the form of :ref:`bundles `. The RPM +format is used as an intermediary step for packaging and determining software +dependencies at OS build time. + +Individual RPMs can sometimes be manually installed on a |CL| system with the +right tools, but that is not the intended use case. + +| + +Can I install a software package from another OS on |CL|? +========================================================= + +Software that is packaged in other formats for other Linux distributions is +not guaranteed to work on |CL| and may be impacted by |CL| updates. + +If the software you're seeking is open source, please submit a request to add +it to |CL|. Submit requests on GitHub\* here: +https://github.com/clearlinux/distribution/issues + +| + +Software availability +********************* + +What software is available on |CL|? +=================================== + +Available software can be found in the `Software Store`_, through the GNOME\* +Software application on the desktop, or by using :ref:`swupd search `. + +| + +Is Google\* Chrome\* available? +=============================== + +The Google Chrome web browser is not distributed as a bundle in |CL| due to +copyright and licensing complexities. + +A discussion on manually installing and maintaining Google Chrome can be found +on GitHub: https://github.com/clearlinux/distribution/issues/422 + +| + +Is FFmpeg available? +==================== + +`FFmpeg`_ is a multimedia software suite, which is commonly used for +various media encoding/decoding, streaming, and playback. + +|CL| does not distribute FFmpeg due to well-known licensing and legal +complexities (See https://www.ffmpeg.org/legal.html and +http://blog.pkh.me/p/13-the-ffmpeg-libav-situation.html). + +Read more in the |CL| repository, including discussion of an alternative +hardware-based solution: +https://github.com/clearlinux/distribution/issues/429. + +While |CL| cannot distribute FFmpeg, a manual solution to build and install +FFmpeg under :file:`/usr/local` has been shared on the community forums: +https://community.clearlinux.org/t/how-to-h264-etc-support-for-firefox-including-ffmpeg-install. + +| + +Is ZFS\* available? +=================== + +ZFS is not available with |CL| because of copyright and licensing +complexities. BTRFS is an alternative filesystem that is available in |CL| +natively. + +A user on GitHub notes that the ZFS kernel module can be compiled, built, and +installed manually: https://github.com/clearlinux/distribution/issues/631 + +| + +Can you add a driver that I need? +================================= + +If a kernel module is available as part of the Linux kernel source tree but +not enabled in the |CL| kernels, in many cases the |CL| team will enable it +upon request. Submit requests on GitHub here: +https://github.com/clearlinux/distribution/issues + +The |CL| team does not typically add out-of-tree kernel modules as a matter of +practice because of the maintenance overhead. If the driver was unable to be +merged upstream, there is a good chance we may be unable to merge it for +similar reasons. + +Kernel modules can be individually built and installed on |CL|. See the +:ref:`kernel modules ` page for more information. + +| + + +.. _`Clear Linux community forums`: https://community.clearlinux.org +.. _`Software Store`: https://clearlinux.org/software +.. _`FFmpeg`: https://ffmpeg.org/ diff --git a/source/clear-linux/concepts/figures/stateless-1.png b/source/clear-linux/concepts/figures/stateless-1.png new file mode 100644 index 00000000..550f37ce Binary files /dev/null and b/source/clear-linux/concepts/figures/stateless-1.png differ diff --git a/source/clear-linux/concepts/figures/stateless-2.png b/source/clear-linux/concepts/figures/stateless-2.png new file mode 100644 index 00000000..94e8af22 Binary files /dev/null and b/source/clear-linux/concepts/figures/stateless-2.png differ diff --git a/source/clear-linux/concepts/stateless.rst b/source/clear-linux/concepts/stateless.rst new file mode 100644 index 00000000..3252d22a --- /dev/null +++ b/source/clear-linux/concepts/stateless.rst @@ -0,0 +1,145 @@ +.. _stateless: + +Stateless +######### + +In most operating systems, files can become intermingled with user and system +data and configurations. + +.. figure:: figures/stateless-1.png + :scale: 45% + :align: center + :alt: Stateless: User and system files mixed + + Figure 1: Without stateless, user and system files become mixed on the filesystem over time. + +|CL-ATTR| has a stateless design philosophy of which the goal is to provide an +:abbr:`OS (operating system)` that functions without excessive user +configuration or customization. Stateless in this context does *not* mean +ephemeral or non-persistent. + +.. contents:: :local: + :depth: 2 + + +File-level separation +********************* + +To accomplish a stateless design the Linux Filesystem Hierarchy is separated +between user-owned areas and |CL|-owned areas. + +.. figure:: figures/stateless-2.png + :scale: 45% + :align: center + :alt: Stateless: User and system files separation + + Figure 2: With stateless, user and system files are separated on the filesystem. + +System areas +============ +File under the :file:`/usr` directory are managed by |CL| as system files. +Files written under the :file:`/usr` directory by users can get removed +through system updates with :ref:`swupd `.This operating +assumption allows |CL| to verify and maintain integrity of system files. + +User areas +========== +Files under the :file:`/etc/`, :file:`/home`, and :file:`/var` directories are +owned and managed by the user. A freshly installed |CL| system will only have +a minimal set of files in the :file:`/etc/` directory and software installed +by |CL| does not write to :file:`/etc`. This operating assumption allows |CL| +users to clearly identify the configuration that makes their system unique. + + +Software Configuration +********************** + +With stateless separation, default software configurations are read in order +from predefined source code, |CL| provided defaults, and user-provided +configuration. + +Default configurations +====================== + +Software in |CL| provides default configuration values so that it is +immediately functional, whenever it is appropriate to do so. + +|CL| distributed software packages may be directly modified to include default +configuration values or default configuration files may be provided by |CL| +under :file:`/usr/share/defaults`. These files can be referenced as templates +for customization. + +For example, the default configuration that Apache uses when installed can be +found at :file:`/usr/share/defaults/httpd/httpd.conf` directory. + + +Overriding configurations +========================= + +If a configuration needs to be changed, the appropriate file should be +modified by the user under :file:`/etc/`. If the configuration file does not +already exist, it can be created in the appropriate location. + +User defined configuration files should contain the minimal set of desired +changes and rely on default configuration for the rest. + +For example, a customized Apache configuration can be used instead by: + +#. Create the destination directory for the configuration: + + .. code :: bash + + sudo mkdir /etc/httpd + +#. Copy the default configuration as a reference template: + + .. code :: bash + + sudo cp /usr/share/defaults/httpd/httpd.conf /etc/httpd/ + +#. Make any desired modifications to the configurations: + + .. code :: bash + + sudoedit /etc/httpd/httpd.conf + +#. Reload the service or reboot the system to pickup any changes: + + .. code :: bash + + systemctl daemon-reload httpd && systemctl restart httpd + + +This pattern can be used to modify the configurations of other programs too. +The `stateless man page`_ has application-specific examples. + + +System reset +************ + +Once advantage of the stateless design is that the system defaults can be +easily restored by simply deleting everything under :file:`/etc/` and +:file:`/var`. + +Running the commands below effectively performs a system reset as if it was +just installed: + +.. code:: + + sudo rm -rf /etc + sudo rm -rf /var + +In other Linux distributions, this can be a catastrophic action that renders a +system unable to boot. + +Additional information +********************** + +* `stateless man page`_ + +* `Where is /etc/fstab in Clear Linux? `_ + + +.. _`stateless man page`: https://github.com/clearlinux/clr-man-pages/blob/master/stateless.7.rst + + diff --git a/source/clear-linux/guides/maintenance/cpu-performance.rst b/source/clear-linux/guides/maintenance/cpu-performance.rst new file mode 100644 index 00000000..10eedb6e --- /dev/null +++ b/source/clear-linux/guides/maintenance/cpu-performance.rst @@ -0,0 +1,196 @@ +.. _cpu-performance: + +CPU Power and Performance +######################### + +Modern x86 :abbr:`CPUs (central processing units)` employ a number of features +and technologies to balance performance, energy, and thermal efficiencies. + +By default, |CL| prioritizes maximum CPU performance with the philosophy that +the faster the program finishes execution, the faster the CPU can return to a +low energy idle state. It is important to understand and evaluate all of these +technologies when troubleshooting or considering changing the defaults. + +.. contents:: :local: + :depth: 1 + +CPU power saving mechanisms +=========================== + +C-states and P-states are both CPU power saving mechanisms that are entered +under different operating conditions. The tradeoff is a slightly longer time +to exit these states when the CPU is needed once again. + +.. _c-states-section: + +C-states (idle states) +---------------------- + +C-states are hardware sleep states that are entered when it is determined that +the CPU is idle and not executing instructions. + +C-states aim to reduce power utilization by increasingly reducing clock +frequency, voltages, and features in each state. + +Although C-states can typically be limited or disabled in a system's UEFI or +BIOS configuration, these settings are overridden when the `intel_idle +driver`_ is in use. + +To view the current cpuidle driver run this command in a terminal: + +.. code:: bash + + cat /sys/devices/system/cpu/cpuidle/current_driver + +For troubleshooting, C-states can be limited with a kernel command line boot +parameter by adding :command:`processor.max_cstate=N intel_idle.max_cstate=N` +or completely disabled with :command:`idle=poll`. + +.. note:: + + * :command:`processor.max_cstate=0` is changed to :command:`processor.max_cstate=1` by the kernel to be a valid value. + + * :command:`intel_idle.max_cstate=0` disables the Intel Idle driver, not set + it to C-state 0. + +.. _p-states-section: + +P-states (performance states) +----------------------------- + +P-states, also known as *SpeedStep* on Intel processors or *Cool'n'Quiet* on +AMD processors, are states entered while the CPU is active and executing +instructions. + +P-states aim to reduce power utilization by adjusting CPU clock frequency and +voltages based on CPU demand. + +P-states can typically be limited or disabled in a system's firmware (UEFI/BIOS). + +Turbo boost +~~~~~~~~~~~ + +`Turbo Boost technology`_, found on some modern Intel CPUs, allows core(s) on +a processor to temporarily operate at a higher than rated CPU clock frequency +to accommodate demanding workloads if the CPU is under defined power and +thermal thresholds. + +Turbo boost is an extension of P-states. As such, changing or limiting +C-states or P-states impact the ability of a process to enter Turbo boost. + +Turbo boost can be disabled in a system's UEFI or BIOS. Turbo boost can also +be disabled within |CL| with the command: + +.. code:: bash + + echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo + +Linux CPU clock frequency scaling +================================= + +The CPUFreq subsystem in Linux allows the OS to control :ref:`C-states +` and :ref:`P-states ` +via CPU drivers and governors that provide algorithms that define how and when +to enter these states. + +Scaling driver +-------------- + +Linux uses the `Intel P-state driver`_, *intel_pstate*, for modern Intel +processors from the Sandy Bridge generation or newer. Other processors may +default to the *acpi-cpufreq* driver which reads values from the systems UEFI +or BIOS. + +To view the current CPU frequency scaling driver run this command in a terminal: + +.. code:: bash + + cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_driver + +Scaling governor +---------------- + +|CL| sets the CPU governor to *performance* which calls for the CPU to operate +at maximum clock frequency. In other words, P-state P0. While this may sound +wasteful at first, it is important to remember that power utilization does not +increase significantly simply because of a locked clock frequency without a +workload. + +To view the current CPU frequency scaling governor run this command in a terminal: + +.. code:: bash + + cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor + +To change the CPU frequency scaling governor: + +#. Disable |CL| enforcement of certain power and performance settings: + + .. code:: bash + + sudo systemctl mask clr-power.timer + +#. Change the governor. In the example below, the governor is set to + *performance*: + + .. code:: bash + + echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor + +The list of all governors can be found in the `Linux kernel documentation on +CPUFreq Governors`_. + +.. note:: + + The intel_pstate driver only supports *performance* and *powersave* governors. + +Thermal management +================== + +`thermald`_ is a Linux thermal management daemon used to prevent the +overheating of platforms. When temperature thresholds are exceeded, thermald +forces a C-state by inserting CPU sleep cycles and adjusts available cooling +methods. This can be especially desirable for laptops. + +By default, thermald is disabled in |CL| and starts automatically if battery +power is detected. thermald can be manually enabled using the systemd service +by running the command: + +.. code:: bash + + sudo systemctl enable thermald + sudo systemctl start thermald + +For more information, see the thermald man page: + +.. code:: bash + + man thermald + +`ThermalMonitor`_ is a GUI application that can visually graph and log +temperatures from thermald. To use ThermalMonitor, add the desktop-apps-extras +bundle and add your user account to the power group: + +.. code:: bash + + sudo swupd bundle-add desktop-apps-extras + sudo usermod -a -G power + ThermalMonitor + +.. note:: + + After adding a new group you must log out and log back in for the new group + to take affect. + + +.. _`Intel P-state driver`: https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt + +.. _`Linux kernel documentation on CPUFreq Governors`: https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt + +.. _thermald: https://01.org/linux-thermal-daemon + +.. _`intel_idle driver`: https://github.com/torvalds/linux/blob/master/drivers/idle/intel_idle.c + +.. _`ThermalMonitor`: https://github.com/intel/thermal_daemon/tree/master/tools/thermal_monitor + +.. _`Turbo Boost technology`: https://www.intel.com/content/www/us/en/architecture-and-technology/turbo-boost/turbo-boost-technology.html diff --git a/source/clear-linux/guides/tooling/debug.rst b/source/clear-linux/guides/tooling/debug.rst new file mode 100644 index 00000000..55bca98a --- /dev/null +++ b/source/clear-linux/guides/tooling/debug.rst @@ -0,0 +1,89 @@ +.. _debug: + +Debug System +############ + +|CL-ATTR| introduces a novel approach to system software debugging using +*clr-debug-info*. On the client side, the |CL| debug system obtains any +necessary debug information on-the-fly over a network during a debugging +session. On the server side, the system curates and compresses debug +information into small pieces for efficient downloading. + +For developers, this avoids the interruption during debugging that usually +happens when debug information is missing. This can be especially useful on +systems where storage is limited. + + +.. contents:: :local: + :depth: 2 + + +Background +---------- + +Software that is compiled and packaged for general usage in an operating +system typically only contains components that are used to execute the +program, like binaries and libraries. Extra developer data like the the +actual source code and symbol information are separated and excluded for +efficiency. + +The debug information helps relate binary code to human readable source code +lines and variables. Most of the time this auxiliary information is not needed +however without it, debugging a program results in limited visibility. + + +Usage +----- + +The clr-debug-info system is integrated into |CL| and seamlessly engages once +installed. + +#. Install the *dev-utils* bundle. + + .. code:: bash + + sudo swupd bundle-add dev-utils + + .. note:: + + The *telemetrics* and *performance-tools* bundles also include + clr-debug-info. + + +#. Start a debugging session against a program using a debugger, like GDB. + For example, to debug *gnome-control-center*: + + .. code:: bash + + gdb /usr/bin/gnome-control-center + +As the program is stepped through and debug information is needed, the +clr_debug_daemon will obtain it in the background. + + +Implementation +-------------- + +The implementation of the |CL| debug system is open source and available on +GitHub at: https://github.com/clearlinux/clr-debug-info/ + +.. figure:: figures/debug-diagram.png + :scale: 75% + :alt: Debug system communication flow + + Figure 1: The communication flow of the |CL| debug system + +The |CL| debug system implements a :abbr:`FUSE (filesystem in userspace)` +filesystem mounted at :file:`/usr/lib/debug` and :file:`/usr/src/debug`. The +FUSE filesystem starts automatically. Its status can be verified with +:command:`systemctl status clr_debug_fuse.service`. + +The *clr_debug_daemon* is responsible for fetching the appropriate package +debug content from the server and making it available for any debugging +programs needed it. It is socket activated whenever a request to the local +FUSE filesystem occurs. Its status can be verified with :command:`systemctl +status clr_debug_daemon.service`. + + +|CL| hosts debuginfo content packaged for consumption by |CL| debug clients at +https://download.clearlinux.org/debuginfo/ diff --git a/source/clear-linux/guides/tooling/figures/debug-diagram.png b/source/clear-linux/guides/tooling/figures/debug-diagram.png new file mode 100644 index 00000000..d85a4045 Binary files /dev/null and b/source/clear-linux/guides/tooling/figures/debug-diagram.png differ diff --git a/source/clear-linux/tutorials/tutorial-proxy.rst b/source/clear-linux/tutorials/tutorial-proxy.rst new file mode 100644 index 00000000..eab5ca47 --- /dev/null +++ b/source/clear-linux/tutorials/tutorial-proxy.rst @@ -0,0 +1,115 @@ +.. _tutorial-proxy: + +Setting up proxy +################ + +This tutorial shows you how to configure your system for use behind an +outbound proxy to access the Internet. + +|CL| :ref:`tooling` applications already benefit from the :ref:`autoproxy` +feature. + +.. contents:: + :local: + :depth: 1 + +Prerequisites +************* + +This tutorial assumes you have installed |CL| on your host system. +For detailed instructions on installing |CL| on a bare metal system, visit +the :ref:`bare metal installation guide `. + +Shells and programs in a desktop session +**************************************** + +Terminal +======== + +Add the following to your ~/.bashrc: + +.. code-block:: bash + + export http_proxy=http://your.http-proxy.url:port + export https_proxy=http://your.https-proxy.url:port + export ftp_proxy=http://your.ftp-proxy.url:port + export socks_proxy=http://your.socks-proxy.url:port + export no_proxy=".your-company-domain.com,localhost" + export HTTP_PROXY=$http_proxy + export HTTPS_PROXY=$https_proxy + export FTP_PROXY=$ftp_proxy + export SOCKS_PROXY=$socks_proxy + export NO_PROXY=$no_proxy + +wget +**** + +Run this command to enable downloading from websites from the terminal: + +.. code-block:: bash + + echo >> ~/.wgetrc <`_: .. figure:: figures/gce/00-sign-in.png :scale: 50 % - :alt: Sign in Google services + :alt: Sign in to Google services + + Figure 1: Google sign in screen #. Google Cloud Platform uses **Projects** to manage resources. Select or create a new project for hosting the |CL| VM. .. note:: - Reference the + Refer to the `Quickstart Using a Linux VM `_ - guide to get familiar with the process of creating VM instances on GCP. + guide to learn about the process of creating VM instances on GCP. #. Navigate to the latest |CL| `release folder `_ - to view currently released ``gce`` image, and download the image archive - *clear--gce.tar.gz*. + to view the currently released :abbr:`GCE (Google Compute Engine\*)` + image, and download the :file:`clear--gce.tar.gz` + image archive. You don't need to uncompress the image archive, the intact file will be uploaded to the Google Cloud Storage later. @@ -42,26 +60,32 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP). :scale: 50 % :alt: Browse Google Cloud Storage + Figure 2: Browse Google Cloud Storage + .. note:: You may need to create a billing account and link to this project before you create a bucket. .. figure:: figures/gce/02-storage-browser.png :scale: 50 % - :alt: Cloud Storage Broswer tool + :alt: Cloud Storage Browser tool + + Figure 3: Cloud Storage Browser tool * Click the ``CREATE BUCKET`` button to enter the bucket creation tool. - The bucket name must be unique as buckets in the Cloud Storage share + The bucket name must be unique because buckets in the Cloud Storage share a single global namespace. - You can leave the rest of options as defaults, and select the + Leave the remaining options set to the defaults, and select the ``Create`` button at the bottom to create a *Bucket*. .. figure:: figures/gce/03-create-bucket.png :scale: 50 % - :alt: Set an unique bucket name + :alt: Set a unique bucket name -#. Once the bucket is created, press the ``Upload files`` button + Figure 4: Set bucket name + +#. Once the bucket is created, click the ``Upload files`` button on the Bucket details page to upload the |CL| GCE image archive to the named bucket: @@ -69,24 +93,32 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP). :scale: 50 % :alt: Cloud Storage bucket is available for storing objects + Figure 5: Cloud Storage bucket + .. figure:: figures/gce/10-image-upload.png :scale: 50 % :alt: Uploading the image source archive file + Figure 6: Uploading the image source archive file + .. figure:: figures/gce/11-bucket-uploaded.png :scale: 50 % :alt: Image archive imported complete + Figure 7: Importing complete + #. Browse the Compute Engine Image library page: * Click the *Navigation menu* icon on the upper left screen menu. - * Hover mouse over the *Compute Engine* menu and select *Images*. + * Hover your mouse over the *Compute Engine* menu and select *Images*. .. figure:: figures/gce/20-gce-image.png :scale: 50 % :alt: Go to Google Compute Engine Image library + Figure 8: Image library + #. On the Compute Engine Image library page, click the ``[+] CREATE IMAGE`` menu item to create a custom image: @@ -94,18 +126,22 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP). :scale: 50 % :alt: Create a Google Compute Engine image + Figure 9: Create image + #. In the VM image creation page, change the image source type to *Cloud Storage file*. #. Under :guilabel:`Cloud Storage file`, select :guilabel:`Browse`. -#. Locate the *clear--gce.tar.gz*, +#. Locate the :file:`clear--gce.tar.gz` file, and click :guilabel:`Select`. .. figure:: figures/gce/21-create-image.png :scale: 50 % :alt: Create the image using the imported image archive object + Figure 10: Create image using imported object + Accept all default options, and click the ``Create`` button at the bottom to import the Clear Linux GCE image to the image library. @@ -113,18 +149,22 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP). :scale: 50 % :alt: Clear Linux Compute Engine image is created + Figure 11: Image is created + #. After the |CL| image is imported, you can launch a VM instance running |CL|: * Click the *Navigation menu* icon on the upper left screen menu. - * Hover mouse over the *Compute Engine* menu group and select + * Hover your mouse over the *Compute Engine* menu group and select the *VM instances* item. .. figure:: figures/gce/30-vm-instances.png :scale: 50 % :alt: Go to VM instances catalog + Figure 12: VM instances catalog + #. If no VM instance was created in this project, you will be prompted to create one. @@ -135,25 +175,33 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP). :scale: 50 % :alt: Prompt for VM creation + Figure 13: VM creation + .. figure:: figures/gce/30-vm-catalog.png :scale: 50 % :alt: List of VM instances - * In :guilabel:`Region`, decide on a region based on the - `Best practices for Compute Engine regions selection`_ . + Figure 14: VM instances list - * Under *Boot disk*, click the ``Change``. + * In :guilabel:`Region`, choose a region based on the + `Best practices for Compute Engine regions selection`_. + + * Under *Boot disk*, click the ``Change`` button. .. figure:: figures/gce/30-create-vm.png :scale: 50 % :alt: Use custom image while creating Clear Linux VM instance + Figure 15: Use custom image + * Select the *Custom images* tab for using Clear Linux OS GCE image. .. figure:: figures/gce/31-select-boot-disk.png :scale: 50 % :alt: Select Clear Linux boot disk to create a VM instance + Figure 16: Select Clear Linux boot disk to create a VM instance + * Scroll down to the bottom of the VM instance creation page, expand the *Management, security, disks, networking, sole tenancy* group. @@ -161,12 +209,14 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP). :scale: 50 % :alt: Clear Linux requires setting up SSH keys + Figure 17: Set up SSH keys + .. note:: |CL| does not allow SSH login with a root account by default. - As a result, you need to configure the VM instance with your - SSH public key, so that you are able to remote access to it. + As a result, you must configure the VM instance with your + SSH public key, so that you are able to access it remotely. - Refer also to :ref:`security` for more details. + Refer to :ref:`security` for more details. * Click the *Security* tab, copy and paste your SSH public key: @@ -174,28 +224,43 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP). :scale: 50 % :alt: Set SSH key for remote login + Figure 18: Set SSH key for remote login + .. warning:: - The username is assigned from characters preceding ``@`` in the - email address, included in the SSH key. The dot symbol "." is not - allowed, as it is an invalid character while creating user accounts - in |CL|. + The username is assigned from characters preceding ``@`` in the email + address, included in the SSH key. The dot symbol "." is not allowed, + because it is an invalid character while creating user accounts in + |CL|. * Click the ``Create`` button to create the |CL| VM. -#. The Clear Linux VM instance is created and offered a public IP address: +#. The Clear Linux VM instance is created and assigned a public IP address: .. figure:: figures/gce/41-vm-created.png :scale: 50 % :alt: Clear Linux VM instance is created and started -#. You can now SSH login to the VM by using the IP address obtained in the + Figure 19: Clear Linux VM instance is created and started + +#. You can now SSH login to the VM using the IP address obtained in the previous step, and the username associated with the SSH public key: .. figure:: figures/gce/42-ssh-vm.png :scale: 50 % - :alt: SSH log in to the Clear Linux VM + :alt: SSH login to the Clear Linux VM + + Figure 20: SSH login to Clear Linux VM + +Related topics +************** + +The following tutorials describe a similar process and may be useful references: + +* :ref:`azure` +* :ref:`aws-web` + .. _Google Cloud Platform: https://cloud.google.com/ -.. _Best practices for Compute Engine regions selection: https://cloud.google.com/solutions/best-practices-compute-engine-region-selection \ No newline at end of file +.. _Best practices for Compute Engine regions selection: https://cloud.google.com/solutions/best-practices-compute-engine-region-selection diff --git a/source/get-started/virtual-machine-install/kvm.rst b/source/get-started/virtual-machine-install/kvm.rst index 8176bedf..9b750643 100644 --- a/source/get-started/virtual-machine-install/kvm.rst +++ b/source/get-started/virtual-machine-install/kvm.rst @@ -61,7 +61,7 @@ Download and launch the virtual machine .. code-block:: bash - curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep '[0-9]'-kvm) + curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep '[0-9]'-kvm'\.') #. Uncompress the downloaded image: diff --git a/source/guides/kernel/kernel-modules-dkms.rst b/source/guides/kernel/kernel-modules-dkms.rst index 3e3cf6fe..8c8ca6ad 100644 --- a/source/guides/kernel/kernel-modules-dkms.rst +++ b/source/guides/kernel/kernel-modules-dkms.rst @@ -129,8 +129,8 @@ Before you begin, you must: Obtain kernel module source =========================== -A required :file:`dkms.conf` file inside of the kernel module's source code directory -informs DKMS how the kernel module should be compiled. +A required :file:`dkms.conf` file inside of the kernel module's source code +directory informs DKMS how the kernel module should be compiled. Kernel modules may come packaged as: @@ -141,10 +141,10 @@ Kernel modules may come packaged as: - Precompiled module binaries only (without source code) Of the package types listed above, only precompiled kernel module binaries -will not work, because |CL| requires kernel modules to be built against -the same kernel source tree before they can be loaded. If you are only able to -obtain source code without a :file:`dkms.conf` file, you must manually create a -:file:`dkms.conf` file, described later in this document. +will not work, because |CL| requires kernel modules to be built against the +same kernel source tree before they can be loaded. If you are only able to +obtain source code without a :file:`dkms.conf` file, you must manually create +a :file:`dkms.conf` file, described later in this document. #. Download the kernel module's source code. @@ -210,8 +210,8 @@ If the kernel module source does not contain a :file:`dkms.conf` file or the create the file. Review the kernel module README documentation for guidance on what needs to be -in the :file:`dkms.conf` file, including special variables that may be required to -build successfully. +in the :file:`dkms.conf` file, including special variables that may be +required to build successfully. Here are some additional resources that can be used for reference: @@ -223,6 +223,10 @@ Here are some additional resources that can be used for reference: * `Sample dkms.conf file`_ in the GitHub\* repository for the DKMS project. +.. note:: + + :command:`AUTOINSTALL=yes` must be set in the dkms.conf for the module to + be automatically recompiled with |CL| updates. The instructions below show a generic example: @@ -240,6 +244,7 @@ The instructions below show a generic example: PACKAGE_NAME=custom_module PACKAGE_VERSION=1.0 DEST_MODULE_LOCATION=/kernel/drivers/other + AUTOINSTALL=yes This example identifies a kernel module named *custom_module* with version *1.0*. diff --git a/source/guides/maintenance/assign-static-ip.rst b/source/guides/maintenance/assign-static-ip.rst index 444fbf21..c535ccab 100644 --- a/source/guides/maintenance/assign-static-ip.rst +++ b/source/guides/maintenance/assign-static-ip.rst @@ -1,71 +1,175 @@ .. _assign-static-ip: -Assign a static IP address to a network interface -################################################# - -Introduction -************ +Assign a static IP address +########################## By default, your |CL-ATTR| system automatically gets an IP address from your network via DHCP. If you do not have a DHCP server on your network or simply want to use a static IP address, follow the steps in this guide. -Process -******* +.. contents:: + :local: + :depth: 1 -#. Create this directory structure: +Identify which program is managing the interface +************************************************ - .. code-block:: bash +New installations of |CL| use NetworkManager as the default network interface +manager for all network connections. - sudo mkdir -p /etc/systemd/network +.. note:: -#. Identify the interface to be assigned the static IP address: + * The *cloud* |CL| images continue to use systemd-networkd to manage + network connections. - .. code-block:: bash + * In earlier |CL| versions, systemd-network was used to manage Ethernet + interfaces and NetworkManager was used for wireless interfaces. - ip addr - The system returns the following: +Before defining a configuration for assigning a static IP address, you should +verify which program is managing the network interface. - .. code-block:: console +#. Check the output of :command:`nmcli device` to see if NetworkManager is + managing the device. - 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 - link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 - inet 127.0.0.1/8 scope host lo - valid_lft forever preferred_lft forever - inet6 ::1/128 scope host - valid_lft forever preferred_lft forever + .. code-block:: bash - 2: wlp1s0: mtu 1500 qdisc mq state DOWN group default qlen 1000 - link/ether 4a:98:8d:e5:43:15 brd ff:ff:ff:ff:ff:ff + nmcli device status - 3: eno1: mtu 1500 qdisc fq state UP group default qlen 1000 - link/ether f4:4d:30:68:96:20 brd ff:ff:ff:ff:ff:ff - inet 10.0.1.2/24 brd 10.54.74.255 scope global dynamic eno1 - valid_lft 6766sec preferred_lft 6766sec - inet6 fe80::f64d:30ff:fe68:9620/64 scope link - valid_lft forever preferred_lft forever + If the STATE column for the device shows *connected* or *disconnected*, the + network configuration is being managed by NetworkManager and the instructions + for :ref:`using NetworkManager ` should be used. - In this example, we will use the `eno1` interface. + If the STATE column for the device shows *unmanaged*, check to see if the + device is being managed by systemd-networkd -#. Create the :file:`70-static.network` file and add the following: - .. code-block:: bash +#. Check the output of :command:`networkctl list` to see if + systemd-networkd is managing the device. - sudo $EDITOR /etc/systemd/network/70-static.network + .. code-block:: bash - [Match] - Name=[interface name] - [Network] - Address=[IP address]/24 - DHCP=yes # to get DNS info, etc. + networkctl list - Replace [interface name] and [IP address] with your specific settings. + If the SETUP column for the device shows *configured*, the network + configuration is being managed by systemd-networkd and the instructions for + :ref:`using systemd-networkd ` should be used. -#. Restart the networkd service: - .. code-block:: bash +.. _nm-static-ip: - sudo systemctl restart systemd-networkd +Using NetworkManager +******************** + +Network connections managed by NetworkManager are stored as files with the +:file:`.nmconnection` file extension in the +:file:`/etc/NetworkManager/system-connections/` directory. + +A few tools exists to aid to manipulate network connections managed by +NetworkManager: + +* nmcli - a command-line tool + +* nmtui - a text user interface that provides a pseudo graphical menu in the + terminal + +* nm-connection-editor - a graphical user interface + +The method below uses the command line tool *nmcli* to modify network +connection. + + +#. Identify the existing connection name. + + .. code:: bash + + nmcli connection show + + The output will look like this: + + .. code:: bash + + NAME UUID TYPE DEVICE + Wired connection 1 00000000-0000-0000-0000-000000000000 802-3-etherneten01 + + If a connection does not exist, it will need to be created with + :command:`nmcli connection add`. + + +#. Modify the connection to use a static IP address. Replace the variables in + brackets with the appropriate values. *[CONNECTION_NAME]* should be + replaced with the NAME from the command above. + + .. code:: + + sudo nmcli connection modify "[CONNECTION_NAME]" \ + ipv4.method "manual" \ + ipv4.addresses "[IP_ADDRESS]/[CIDR_NETMASK]" \ + ipv4.gateway "[GATEWAY_IP_ADDRESS]" \ + ipv4.dns "[PRIMARY_DNS_IP],[SECONDARY_DNS_IP]" + + + See the `nmcli developer page `_ for more + configuration options. For advanced configurations, the + :file:`/etc/NetworkManager/system-connections/*.nmconnection`. can be edited + directly. + +#. Verify your static IP address details have been set + + .. code-block:: bash + + nmcli + + + +.. _networkd-static-ip: + +Using systemd-networkd +********************** + +Network connections managed by systemd-networkd are stored as files with the +:file:`.network` file extension the :file:`/etc/systemd/network/` directory. + +Files to manipulate network connections managed by systemd-networkd must be +created manually. + +#. Create the :file:`/etc/systemd/network` directory if it doesn't exist already: + + .. code-block:: bash + + sudo mkdir -p /etc/systemd/network + +#. Create a :file:`.network` file and add the following content. Replace the + variables in brackets with the appropriate values. *[INTERFACE_NAME]* + should be replaced with LINK from the output of :command:`networkctl list` + ran previously. + + .. code-block:: bash + + sudo $EDITOR /etc/systemd/network/70-static.network + + [Match] + Name=[INTERFACE_NAME] + + [Network] + Address=[IP_ADDRESS]/[CIDR_NETMASK] + Gateway=[GATEWAY_IP_ADDRESS] + DNS=[PRIMARY_DNS_IP] + DNS=[SECONDARY_DNS_IP] + + See the `systemd-network man page + `_ + for more configuration options. + +#. Restart the systemd-networkd service: + + .. code-block:: bash + + sudo systemctl restart systemd-networkd + +#. Verify your static IP address details have been set + + .. code-block:: bash + + networkctl status -**Congratulations!** You have successfully assigned a static IP address. diff --git a/source/reference/image-types.rst b/source/reference/image-types.rst index a87fbd56..cc5d1325 100644 --- a/source/reference/image-types.rst +++ b/source/reference/image-types.rst @@ -35,14 +35,11 @@ Table 2 lists the currently available images that are platform specific. * - Image Type - Description - * - installer.img - - Preferred image of |CL| with interactive installer. + * - live-desktop.img or live-desktop.iso + - Image for booting to GNOME\* desktop to preview or install the OS. - * - installer.iso - - ISO of |CL| with interactive installer. Only for special cases where ISO image format is required (not for use with a USB key) - - * - live.img - - Image for live booting into memory, without requiring installaton. + * - live-server.img or live-server.iso + - Image for booting to server command prompt to preview or install the OS. .. list-table:: Table 2: Types of platform-specific |CL| images :widths: 15, 85 @@ -51,20 +48,23 @@ Table 2 lists the currently available images that are platform specific. * - Image Type - Description + * - aws.img + - Image suitable for use with Amazon\* AWS\*. + * - azure.vhd - - Virtual Hard Disk for use on Microsoft\* Azure\* cloud platform + - Virtual Hard Disk for use on Microsoft\* Azure\* cloud platform. * - azure-docker.vhd - - Virtual Hard Disk for use on Microsoft Azure cloud platform with Docker\* pre-installed + - Virtual Hard Disk for use on Microsoft Azure cloud platform with Docker\* pre-installed. * - azure-machine-learning.vhd - - Virtual Hard Disk for use on Microsoft Azure cloud platform with the `machine-learning-basic` bundle installed + - Virtual Hard Disk for use on Microsoft Azure cloud platform with the `machine-learning-basic` bundle installed. - * - cloud.img - - Image for use by cloud deployments such as OpenStack\* + * - cloudguest.img + - Image with generic cloud guest virtual machine (VM) requirements installed. - * - containers.img - - Image for use by Clear Containers runtime. Includes `optimized kernel`_ for Clear Containers. + * - gce.tar + - Image with the Google Compute Engine (GCE) specific kernel. * - hyperv.vhdx - Virtual Hard Disk for use with Microsoft Hyper-V\* hypervisor. Includes `optimized kernel`_ for Hyper-V. @@ -73,9 +73,15 @@ Table 2 lists the currently available images that are platform specific. - Image for booting in a simple VM with start_qemu.sh. Includes `optimized kernel`_ for KVM. + * - kvm-legacy.img + - Image for booting in a simple VM using legacy BIOS, if using start_qemu.sh make sure to remove -bios parameter. + + * - pxe.tar + - Image suitable for use with PXE server. + * - vmware.vmdk - Virtual Machine Disk for VMware\* platforms inclduing Player, Workstation, and ESXi. -.. _images: https://cdn.download.clearlinux.org/image +.. _images: https://clearlinux.org/downloads .. _`optimized kernel`: https://clearlinux.org/documentation/clear-linux/reference/compatible-kernels diff --git a/source/tooling/autoproxy.rst b/source/tooling/autoproxy.rst index 129cae51..3b850229 100644 --- a/source/tooling/autoproxy.rst +++ b/source/tooling/autoproxy.rst @@ -20,22 +20,22 @@ manually configure the proxies. Corporate and private networks can be very complex, needing to restrict and control network connections for security reasons. The typical side effects -are limited or blocked connectivity and requiring manual configuration of -proxies to perform the most mundane tasks such as cloning a repo or checking +are limited or blocked connectivity, and require manual configuration of +proxies to perform the most mundane tasks, such as cloning a repo or checking for updates. With |CL|, all of the work is done behind the scenes to effortlessly use your network and have connections “just work”. -This feature removes massive complications in network connectivity due to -proxy issues. You can automate tasks like unit testing without worrying -about the proxy not being set and you can remove unset proxies from the +This feature removes severe complications with network connectivity due to +proxy issues. You can automate tasks, such as unit testing, without worrying +about the proxy not being set, and you can remove unset proxies from the equation when dealing with network unavailability across systems. How it works ************ -We designed Autoproxy around tools provided by most Linux +We designed Autoproxy around tools provided by most Linux\* distributions with a few minor additions and modifications. We leveraged the -DHCP and network information provided from systemd and created a +DHCP and network information obtained from systemd and created a PAC-discovery daemon. The daemon uses the information to resolve a URL for a PAC file. The daemon then passes the URL into PACrunner\*. PACrunner downloads the PAC file and uses the newly implemented Duktape\* engine to @@ -49,7 +49,7 @@ parse it. From that point on, any cURL\* or network requests query PACrunner for the correct proxy to use. We modified the cURL library to communicate with PACrunner over DBus. However, cURL will ignore PACrunner and run normally if -no PAC file is loaded or if you set any proxies manually. Thus, your +no PAC file is loaded or if you manually set any proxies. Thus, your environment settings are respected and no time is wasted trying to resolve a proxy. All these steps happen in the background with no user interaction. diff --git a/source/tooling/autospec.rst b/source/tooling/autospec.rst index f07252f2..000bb9e8 100644 --- a/source/tooling/autospec.rst +++ b/source/tooling/autospec.rst @@ -3,8 +3,8 @@ autospec ######## -**autospec** is a tool to assist in the automated creation and maintenance of -RPM packaging in |CL-ATTR|. Where a standard RPM build process using +**autospec** is a tool used to assist with the automated creation and maintenance of +RPM packaging in |CL-ATTR|. Where a standard :abbr:`RPM (RPM Package Manager)` build process using :command:`rpmbuild` requires a tarball and :file:`.spec` file to start, autospec requires only a tarball and package name to start. @@ -16,17 +16,17 @@ Description *********** The autospec tool attempts to infer the requirements of the :file:`.spec` file -by analyzing the source code and :file:`Makefile` information. It will -continuously run updated builds based on new information discovered from build +by analyzing the source code and :file:`Makefile` information. It +continuously runs updated builds based on new information discovered from build failures until it has a complete and valid :file:`.spec` file. If needed, you can influence the behavior of autospec and customize the build by providing optional `control files`_ to the autospec tool. -autospec uses mock as a sandbox to run the builds. Visit the `mock wiki`_ for +autospec uses **mock** as a sandbox to run the builds. Visit the `mock wiki`_ for additional information on using mock. -For a general understanding of how RPMs work, visit the `rpm website`_ or the -`RPM Packaging Guide`_ . +For a general understanding of how an RPM works, visit +the `rpm website`_ or the `RPM Packaging Guide`_ . How it works ************ @@ -44,7 +44,7 @@ The setup for building source in |CL| must be completed before using the autospec tool. Refer to `Setup environment to build source`_ for instructions on completing -setup. +the setup. Create an RPM ============= @@ -52,7 +52,7 @@ Create an RPM The basic autospec process is described in the following steps: #. The :command:`make autospec` command generates a :file:`.spec` file based on - analysis of code and existing control files. + the analysis of code and existing control files. Any control files should be located in the same directory as the resulting :file:`.spec` file. @@ -65,13 +65,13 @@ The basic autospec process is described in the following steps: #. autospec detects any missed declarations in the :file:`.spec`. -#. If build errors occur, autospec will scan the build log to try and detect +#. If build errors occur, autospec scans the build log to try to detect the root cause. -#. If autospec detects the root cause and knows how to continue, it will restart +#. If autospec detects the root cause and knows how to continue, it restarts the build automatically at step 1 with updated build instructions. -#. Otherwise, autospec will stop the build for user inspection to resolve the +#. Otherwise, autospec stops the build for user inspection to resolve the errors. Respond to the build process output by fixing source code issues and/or editing control files to resolve issues, which may include dependencies or exclusions. See `autospec README`_ for more information on @@ -79,7 +79,7 @@ The basic autospec process is described in the following steps: The user resumes the process at step 1 after errors are resolved. - If a binary dependency doesn't exist in |CL|, you will need to build it + If a binary dependency doesn't exist in |CL|, you must build it before running autospec again. Following these steps, autospec continues to rebuild the package, based on @@ -95,10 +95,10 @@ Complete `Setup environment to build source`_ before using these examples. :local: :depth: 1 -Example 1: Build RPM with existing spec file -============================================ +Example 1: Build RPM with an existing spec file +=============================================== -This example shows how to build a RPM from a pre-packaged upstream package, with +This example shows how to build a RPM from a pre-packaged upstream package with an existing spec file. The example uses the ``dmidecode`` package. #. Navigate to the autospec workspace and clone the ``dmidecode`` package: @@ -110,7 +110,7 @@ an existing spec file. The example uses the ``dmidecode`` package. .. note:: - You can clone all package repos at once using: + You can clone all package repos at once using the following command: .. code-block:: bash @@ -147,7 +147,7 @@ create a simple helloclear RPM. cd ~/clearlinux make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear" - The resulting RPMs are in :file:`./packages/helloclear/rpms`. Builde logs and + The resulting RPMs are in :file:`./packages/helloclear/rpms`. Build logs and additional RPMs are in :file:`./packages/helloclear/results`. Example 3: Generate a new spec file with a pre-defined package @@ -198,8 +198,8 @@ Example 4: Provide control files to autospec ============================================ This example shows how to modify control files to correct build failures that -autospec is unable to resolve. In this example you will add a missing license -and dependencies in order for autospec to complete a successful build. +autospec is unable to resolve. In this example, you will add a missing license +and dependencies so autospec can complete a successful build. #. Navigate to the autospec workspace: @@ -226,7 +226,7 @@ and dependencies in order for autospec to complete a successful build. make autospecnew URL="https://github.com/OPAE/opae-sdk/archive/0.13.0.tar.gz" NAME="opae-sdk" - This will give an error for a missing license file: + This results in an error for a missing license file: .. code-block:: console @@ -238,7 +238,7 @@ and dependencies in order for autospec to complete a successful build. cd packages/opae-sdk -#. Add one or more valid license identifier from the +#. Add one or more valid license identifiers from the `SPDX License List `_. In the example below, two different licenses are appropriate based on the opae-sdk project licensing: @@ -253,7 +253,7 @@ and dependencies in order for autospec to complete a successful build. make autospec - This will result in a generic error: + This results in a generic error: .. code-block:: console @@ -265,7 +265,7 @@ and dependencies in order for autospec to complete a successful build. cat ./results/build.log - In the build log, you will find details for the specific failures. In this + The build log contains details for the specific failures. In this instance, there are missing dependencies: .. code-block:: console @@ -279,7 +279,7 @@ and dependencies in order for autospec to complete a successful build. linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae #. Search the spec files of upstream |CL| packages to see if the json-c library - is availabe. In this case, it does exist and we'll add the json-c 'dev' + is available. In this case, it does exist and we'll add the json-c 'dev' package into the buildreq_add: .. code-block:: bash @@ -302,7 +302,7 @@ and dependencies in order for autospec to complete a successful build. grep 'libuuid\.so$' ~/clearlinux/packages/*/*.spec echo "util-linux-dev" >> buildreq_add -#. Run autospec again and find the successfully-generated RPMs in the rpms +#. Run autospec again and find the successfully-generated RPMs in the :file:`rpms` directory: .. code-block:: bash @@ -384,7 +384,7 @@ To test an autospec-created package inside a VM: The code that makes this possible can be viewed by searching for the *install:* target in the `Makefile.common file on GitHub`_. -#. Return back to the :file:`~/clearlinux` directory and start the |CL| VM: +#. Return to the :file:`~/clearlinux` directory and start the |CL| VM: .. code-block:: bash @@ -411,7 +411,7 @@ Test directly on a development machine The |CL| development tooling also includes a method to extract autospec-created RPMs locally onto a |CL| development system for testing. -Extracting an RPM directly onto a system offers quicker testing, however +Extracting an RPM directly onto a system offers quicker testing; however conflicts may occur and responsibility to remove the software after testing is up to the developer. @@ -456,7 +456,7 @@ automated for you with a setup script. It uses tools from the The setup script creates a workspace in the :file:`clearlinux` folder, with the subfolders :file:`Makefile`, :file:`packages`, and :file:`projects`. The :file:`projects` folder contains the main tools used for making packages in -|CL|: `autospec` and `common`. +|CL| :file:`autospec` and :file:`common`. Follow these steps to setup the workspace and tooling for building source: diff --git a/source/tooling/ister.rst b/source/tooling/ister.rst index 4f29229a..bc8e399e 100644 --- a/source/tooling/ister.rst +++ b/source/tooling/ister.rst @@ -3,7 +3,7 @@ ister.py image builder ###################### -The `ister.py tool`_ is a template based installer used by |CL-ATTR| to produce +The `ister.py tool`_ is a template-based installer used by |CL-ATTR| to produce images for each release. The same ister tool is available for use in |CL| to create custom images based on an upstream image. @@ -14,21 +14,21 @@ create custom images based on an upstream image. Description *********** -|CL| is a rolling release and produces on average 10 releases per week using the +|CL| is a rolling release and produces an average of 10 releases per week using the ister tool. With each release we produce multiple -`image types for different environments`_ and use cases such as installers, -Hyper-V, KVM, or VMWare. +:ref:`image types for different environments ` and use cases such +as installers, Hyper-V, KVM, or VMWare. -Each image has a JSON configuration file used by ister to generate the image. -These JSON configuration files describe the image type, partitions, version, -and which bundles will be preinstalled by default with the image. For each image -type we produce, the corresponding JSON configuration file for the image is also +Each image has a JSON configuration file that is used by ister to generate the +image. These JSON configuration files describe the image type, partitions, version, +and bundles that will be preinstalled by default with the image. For each image +type we produce, the corresponding JSON configuration file for the image also is published. The :ref:`mixer` tool also uses ister to build images for your custom mix. Like upstream images, a JSON configuration file is defined for the image, which ister uses to generate the image. Refer to the :ref:`mixer` guide -for instruction on using ister to build an image for a custom mix. +for instructions on using ister to build an image for a custom mix. Examples ******** @@ -37,7 +37,7 @@ Recreate an upstream image ========================== The published configuration files for upstream images may be used to recreate an -image, for example when you want to: +image. Here are some examples: * Use an older version of |CL| and the image is no longer available (only after March 2017). @@ -64,7 +64,7 @@ configuration file: :file:`/clear/config/image`. For example: ``https://cdn.download.clearlinux.org/releases/15700/clear/config/image/`` -#. Download “PostNonChroot” script (if applicable). +#. Download the “PostNonChroot” script (if applicable). The JSON configuration file for the image may have an accompanying “PostNonChroot” script that is executed at the end of the image creation @@ -88,7 +88,6 @@ Related topics * :ref:`bulk-provision` .. _ister.py tool: https://github.com/bryteise/ister -.. _image types for different environments: https://cdn.download.clearlinux.org/image/README-IMAGES.html .. _Configuration files for the current release: https://cdn.download.clearlinux.org/current/config/image/ .. _Previous releases: https://cdn.download.clearlinux.org/releases/ .. _Install a bundle: https://clearlinux.org/documentation/clear-linux/guides/maintenance/swupd-guide#adding-a-bundle \ No newline at end of file diff --git a/source/tooling/swupd-guide.rst b/source/tooling/swupd-guide.rst index 69fc78bb..dedd7499 100644 --- a/source/tooling/swupd-guide.rst +++ b/source/tooling/swupd-guide.rst @@ -8,77 +8,68 @@ software. .. contents:: :local: - :depth: 2 + :depth: 1 Description *********** :command:`swupd` has two main functions: -#. It manages software replacing APT or YUM, installing bundles +#. Manages software and replaces APT or YUM, by installing bundles rather than packages. -#. It checks for system updates and installs them. +#. Checks for system updates and installs them. -:ref:`Bundles ` are the smallest granularity component that is -managed by |CL| and contain everything needed to deliver a software -capability. Rather than downloading a cascade of package dependencies when -installing a piece of software, a bundle comes with all of its dependencies. -:command:`swupd` manages overlapping dependencies behind the scenes ensuring -that all software is compatible across the system. +:command:`swupd` manages overlapping dependencies behind the scenes, ensuring +that all software is compatible across the system. It can be used to verify +the OS, clean cached files, and fix issues. + +:ref:`Bundles ` contain everything needed to deliver a software +capability. They are the smallest granularity component that is +managed by |CL|. A bundle comes with all of its dependencies, rather than +downloading a cascade of package dependencies when installing a piece of +software. Versioning ========== -In a traditional distribution, the process of describing current software -versioning usually involves: +Using package managers to keep track of software version compatibility or +compare multiple systems on many Linux distributions can be cumbersome. -- Listing and keeping track of the current OS release (generally - uninformative about any singular packages or functionality). +With |CL| :command:`swupd`, versioning happens at the individual file-level. +This means |CL| generates an entirely new OS version with any set of software +changes to the system (including software downgrades or removals). This +rolling release versioning model is similar to :command:`git` internal version +tracking, where any of the individual file commits are tracked and move the +pointer forward when changed. -- Keeping track of packages and repositories being used, and updating them - individually. - -- Listing and tracking every package available and installed on the - system, none of which are directly tied to the current OS release. - -This can be done effectively, but given the nearly endless combinations of -packages and versions of packages a server may have, it quickly becomes -non-trivial to define what "version" the system is and what software it -is running without explicitly going through each system and inspecting -every package. - -With |CL|, we need track: - -- One number - -A number representing the **current** release of the OS is sufficient to -describe the versions of all the software on the OS. Each build is -composed of a specific set of bundles made from a particular version of -packages. This matters on a daily basis to system administrators, who -need to determine which of their systems do not have the latest security -fixes, or which combinations of software have been tested. Every release -of the same number is guaranteed to contain the same versions of software, -so there's no ambiguity between two systems running the same version of |CL|. +A number that represents the **current** release of the OS describes the +versions of all the software on the OS. Each build is composed of a specific +set of bundles made from a particular version of packages. This matters on a +daily basis to system administrators, who need to determine which of their +systems do not have the latest security fixes, or which combinations of +software have been tested. Every release of the same number is guaranteed to +contain the same versions of software, so there's no ambiguity between two +systems running the same version of |CL|. Updating ======== -|CL| enforces regular updating of the OS by default and will automatically -check for updates against a version server. The content server provides the +|CL| enforces regular updating of the OS by default and automatically +checks for updates against a version server. The content server provides the file and metadata content for all versions and can be the same as the version server. The content url server provides metadata in the form of -manifests. These Manifest files list and describe file contents, symlinks, +*manifests*, which list and describe file contents, symlinks, and directories. Additionally, the actual content is provided to clients in the form of archive files. Software updates with |CL| are also efficient. Unlike package-based -distributions, :command:`swupd` only updates files that have changed rather +distributions, :command:`swupd` only updates files that have changed, rather than entire packages. For example, it is quite common for an OS security -patch to be as small as 15 KB. Using binary deltas, the |CL| is able to +patch to be as small as 15 KB. Using binary deltas, |CL| is able to apply only what is needed. -To get a more detailed understanding of how to generate update content for -|CL| see the :ref:`mixer ` tool. +For details on how to generate update content for |CL|, see the +:ref:`mixer ` tool. How it works ************ @@ -93,42 +84,41 @@ Prerequisites Updates ======= -|CL| updates are automatic by default but can be set to occur only on +|CL| updates are automatic by default, but can be set to occur only on demand. :command:`swupd` makes sure that regular updates are simple and secure. It can also check the validity of currently installed files and -software and correct any problems. +software, and can correct any problems. Manifests --------- -The Clear Linux OS software update content consists of data and -metadata. The data is the files that end up in the OS. The metadata -contains relevant information to properly provision the data to the OS -file system, as well as update the system and add or remove additional -content to the OS. +The |CL| software update content consists of data and metadata. The data is +the files that end up in the OS. The metadata contains relevant information to +properly provision the data to the OS file system, as well as update the +system and add or remove additional content to the OS. -The Manifests are mostly long lists of hashes that describe content. +The manifests are mostly long lists of hashes that describe content. Each bundle gets its own manifest file. There is a master manifest file that describes all manifests to tie it all together. Fullfiles, packs, and delta packs --------------------------------- -The data that an update provisions to a system can be obtained in -three different ways. There are three different methods, and they -exist to optimize the delivery of content and speed up updates. +To speed up updates and optimize content delivery, update data provisioned to +a system is obtained by one of the following methods: -Fullfiles are always generated for every file in every release. This -allows any Clear Linux OS to obtain the exact copy of the content -for each version directly. This would be used if the OS verification -needed to replace a single file, for instance. +* *Fullfiles* are always generated for every file in every release. This + allows any |CL| to obtain the exact copy of the content + for each version directly. This is used if the OS verification + needs to replace a single file, for instance. -Packs are available for some releases and combine many files to speed -up the creation of installation media and large updates. Delta packs -are an optimized version of packs that only contain updates (binary -diffs) and cannot be used without having the original file content. +* *Packs* are available for some releases. They combine many files to speed + up the creation of installation media and large updates. -Bundle Search +* *Delta packs* are an optimized version of packs that only contain updates + (binary diffs). They cannot be used without having the original file content. + +Bundle search ============= :command:`swupd` search downloads manifest data and searches for @@ -139,21 +129,21 @@ Only the base bundle is returned. Bundles can contain other bundles via includes. For more details, see `Bundle Definition Files`_ and its subdirectory bundles. -Bundles that are already installed, will be marked [installed] in search +Bundles that are already installed are marked **(installed)** in search results. -Optionally, you can review our `bundles`_ on GitHub. +Optionally, you can review our `bundles`_ on GitHub\*. Examples ******** -Example 1: Disable and Enable automatic updates +Example 1: Disable and enable automatic updates =============================================== -|CL| updates are automatic by default but can be set to occur only +|CL| updates are automatic by default, but can be set to occur only on demand. -#. First verify your current auto-update setting. +#. Verify your current auto-update setting. .. code-block:: bash @@ -202,7 +192,7 @@ Example 2: Find and install Kata\* Containers Kata Containers is a popular container implementation. Unlike other container implementations, each Kata Container has its own -kernel instance and runs on its own :abbr:`Virtual Machine (VM)` for +kernel instance and runs on its own :abbr:`VM (Virtual Machine)` for improved security. |CL| makes it very easy to install, since you only need to add @@ -211,34 +201,33 @@ number of dependencies. Also, check out our tutorial: :ref:`kata`. #. Find the right bundle. - * To return all possible matches for the search string enter - :command:`swupd search`, followed by 'kata': + To return all possible matches for the search string, enter + :command:`swupd search`, followed by 'kata': - .. code-block:: bash + .. code-block:: bash - sudo swupd search kata + sudo swupd search kata - The output should be similar to: + The output should be similar to: - .. code-block:: console + .. code-block:: console - Bundle with the best search result: + Bundle with the best search result: - containers-virt - Run container applications from Dockerhub in lightweight virtual machines + containers-virt - Run container applications from Dockerhub in lightweight virtual machines - This bundle can be installed with: + This bundle can be installed with: - swupd bundle-add containers-virt + swupd bundle-add containers-virt - Alternative bundle options are + Alternative bundle options are - cloud-native-basic - Contains ClearLinux native software for Cloud + cloud-native-basic - Contains ClearLinux native software for Cloud - .. note:: + .. note:: - If your search does not produce results with a specific - term, shorten the search term. For example, use *kube* instead of - *kubernetes*. + If your search does not produce results with a specific term, shorten + the search term. For example, use *kube* instead of *kubernetes*. #. Add the bundle. @@ -248,7 +237,7 @@ number of dependencies. Also, check out our tutorial: :ref:`kata`. .. note:: - To add multiple bundles simply add a space followed by the bundle name. + To add multiple bundles, add a space followed by the bundle name. The output of a successful installation should be similar to: @@ -279,7 +268,7 @@ been added to, overwritten, removed, or modified (e.g., permissions). sudo swupd diagnose All directories that are watched by :command:`swupd` are verified according -to the manifest data and hash mismatches are flagged as follows: +to the manifest data. Hash mismatches are flagged as follows: .. code-block:: console @@ -295,47 +284,49 @@ to the manifest data and hash mismatches are flagged as follows: 423 files did not match Verify successful -In this case, python packages that were installed on top of the default +In this case, Python\* packages that were installed on top of the default install were flagged as mismatched. :command:`swupd` can be directed to ignore or fix issues based on command line options. :command:`swupd` can correct any issues it detects. Additional directives -can be added including a white list of directories that will be ignored. +can be added including a white list of directories to be ignored. -The following command will repair issues, remove unknown items, and -ignore files or directories matching `/usr/lib/python`: +The following command repairs issues, removes unknown items, and +ignores files or directories matching :file:`/usr/lib/python`: .. code-block:: bash sudo swupd repair --picky --picky-whitelist=/usr/lib/python -Quick Reference +Quick reference *************** swupd info - To see the currently installed version and update servers. + Returns the currently installed version and update servers. swupd update - To update to a specific version or with no arguments to update to latest. + Updates to a specific version or updates to latest version if no + arguments are used. swupd bundle-list [--all] - To list installed bundles. + Lists installed bundles. swupd bundle-add [-b] - To find a bundle that contains your search term. + Finds a bundle that contains your search term. swupd bundle-add - To add a bundle. + Adds a bundle. swupd bundle-remove - To remove a bundle. + Removes a bundle. swupd --help - For additional :command:`swupd` commands. + Lists additional :command:`swupd` commands. man swupd - To reference the :command:`swupd` man page, or see the - `source documentation`_ available on github. + Opens the :command:`swupd` man page. + +Refer to :command:`swupd` `source documentation`_ on GitHub for more details. Related topics ************** diff --git a/source/tutorials/dlrs/dlrs.rst b/source/tutorials/dlrs/dlrs.rst index 7d51ba4b..3098432c 100644 --- a/source/tutorials/dlrs/dlrs.rst +++ b/source/tutorials/dlrs/dlrs.rst @@ -6,7 +6,6 @@ Deep Learning Reference Stack This tutorial describes how to run benchmarking workloads for TensorFlow\*, PyTorch\*, and Kubeflow in |CL-ATTR| using the Deep Learning Reference Stack. - .. contents:: :local: :depth: 1 @@ -26,14 +25,14 @@ The Deep Learning Reference Stack is available in the following versions: for Deep Neural Networks (Intel® MKL-DNN) primitives and introduces support for Intel® AVX-512 Vector Neural Network Instructions (VNNI). * `Intel MKL-DNN`_, which includes the TensorFlow framework optimized using - Intel® Math Kernel Library for Deep Neural Networks (Intel® MKL-DNN) primitives. + Intel® Math Kernel Library for Deep Neural Networks (Intel® MKL-DNN) + primitives. * `Eigen`_, which includes `TensorFlow`_ optimized for Intel® architecture. * `PyTorch with OpenBLAS`_, which includes PyTorch with OpenBlas. * `PyTorch with Intel MKL-DNN`_, which includes PyTorch optimized using Intel® Math Kernel Library (Intel® MKL) and Intel MKL-DNN. - -.. note:: +.. important:: To take advantage of the Intel® AVX-512 and VNNI functionality with the Deep Learning Reference Stack, you must use the following hardware: @@ -41,14 +40,16 @@ The Deep Learning Reference Stack is available in the following versions: * Intel® AVX-512 images require an Intel® Xeon® Scalable Platform * VNNI requires a 2nd generation Intel® Xeon® Scalable Platform - Stack features ============== * Deep Learning Reference Stack `V3.0 release announcement`_. -* Deep Learning Reference Stack v2.0 including current `PyTorch benchmark results`_. -* Deep Learning Reference Stack v1.0 including current `TensorFlow benchmark results`_. -* `Release notes on Github\*`_ for the latest release of Deep Learning Reference Stack. +* Deep Learning Reference Stack v2.0 including current + `PyTorch benchmark results`_. +* Deep Learning Reference Stack v1.0 including current + `TensorFlow benchmark results`_. +* `Release notes on Github\*`_ for the latest release of Deep Learning Reference + Stack. .. note:: @@ -58,19 +59,21 @@ Stack features Prerequisites ============= -* :ref:`Install ` |CL| on your host system. +* :ref:`Install ` |CL| on your host system * :command:`containers-basic` bundle * :command:`cloud-native-basic` bundle In |CL|, :command:`containers-basic` includes Docker\*, which is required for TensorFlow and PyTorch benchmarking. Use the :command:`swupd` utility to -check if :command:`containers-basic` and :command:`cloud-native-basic` are present: +check if :command:`containers-basic` and :command:`cloud-native-basic` are +present: .. code-block:: bash sudo swupd bundle-list -To install the :command:`containers-basic` or :command:`cloud-native-basic` bundles, enter: +To install the :command:`containers-basic` or :command:`cloud-native-basic` +bundles, enter: .. code-block:: bash @@ -91,7 +94,7 @@ Version compatibility We validated these steps against the following software package versions: -* |CL| 26240 (Lower version not supported.) +* |CL| 26240 (Lower version not supported) * Docker 18.06.1 * Kubernetes 1.11.3 * Go 1.11.12 @@ -137,8 +140,8 @@ TensorFlow. You can replace the model with one of your choice supported by the TensorFlow benchmarks. - If you are using an FP32 based model, it can be converted to an int8 model using `Intel® quantization tools`_ - + If you are using an FP32 based model, it can be converted to an int8 model + using `Intel® quantization tools`_. PyTorch single and multi-node benchmarks **************************************** @@ -182,6 +185,11 @@ Kubeflow multi-node benchmarks The benchmark workload runs in a Kubernetes cluster. The tutorial uses `Kubeflow`_ for the Machine Learning workload deployment on three nodes. +.. warning:: + + If you choose the Intel® MKL-DNN or Intel® MKL-DNN-VNNI image, your platform must support the Intel® AVX-512 instruction set. Otherwise, an *illegal instruction* error may appear, and you won’t be able to complete this tutorial. + + Kubernetes setup ================ @@ -196,12 +204,48 @@ We used `flannel`_ as the network provider for these tests. If you prefer a different network layer, refer to the Kubernetes `networking documentation`_ for setup. +Kubectl +======= + +You can use kubectl to run commands against your Kubernetes cluster. Refer to +the `kubectl overview`_ for details on syntax and operations. Once you have a +working cluster on Kubernetes, use the following YAML script to start a pod with +a simple shell script, and keep the pod open. + +#. Copy this example.yaml script to your system: + + .. code-block:: console + + apiVersion: v1 + kind: Pod + metadata: + name: example-pod + labels: + app: ex-pod + spec: + containers: + - name: ex-pod-container + image: clearlinux/stacks-dlrs-mkl:latest + command: ['/bin/bash', '-c', '--'] + args: [ "while true; do sleep 30; done" ] + +#. Execute the script with kubectl: + + .. code-block:: bash + + kubectl apply –f /example.yaml + +This script opens a single pod. More robust solutions would create a deployment +or inject a python script or larger shell script into the container. + Images ====== -You must add `launcher.py` to the Docker image to include the Deep +You must add `launcher.py`_ to the Docker image to include the Deep Learning Reference Stack and put the benchmarks repo in the correct -location. From the Docker image, run the following: +location. Note that this tutorial uses Kubeflow v0.4.0, and cannot guarantee results if you use a different version. + +From the Docker image, run the following: .. code-block:: bash @@ -210,7 +254,7 @@ location. From the Docker image, run the following: cp launcher.py /opt chmod u+x /opt/* -Your entry point becomes: :file:`/opt/launcher.py` +Your entry point becomes: :file:`/opt/launcher.py`. This builds an image that can be consumed directly by TFJob from Kubeflow. @@ -220,9 +264,9 @@ ksonnet\* Kubeflow uses ksonnet\* to manage deployments, so you must install it before setting up Kubeflow. -ksonnet was added to the :command:`cloud-native-basic` bundle in |CL| version 27550. If -you are using an older |CL| version (not recommended), you must manually -install ksonnet as described below. +ksonnet was added to the :command:`cloud-native-basic` bundle in |CL| version +27550. If you are using an older |CL| version (not recommended), you must +manually install ksonnet as described below. On |CL|, follow these steps: @@ -294,7 +338,8 @@ Run a TFJob Replace with the image name you specified in previous steps. -#. Generate Kubernetes manifests for the workloads and apply them using these commands: +#. Generate Kubernetes manifests for the workloads and apply them using these + commands: .. code-block:: bash @@ -372,7 +417,8 @@ A new, blank notebook is displayed, with a cell ready for input. :alt: New blank notebook -To verify that PyTorch is working, copy the following snippet into the blank cell, and run the cell. +To verify that PyTorch is working, copy the following snippet into the blank +cell, and run the cell. .. code-block:: console @@ -405,7 +451,6 @@ Related topics * :ref:`kubernetes` tutorial * `Jupyter Notebook`_ - .. _TensorFlow: https://www.tensorflow.org/ .. _Kubeflow: https://www.kubeflow.org/ @@ -446,6 +491,10 @@ Related topics .. _Jupyter Notebook: https://jupyter.org/ +.. _kubectl overview: https://kubernetes.io/docs/reference/kubectl/overview/ + +.. _launcher.py: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dlrs/kubeflow + .. _Release notes on Github\*: https://github.com/clearlinux/dockerfiles/blob/master/stacks/dlrs/releasenote.md .. _Intel® quantization tools: https://github.com/IntelAI/tools/blob/master/tensorflow_quantization/README.md#quantization-tools diff --git a/source/tutorials/tutorials.rst b/source/tutorials/tutorials.rst index aebbfd30..0d5300e9 100644 --- a/source/tutorials/tutorials.rst +++ b/source/tutorials/tutorials.rst @@ -28,3 +28,4 @@ Explore our tutorials to discover what you can do with |CL|! nvidia dars redis + tutorial-proxy