Merge branch 'master' of github.com:clearlinux/clear-linux-documentation into rtd-theme

Signed-off-by: Kevin Putnam <kevin.putnam@intel.com>
This commit is contained in:
Kevin Putnam
2019-07-02 10:38:18 -07:00
20 changed files with 1226 additions and 277 deletions
+189
View File
@@ -0,0 +1,189 @@
.. _faq:
Frequently Asked Questions (FAQ)
################################
Below is a list of commonly asked questions with answers sourced from the
|CL-ATTR| team and `Clear Linux community forums`_.
.. contents:: :local:
:depth: 2
General
*******
Why did you make another distro?
================================
The |CL| team felt that performance was left on the table with Linux software.
|CL| takes a holistic approach to improving performance across the stack. We
also wanted to take more modern approaches with OS updates and tooling.
|
Can other distros copy |CL| improvements?
=========================================
Yes, we absolutely love open source reuse and upstreaming improvements.
|
How often do you update?
========================
The |CL| team puts out multiple releases a week, often releasing 2 or more
times a day. This rolling release approach allows |CL| to remain agile to
upstream changes and security patches.
|
Is telemetry required?
======================
The telemetry solution provided by |CL| is entirely optional and customizable.
It is disabled by default. If you do choose to enable telemetry, the data
helps the |CL| team proactively identify and resolve bugs. See the
:ref:`telemetry <telemetry-about>` page for more information.
|
What is the default firewall?
=============================
|CL| packages :command:`iptables` as a bundle, however, there are no default
firewall rules. All network traffic is allowed by default.
|
Where are the files that I usually see under /etc like fstab?
=============================================================
|CL| has a stateless design that maintains a separation between system files
and user files. Default values are stored under :file:`/usr/share/defaults/`.
Files under :file:`/etc/` are not created unless you create one.
A blog post explaining how this is accomplished with :file:`/etc/fstab/`
specifically is available here:
https://clearlinux.org/news-blogs/where-etcfstab-clear-linux
|
Software packages
*****************
How is software installed and updated?
======================================
|CL| provides software in the form of :ref:`bundles <bundles-about>` and
updates software with :ref:`swupd <swupd-about>`.
:ref:`FlatPak\* <flatpak>` is an application virtualization solution that allows
more software to be available to |CL| users by augmenting the software |CL|
packages natively with software available through FlatPak.
Our goal is to have software packaged natively and made available through
bundles whenever possible.
|
Does |CL| use RPMs like other distros?
======================================
|CL| provides software in the form of :ref:`bundles <bundles-about>`. The RPM
format is used as an intermediary step for packaging and determining software
dependencies at OS build time.
Individual RPMs can sometimes be manually installed on a |CL| system with the
right tools, but that is not the intended use case.
|
Can I install a software package from another OS on |CL|?
=========================================================
Software that is packaged in other formats for other Linux distributions is
not guaranteed to work on |CL| and may be impacted by |CL| updates.
If the software you're seeking is open source, please submit a request to add
it to |CL|. Submit requests on GitHub\* here:
https://github.com/clearlinux/distribution/issues
|
Software availability
*********************
What software is available on |CL|?
===================================
Available software can be found in the `Software Store`_, through the GNOME\*
Software application on the desktop, or by using :ref:`swupd search <bundle-commands>`.
|
Is Google\* Chrome\* available?
===============================
The Google Chrome web browser is not distributed as a bundle in |CL| due to
copyright and licensing complexities.
A discussion on manually installing and maintaining Google Chrome can be found
on GitHub: https://github.com/clearlinux/distribution/issues/422
|
Is FFmpeg available?
====================
`FFmpeg`_ is a multimedia software suite, which is commonly used for
various media encoding/decoding, streaming, and playback.
|CL| does not distribute FFmpeg due to well-known licensing and legal
complexities (See https://www.ffmpeg.org/legal.html and
http://blog.pkh.me/p/13-the-ffmpeg-libav-situation.html).
Read more in the |CL| repository, including discussion of an alternative
hardware-based solution:
https://github.com/clearlinux/distribution/issues/429.
While |CL| cannot distribute FFmpeg, a manual solution to build and install
FFmpeg under :file:`/usr/local` has been shared on the community forums:
https://community.clearlinux.org/t/how-to-h264-etc-support-for-firefox-including-ffmpeg-install.
|
Is ZFS\* available?
===================
ZFS is not available with |CL| because of copyright and licensing
complexities. BTRFS is an alternative filesystem that is available in |CL|
natively.
A user on GitHub notes that the ZFS kernel module can be compiled, built, and
installed manually: https://github.com/clearlinux/distribution/issues/631
|
Can you add a driver that I need?
=================================
If a kernel module is available as part of the Linux kernel source tree but
not enabled in the |CL| kernels, in many cases the |CL| team will enable it
upon request. Submit requests on GitHub here:
https://github.com/clearlinux/distribution/issues
The |CL| team does not typically add out-of-tree kernel modules as a matter of
practice because of the maintenance overhead. If the driver was unable to be
merged upstream, there is a good chance we may be unable to merge it for
similar reasons.
Kernel modules can be individually built and installed on |CL|. See the
:ref:`kernel modules <kernel-modules>` page for more information.
|
.. _`Clear Linux community forums`: https://community.clearlinux.org
.. _`Software Store`: https://clearlinux.org/software
.. _`FFmpeg`: https://ffmpeg.org/
Binary file not shown.

After

Width:  |  Height:  |  Size: 94 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 88 KiB

+145
View File
@@ -0,0 +1,145 @@
.. _stateless:
Stateless
#########
In most operating systems, files can become intermingled with user and system
data and configurations.
.. figure:: figures/stateless-1.png
:scale: 45%
:align: center
:alt: Stateless: User and system files mixed
Figure 1: Without stateless, user and system files become mixed on the filesystem over time.
|CL-ATTR| has a stateless design philosophy of which the goal is to provide an
:abbr:`OS (operating system)` that functions without excessive user
configuration or customization. Stateless in this context does *not* mean
ephemeral or non-persistent.
.. contents:: :local:
:depth: 2
File-level separation
*********************
To accomplish a stateless design the Linux Filesystem Hierarchy is separated
between user-owned areas and |CL|-owned areas.
.. figure:: figures/stateless-2.png
:scale: 45%
:align: center
:alt: Stateless: User and system files separation
Figure 2: With stateless, user and system files are separated on the filesystem.
System areas
============
File under the :file:`/usr` directory are managed by |CL| as system files.
Files written under the :file:`/usr` directory by users can get removed
through system updates with :ref:`swupd <swupd-about>`.This operating
assumption allows |CL| to verify and maintain integrity of system files.
User areas
==========
Files under the :file:`/etc/`, :file:`/home`, and :file:`/var` directories are
owned and managed by the user. A freshly installed |CL| system will only have
a minimal set of files in the :file:`/etc/` directory and software installed
by |CL| does not write to :file:`/etc`. This operating assumption allows |CL|
users to clearly identify the configuration that makes their system unique.
Software Configuration
**********************
With stateless separation, default software configurations are read in order
from predefined source code, |CL| provided defaults, and user-provided
configuration.
Default configurations
======================
Software in |CL| provides default configuration values so that it is
immediately functional, whenever it is appropriate to do so.
|CL| distributed software packages may be directly modified to include default
configuration values or default configuration files may be provided by |CL|
under :file:`/usr/share/defaults`. These files can be referenced as templates
for customization.
For example, the default configuration that Apache uses when installed can be
found at :file:`/usr/share/defaults/httpd/httpd.conf` directory.
Overriding configurations
=========================
If a configuration needs to be changed, the appropriate file should be
modified by the user under :file:`/etc/`. If the configuration file does not
already exist, it can be created in the appropriate location.
User defined configuration files should contain the minimal set of desired
changes and rely on default configuration for the rest.
For example, a customized Apache configuration can be used instead by:
#. Create the destination directory for the configuration:
.. code :: bash
sudo mkdir /etc/httpd
#. Copy the default configuration as a reference template:
.. code :: bash
sudo cp /usr/share/defaults/httpd/httpd.conf /etc/httpd/
#. Make any desired modifications to the configurations:
.. code :: bash
sudoedit /etc/httpd/httpd.conf
#. Reload the service or reboot the system to pickup any changes:
.. code :: bash
systemctl daemon-reload httpd && systemctl restart httpd
This pattern can be used to modify the configurations of other programs too.
The `stateless man page`_ has application-specific examples.
System reset
************
Once advantage of the stateless design is that the system defaults can be
easily restored by simply deleting everything under :file:`/etc/` and
:file:`/var`.
Running the commands below effectively performs a system reset as if it was
just installed:
.. code::
sudo rm -rf /etc
sudo rm -rf /var
In other Linux distributions, this can be a catastrophic action that renders a
system unable to boot.
Additional information
**********************
* `stateless man page`_
* `Where is /etc/fstab in Clear Linux? <https://clearlinux.org/news-blogs/where-etcfstab-clear-linux>`_
.. _`stateless man page`: https://github.com/clearlinux/clr-man-pages/blob/master/stateless.7.rst
@@ -0,0 +1,196 @@
.. _cpu-performance:
CPU Power and Performance
#########################
Modern x86 :abbr:`CPUs (central processing units)` employ a number of features
and technologies to balance performance, energy, and thermal efficiencies.
By default, |CL| prioritizes maximum CPU performance with the philosophy that
the faster the program finishes execution, the faster the CPU can return to a
low energy idle state. It is important to understand and evaluate all of these
technologies when troubleshooting or considering changing the defaults.
.. contents:: :local:
:depth: 1
CPU power saving mechanisms
===========================
C-states and P-states are both CPU power saving mechanisms that are entered
under different operating conditions. The tradeoff is a slightly longer time
to exit these states when the CPU is needed once again.
.. _c-states-section:
C-states (idle states)
----------------------
C-states are hardware sleep states that are entered when it is determined that
the CPU is idle and not executing instructions.
C-states aim to reduce power utilization by increasingly reducing clock
frequency, voltages, and features in each state.
Although C-states can typically be limited or disabled in a system's UEFI or
BIOS configuration, these settings are overridden when the `intel_idle
driver`_ is in use.
To view the current cpuidle driver run this command in a terminal:
.. code:: bash
cat /sys/devices/system/cpu/cpuidle/current_driver
For troubleshooting, C-states can be limited with a kernel command line boot
parameter by adding :command:`processor.max_cstate=N intel_idle.max_cstate=N`
or completely disabled with :command:`idle=poll`.
.. note::
* :command:`processor.max_cstate=0` is changed to :command:`processor.max_cstate=1` by the kernel to be a valid value.
* :command:`intel_idle.max_cstate=0` disables the Intel Idle driver, not set
it to C-state 0.
.. _p-states-section:
P-states (performance states)
-----------------------------
P-states, also known as *SpeedStep* on Intel processors or *Cool'n'Quiet* on
AMD processors, are states entered while the CPU is active and executing
instructions.
P-states aim to reduce power utilization by adjusting CPU clock frequency and
voltages based on CPU demand.
P-states can typically be limited or disabled in a system's firmware (UEFI/BIOS).
Turbo boost
~~~~~~~~~~~
`Turbo Boost technology`_, found on some modern Intel CPUs, allows core(s) on
a processor to temporarily operate at a higher than rated CPU clock frequency
to accommodate demanding workloads if the CPU is under defined power and
thermal thresholds.
Turbo boost is an extension of P-states. As such, changing or limiting
C-states or P-states impact the ability of a process to enter Turbo boost.
Turbo boost can be disabled in a system's UEFI or BIOS. Turbo boost can also
be disabled within |CL| with the command:
.. code:: bash
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
Linux CPU clock frequency scaling
=================================
The CPUFreq subsystem in Linux allows the OS to control :ref:`C-states
<c-states-section>` and :ref:`P-states <P-states-section>`
via CPU drivers and governors that provide algorithms that define how and when
to enter these states.
Scaling driver
--------------
Linux uses the `Intel P-state driver`_, *intel_pstate*, for modern Intel
processors from the Sandy Bridge generation or newer. Other processors may
default to the *acpi-cpufreq* driver which reads values from the systems UEFI
or BIOS.
To view the current CPU frequency scaling driver run this command in a terminal:
.. code:: bash
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_driver
Scaling governor
----------------
|CL| sets the CPU governor to *performance* which calls for the CPU to operate
at maximum clock frequency. In other words, P-state P0. While this may sound
wasteful at first, it is important to remember that power utilization does not
increase significantly simply because of a locked clock frequency without a
workload.
To view the current CPU frequency scaling governor run this command in a terminal:
.. code:: bash
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
To change the CPU frequency scaling governor:
#. Disable |CL| enforcement of certain power and performance settings:
.. code:: bash
sudo systemctl mask clr-power.timer
#. Change the governor. In the example below, the governor is set to
*performance*:
.. code:: bash
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
The list of all governors can be found in the `Linux kernel documentation on
CPUFreq Governors`_.
.. note::
The intel_pstate driver only supports *performance* and *powersave* governors.
Thermal management
==================
`thermald`_ is a Linux thermal management daemon used to prevent the
overheating of platforms. When temperature thresholds are exceeded, thermald
forces a C-state by inserting CPU sleep cycles and adjusts available cooling
methods. This can be especially desirable for laptops.
By default, thermald is disabled in |CL| and starts automatically if battery
power is detected. thermald can be manually enabled using the systemd service
by running the command:
.. code:: bash
sudo systemctl enable thermald
sudo systemctl start thermald
For more information, see the thermald man page:
.. code:: bash
man thermald
`ThermalMonitor`_ is a GUI application that can visually graph and log
temperatures from thermald. To use ThermalMonitor, add the desktop-apps-extras
bundle and add your user account to the power group:
.. code:: bash
sudo swupd bundle-add desktop-apps-extras
sudo usermod -a -G power <USER>
ThermalMonitor
.. note::
After adding a new group you must log out and log back in for the new group
to take affect.
.. _`Intel P-state driver`: https://www.kernel.org/doc/Documentation/cpu-freq/intel-pstate.txt
.. _`Linux kernel documentation on CPUFreq Governors`: https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt
.. _thermald: https://01.org/linux-thermal-daemon
.. _`intel_idle driver`: https://github.com/torvalds/linux/blob/master/drivers/idle/intel_idle.c
.. _`ThermalMonitor`: https://github.com/intel/thermal_daemon/tree/master/tools/thermal_monitor
.. _`Turbo Boost technology`: https://www.intel.com/content/www/us/en/architecture-and-technology/turbo-boost/turbo-boost-technology.html
@@ -0,0 +1,89 @@
.. _debug:
Debug System
############
|CL-ATTR| introduces a novel approach to system software debugging using
*clr-debug-info*. On the client side, the |CL| debug system obtains any
necessary debug information on-the-fly over a network during a debugging
session. On the server side, the system curates and compresses debug
information into small pieces for efficient downloading.
For developers, this avoids the interruption during debugging that usually
happens when debug information is missing. This can be especially useful on
systems where storage is limited.
.. contents:: :local:
:depth: 2
Background
----------
Software that is compiled and packaged for general usage in an operating
system typically only contains components that are used to execute the
program, like binaries and libraries. Extra developer data like the the
actual source code and symbol information are separated and excluded for
efficiency.
The debug information helps relate binary code to human readable source code
lines and variables. Most of the time this auxiliary information is not needed
however without it, debugging a program results in limited visibility.
Usage
-----
The clr-debug-info system is integrated into |CL| and seamlessly engages once
installed.
#. Install the *dev-utils* bundle.
.. code:: bash
sudo swupd bundle-add dev-utils
.. note::
The *telemetrics* and *performance-tools* bundles also include
clr-debug-info.
#. Start a debugging session against a program using a debugger, like GDB.
For example, to debug *gnome-control-center*:
.. code:: bash
gdb /usr/bin/gnome-control-center
As the program is stepped through and debug information is needed, the
clr_debug_daemon will obtain it in the background.
Implementation
--------------
The implementation of the |CL| debug system is open source and available on
GitHub at: https://github.com/clearlinux/clr-debug-info/
.. figure:: figures/debug-diagram.png
:scale: 75%
:alt: Debug system communication flow
Figure 1: The communication flow of the |CL| debug system
The |CL| debug system implements a :abbr:`FUSE (filesystem in userspace)`
filesystem mounted at :file:`/usr/lib/debug` and :file:`/usr/src/debug`. The
FUSE filesystem starts automatically. Its status can be verified with
:command:`systemctl status clr_debug_fuse.service`.
The *clr_debug_daemon* is responsible for fetching the appropriate package
debug content from the server and making it available for any debugging
programs needed it. It is socket activated whenever a request to the local
FUSE filesystem occurs. Its status can be verified with :command:`systemctl
status clr_debug_daemon.service`.
|CL| hosts debuginfo content packaged for consumption by |CL| debug clients at
https://download.clearlinux.org/debuginfo/
Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

@@ -0,0 +1,115 @@
.. _tutorial-proxy:
Setting up proxy
################
This tutorial shows you how to configure your system for use behind an
outbound proxy to access the Internet.
|CL| :ref:`tooling` applications already benefit from the :ref:`autoproxy`
feature.
.. contents::
:local:
:depth: 1
Prerequisites
*************
This tutorial assumes you have installed |CL| on your host system.
For detailed instructions on installing |CL| on a bare metal system, visit
the :ref:`bare metal installation guide <bare-metal-install-desktop>`.
Shells and programs in a desktop session
****************************************
Terminal
========
Add the following to your ~/.bashrc:
.. code-block:: bash
export http_proxy=http://your.http-proxy.url:port
export https_proxy=http://your.https-proxy.url:port
export ftp_proxy=http://your.ftp-proxy.url:port
export socks_proxy=http://your.socks-proxy.url:port
export no_proxy=".your-company-domain.com,localhost"
export HTTP_PROXY=$http_proxy
export HTTPS_PROXY=$https_proxy
export FTP_PROXY=$ftp_proxy
export SOCKS_PROXY=$socks_proxy
export NO_PROXY=$no_proxy
wget
****
Run this command to enable downloading from websites from the terminal:
.. code-block:: bash
echo >> ~/.wgetrc <<EOF
http_proxy = your.http-proxy.url:port
https_proxy = your.https-proxy.url:port
ftp_proxy = your.http-proxy.url:port
no_proxy = your-company-domain.com, localhost
EOF
System service (Docker)
***********************
For Docker (and other services that use systemd), you can follow these steps to configure and check proxy settings:
.. note::
Be sure to use :command:`sudo`, as you will need elevated permissions.
#. Create :file:`/etc/systemd/system/docker.service.d` directory to host
configuration information for the Docker service.
#. Create :file:`/etc/systemd/system/docker.service.d/http-proxy.conf` and add:
.. code-block:: bash
[Service]
Environment="HTTP_PROXY=http://your.http-proxy.url:port/"
Environment="HTTPS_PROXY=http://your.https-proxy.url:port/"
#. Load the changes and restart the service:
.. code-block:: bash
sudo systemctl daemon-reload
sudo systemctl restart docker
#. Verify that changes have loaded:
.. code-block:: bash
systemctl show --property=Environment docker
.. code-block:: console
Environment=HTTP_PROXY=http://your.http-proxy.url:port/ HTTPS_PROXY=http://your.https-proxy.url:port/
.. note::
This process enables the ability to successfully run ``docker pull``.
Containers themselves must be configured independently.
git over ssh
************
Add the following to your :file:`~/.ssh/config`:
.. code-block:: bash
host github.com
port 22
user git
ProxyCommand connect-proxy -S your.ssh-proxy.url:port %h %p
.. note::
Though netcat is included with Clear Linux, it is not the BSD version,
which is the one usually used to enable git over ssh.
+1 -6
View File
@@ -78,10 +78,5 @@ own records, then you must set up your own telemetry backend server.
Next steps
**********
To put this concept into practice, see the following resources:
To put this concept into practice, refer to :ref:`telem-guide`.
* :ref:`telem-guide`
* `Telemetry feature description`_
.. _`Telemetry feature description`:
https://clearlinux.org/features/telemetry
@@ -1,31 +1,49 @@
.. _gce:
Launch |CL-ATTR| Compute Engine on Google\* Cloud Platform
Launch |CL-ATTR| Compute Engine on Google Cloud Platform\*
##########################################################
This tutorial walks you through the steps to create a virtual machine
instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP).
instance of |CL-ATTR| on `Google Cloud Platform`_ (:abbr:`GCP (Google Cloud Platform)`).
#. Sign in your Google account on the
.. contents:: :local:
:depth: 1
Prerequisites
*************
* Set up a Google account and a GCP billing account.
* Generate and install a user SSH key in the Linux PCs that will connect to
the VMs in GCP.
Process
*******
#. Sign in to your Google\* account on the
`Google Cloud Console <https://console.cloud.google.com/>`_:
.. figure:: figures/gce/00-sign-in.png
:scale: 50 %
:alt: Sign in Google services
:alt: Sign in to Google services
Figure 1: Google sign in screen
#. Google Cloud Platform uses **Projects** to manage resources.
Select or create a new project for hosting the |CL| VM.
.. note::
Reference the
Refer to the
`Quickstart Using a Linux VM <https://cloud.google.com/compute/docs/quickstart-linux>`_
guide to get familiar with the process of creating VM instances on GCP.
guide to learn about the process of creating VM instances on GCP.
#. Navigate to the latest |CL|
`release folder <https://download.clearlinux.org/releases/current/clear/>`_
to view currently released ``gce`` image, and download the image archive
*clear-<release number>-gce.tar.gz*.
to view the currently released :abbr:`GCE (Google Compute Engine\*)`
image, and download the :file:`clear-<release number>-gce.tar.gz`
image archive.
You don't need to uncompress the image archive, the intact file will
be uploaded to the Google Cloud Storage later.
@@ -42,26 +60,32 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP).
:scale: 50 %
:alt: Browse Google Cloud Storage
Figure 2: Browse Google Cloud Storage
.. note::
You may need to create a billing account and link to this project
before you create a bucket.
.. figure:: figures/gce/02-storage-browser.png
:scale: 50 %
:alt: Cloud Storage Broswer tool
:alt: Cloud Storage Browser tool
Figure 3: Cloud Storage Browser tool
* Click the ``CREATE BUCKET`` button to enter the bucket creation tool.
The bucket name must be unique as buckets in the Cloud Storage share
The bucket name must be unique because buckets in the Cloud Storage share
a single global namespace.
You can leave the rest of options as defaults, and select the
Leave the remaining options set to the defaults, and select the
``Create`` button at the bottom to create a *Bucket*.
.. figure:: figures/gce/03-create-bucket.png
:scale: 50 %
:alt: Set an unique bucket name
:alt: Set a unique bucket name
#. Once the bucket is created, press the ``Upload files`` button
Figure 4: Set bucket name
#. Once the bucket is created, click the ``Upload files`` button
on the Bucket details page to upload the |CL| GCE image archive
to the named bucket:
@@ -69,24 +93,32 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP).
:scale: 50 %
:alt: Cloud Storage bucket is available for storing objects
Figure 5: Cloud Storage bucket
.. figure:: figures/gce/10-image-upload.png
:scale: 50 %
:alt: Uploading the image source archive file
Figure 6: Uploading the image source archive file
.. figure:: figures/gce/11-bucket-uploaded.png
:scale: 50 %
:alt: Image archive imported complete
Figure 7: Importing complete
#. Browse the Compute Engine Image library page:
* Click the *Navigation menu* icon on the upper left screen menu.
* Hover mouse over the *Compute Engine* menu and select *Images*.
* Hover your mouse over the *Compute Engine* menu and select *Images*.
.. figure:: figures/gce/20-gce-image.png
:scale: 50 %
:alt: Go to Google Compute Engine Image library
Figure 8: Image library
#. On the Compute Engine Image library page, click the ``[+] CREATE IMAGE``
menu item to create a custom image:
@@ -94,18 +126,22 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP).
:scale: 50 %
:alt: Create a Google Compute Engine image
Figure 9: Create image
#. In the VM image creation page, change the image source type to
*Cloud Storage file*.
#. Under :guilabel:`Cloud Storage file`, select :guilabel:`Browse`.
#. Locate the *clear-<release number>-gce.tar.gz*,
#. Locate the :file:`clear-<release number>-gce.tar.gz` file,
and click :guilabel:`Select`.
.. figure:: figures/gce/21-create-image.png
:scale: 50 %
:alt: Create the image using the imported image archive object
Figure 10: Create image using imported object
Accept all default options, and click the ``Create`` button
at the bottom to import the Clear Linux GCE image to the image library.
@@ -113,18 +149,22 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP).
:scale: 50 %
:alt: Clear Linux Compute Engine image is created
Figure 11: Image is created
#. After the |CL| image is imported, you can launch a VM instance running
|CL|:
* Click the *Navigation menu* icon on the upper left screen menu.
* Hover mouse over the *Compute Engine* menu group and select
* Hover your mouse over the *Compute Engine* menu group and select
the *VM instances* item.
.. figure:: figures/gce/30-vm-instances.png
:scale: 50 %
:alt: Go to VM instances catalog
Figure 12: VM instances catalog
#. If no VM instance was created in this project, you will be prompted to
create one.
@@ -135,25 +175,33 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP).
:scale: 50 %
:alt: Prompt for VM creation
Figure 13: VM creation
.. figure:: figures/gce/30-vm-catalog.png
:scale: 50 %
:alt: List of VM instances
* In :guilabel:`Region`, decide on a region based on the
`Best practices for Compute Engine regions selection`_ .
Figure 14: VM instances list
* Under *Boot disk*, click the ``Change``.
* In :guilabel:`Region`, choose a region based on the
`Best practices for Compute Engine regions selection`_.
* Under *Boot disk*, click the ``Change`` button.
.. figure:: figures/gce/30-create-vm.png
:scale: 50 %
:alt: Use custom image while creating Clear Linux VM instance
Figure 15: Use custom image
* Select the *Custom images* tab for using Clear Linux OS GCE image.
.. figure:: figures/gce/31-select-boot-disk.png
:scale: 50 %
:alt: Select Clear Linux boot disk to create a VM instance
Figure 16: Select Clear Linux boot disk to create a VM instance
* Scroll down to the bottom of the VM instance creation page,
expand the *Management, security, disks, networking, sole tenancy* group.
@@ -161,12 +209,14 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP).
:scale: 50 %
:alt: Clear Linux requires setting up SSH keys
Figure 17: Set up SSH keys
.. note::
|CL| does not allow SSH login with a root account by default.
As a result, you need to configure the VM instance with your
SSH public key, so that you are able to remote access to it.
As a result, you must configure the VM instance with your
SSH public key, so that you are able to access it remotely.
Refer also to :ref:`security` for more details.
Refer to :ref:`security` for more details.
* Click the *Security* tab, copy and paste your SSH public key:
@@ -174,28 +224,43 @@ instance of |CL-ATTR| on `Google Cloud Platform`_ (GCP).
:scale: 50 %
:alt: Set SSH key for remote login
Figure 18: Set SSH key for remote login
.. warning::
The username is assigned from characters preceding ``@`` in the
email address, included in the SSH key. The dot symbol "." is not
allowed, as it is an invalid character while creating user accounts
in |CL|.
The username is assigned from characters preceding ``@`` in the email
address, included in the SSH key. The dot symbol "." is not allowed,
because it is an invalid character while creating user accounts in
|CL|.
* Click the ``Create`` button to create the |CL| VM.
#. The Clear Linux VM instance is created and offered a public IP address:
#. The Clear Linux VM instance is created and assigned a public IP address:
.. figure:: figures/gce/41-vm-created.png
:scale: 50 %
:alt: Clear Linux VM instance is created and started
#. You can now SSH login to the VM by using the IP address obtained in the
Figure 19: Clear Linux VM instance is created and started
#. You can now SSH login to the VM using the IP address obtained in the
previous step, and the username associated with the SSH public key:
.. figure:: figures/gce/42-ssh-vm.png
:scale: 50 %
:alt: SSH log in to the Clear Linux VM
:alt: SSH login to the Clear Linux VM
Figure 20: SSH login to Clear Linux VM
Related topics
**************
The following tutorials describe a similar process and may be useful references:
* :ref:`azure`
* :ref:`aws-web`
.. _Google Cloud Platform: https://cloud.google.com/
.. _Best practices for Compute Engine regions selection: https://cloud.google.com/solutions/best-practices-compute-engine-region-selection
.. _Best practices for Compute Engine regions selection: https://cloud.google.com/solutions/best-practices-compute-engine-region-selection
@@ -61,7 +61,7 @@ Download and launch the virtual machine
.. code-block:: bash
curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep '[0-9]'-kvm)
curl -O https://cdn.download.clearlinux.org/image/$(curl https://cdn.download.clearlinux.org/image/latest-images | grep '[0-9]'-kvm'\.')
#. Uncompress the downloaded image:
+13 -8
View File
@@ -129,8 +129,8 @@ Before you begin, you must:
Obtain kernel module source
===========================
A required :file:`dkms.conf` file inside of the kernel module's source code directory
informs DKMS how the kernel module should be compiled.
A required :file:`dkms.conf` file inside of the kernel module's source code
directory informs DKMS how the kernel module should be compiled.
Kernel modules may come packaged as:
@@ -141,10 +141,10 @@ Kernel modules may come packaged as:
- Precompiled module binaries only (without source code)
Of the package types listed above, only precompiled kernel module binaries
will not work, because |CL| requires kernel modules to be built against
the same kernel source tree before they can be loaded. If you are only able to
obtain source code without a :file:`dkms.conf` file, you must manually create a
:file:`dkms.conf` file, described later in this document.
will not work, because |CL| requires kernel modules to be built against the
same kernel source tree before they can be loaded. If you are only able to
obtain source code without a :file:`dkms.conf` file, you must manually create
a :file:`dkms.conf` file, described later in this document.
#. Download the kernel module's source code.
@@ -210,8 +210,8 @@ If the kernel module source does not contain a :file:`dkms.conf` file or the
create the file.
Review the kernel module README documentation for guidance on what needs to be
in the :file:`dkms.conf` file, including special variables that may be required to
build successfully.
in the :file:`dkms.conf` file, including special variables that may be
required to build successfully.
Here are some additional resources that can be used for reference:
@@ -223,6 +223,10 @@ Here are some additional resources that can be used for reference:
* `Sample dkms.conf file`_ in the GitHub\* repository for the DKMS project.
.. note::
:command:`AUTOINSTALL=yes` must be set in the dkms.conf for the module to
be automatically recompiled with |CL| updates.
The instructions below show a generic example:
@@ -240,6 +244,7 @@ The instructions below show a generic example:
PACKAGE_NAME=custom_module
PACKAGE_VERSION=1.0
DEST_MODULE_LOCATION=/kernel/drivers/other
AUTOINSTALL=yes
This example identifies a kernel module named *custom_module* with version
*1.0*.
+147 -43
View File
@@ -1,71 +1,175 @@
.. _assign-static-ip:
Assign a static IP address to a network interface
#################################################
Introduction
************
Assign a static IP address
##########################
By default, your |CL-ATTR| system automatically gets an IP address from your
network via DHCP. If you do not have a DHCP server on your network or simply
want to use a static IP address, follow the steps in this guide.
Process
*******
.. contents::
:local:
:depth: 1
#. Create this directory structure:
Identify which program is managing the interface
************************************************
.. code-block:: bash
New installations of |CL| use NetworkManager as the default network interface
manager for all network connections.
sudo mkdir -p /etc/systemd/network
.. note::
#. Identify the interface to be assigned the static IP address:
* The *cloud* |CL| images continue to use systemd-networkd to manage
network connections.
.. code-block:: bash
* In earlier |CL| versions, systemd-network was used to manage Ethernet
interfaces and NetworkManager was used for wireless interfaces.
ip addr
The system returns the following:
Before defining a configuration for assigning a static IP address, you should
verify which program is managing the network interface.
.. code-block:: console
#. Check the output of :command:`nmcli device` to see if NetworkManager is
managing the device.
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
.. code-block:: bash
2: wlp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 4a:98:8d:e5:43:15 brd ff:ff:ff:ff:ff:ff
nmcli device status
3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP group default qlen 1000
link/ether f4:4d:30:68:96:20 brd ff:ff:ff:ff:ff:ff
inet 10.0.1.2/24 brd 10.54.74.255 scope global dynamic eno1
valid_lft 6766sec preferred_lft 6766sec
inet6 fe80::f64d:30ff:fe68:9620/64 scope link
valid_lft forever preferred_lft forever
If the STATE column for the device shows *connected* or *disconnected*, the
network configuration is being managed by NetworkManager and the instructions
for :ref:`using NetworkManager <nm-static-ip>` should be used.
In this example, we will use the `eno1` interface.
If the STATE column for the device shows *unmanaged*, check to see if the
device is being managed by systemd-networkd
#. Create the :file:`70-static.network` file and add the following:
.. code-block:: bash
#. Check the output of :command:`networkctl list` to see if
systemd-networkd is managing the device.
sudo $EDITOR /etc/systemd/network/70-static.network
.. code-block:: bash
[Match]
Name=[interface name]
[Network]
Address=[IP address]/24
DHCP=yes # to get DNS info, etc.
networkctl list
Replace [interface name] and [IP address] with your specific settings.
If the SETUP column for the device shows *configured*, the network
configuration is being managed by systemd-networkd and the instructions for
:ref:`using systemd-networkd <networkd-static-ip>` should be used.
#. Restart the networkd service:
.. code-block:: bash
.. _nm-static-ip:
sudo systemctl restart systemd-networkd
Using NetworkManager
********************
Network connections managed by NetworkManager are stored as files with the
:file:`.nmconnection` file extension in the
:file:`/etc/NetworkManager/system-connections/` directory.
A few tools exists to aid to manipulate network connections managed by
NetworkManager:
* nmcli - a command-line tool
* nmtui - a text user interface that provides a pseudo graphical menu in the
terminal
* nm-connection-editor - a graphical user interface
The method below uses the command line tool *nmcli* to modify network
connection.
#. Identify the existing connection name.
.. code:: bash
nmcli connection show
The output will look like this:
.. code:: bash
NAME UUID TYPE DEVICE
Wired connection 1 00000000-0000-0000-0000-000000000000 802-3-etherneten01
If a connection does not exist, it will need to be created with
:command:`nmcli connection add`.
#. Modify the connection to use a static IP address. Replace the variables in
brackets with the appropriate values. *[CONNECTION_NAME]* should be
replaced with the NAME from the command above.
.. code::
sudo nmcli connection modify "[CONNECTION_NAME]" \
ipv4.method "manual" \
ipv4.addresses "[IP_ADDRESS]/[CIDR_NETMASK]" \
ipv4.gateway "[GATEWAY_IP_ADDRESS]" \
ipv4.dns "[PRIMARY_DNS_IP],[SECONDARY_DNS_IP]"
See the `nmcli developer page <https://developer.gnome.org/NetworkManager/stable/nmcli.html>`_ for more
configuration options. For advanced configurations, the
:file:`/etc/NetworkManager/system-connections/*.nmconnection`. can be edited
directly.
#. Verify your static IP address details have been set
.. code-block:: bash
nmcli
.. _networkd-static-ip:
Using systemd-networkd
**********************
Network connections managed by systemd-networkd are stored as files with the
:file:`.network` file extension the :file:`/etc/systemd/network/` directory.
Files to manipulate network connections managed by systemd-networkd must be
created manually.
#. Create the :file:`/etc/systemd/network` directory if it doesn't exist already:
.. code-block:: bash
sudo mkdir -p /etc/systemd/network
#. Create a :file:`.network` file and add the following content. Replace the
variables in brackets with the appropriate values. *[INTERFACE_NAME]*
should be replaced with LINK from the output of :command:`networkctl list`
ran previously.
.. code-block:: bash
sudo $EDITOR /etc/systemd/network/70-static.network
[Match]
Name=[INTERFACE_NAME]
[Network]
Address=[IP_ADDRESS]/[CIDR_NETMASK]
Gateway=[GATEWAY_IP_ADDRESS]
DNS=[PRIMARY_DNS_IP]
DNS=[SECONDARY_DNS_IP]
See the `systemd-network man page
<https://www.freedesktop.org/software/systemd/man/systemd.network.html>`_
for more configuration options.
#. Restart the systemd-networkd service:
.. code-block:: bash
sudo systemctl restart systemd-networkd
#. Verify your static IP address details have been set
.. code-block:: bash
networkctl status
**Congratulations!** You have successfully assigned a static IP address.
+21 -15
View File
@@ -35,14 +35,11 @@ Table 2 lists the currently available images that are platform specific.
* - Image Type
- Description
* - installer.img
- Preferred image of |CL| with interactive installer.
* - live-desktop.img or live-desktop.iso
- Image for booting to GNOME\* desktop to preview or install the OS.
* - installer.iso
- ISO of |CL| with interactive installer. Only for special cases where ISO image format is required (not for use with a USB key)
* - live.img
- Image for live booting into memory, without requiring installaton.
* - live-server.img or live-server.iso
- Image for booting to server command prompt to preview or install the OS.
.. list-table:: Table 2: Types of platform-specific |CL| images
:widths: 15, 85
@@ -51,20 +48,23 @@ Table 2 lists the currently available images that are platform specific.
* - Image Type
- Description
* - aws.img
- Image suitable for use with Amazon\* AWS\*.
* - azure.vhd
- Virtual Hard Disk for use on Microsoft\* Azure\* cloud platform
- Virtual Hard Disk for use on Microsoft\* Azure\* cloud platform.
* - azure-docker.vhd
- Virtual Hard Disk for use on Microsoft Azure cloud platform with Docker\* pre-installed
- Virtual Hard Disk for use on Microsoft Azure cloud platform with Docker\* pre-installed.
* - azure-machine-learning.vhd
- Virtual Hard Disk for use on Microsoft Azure cloud platform with the `machine-learning-basic` bundle installed
- Virtual Hard Disk for use on Microsoft Azure cloud platform with the `machine-learning-basic` bundle installed.
* - cloud.img
- Image for use by cloud deployments such as OpenStack\*
* - cloudguest.img
- Image with generic cloud guest virtual machine (VM) requirements installed.
* - containers.img
- Image for use by Clear Containers runtime. Includes `optimized kernel`_ for Clear Containers.
* - gce.tar
- Image with the Google Compute Engine (GCE) specific kernel.
* - hyperv.vhdx
- Virtual Hard Disk for use with Microsoft Hyper-V\* hypervisor. Includes `optimized kernel`_ for Hyper-V.
@@ -73,9 +73,15 @@ Table 2 lists the currently available images that are platform specific.
- Image for booting in a simple VM with start_qemu.sh. Includes
`optimized kernel`_ for KVM.
* - kvm-legacy.img
- Image for booting in a simple VM using legacy BIOS, if using start_qemu.sh make sure to remove -bios parameter.
* - pxe.tar
- Image suitable for use with PXE server.
* - vmware.vmdk
- Virtual Machine Disk for VMware\* platforms inclduing Player, Workstation, and ESXi.
.. _images: https://cdn.download.clearlinux.org/image
.. _images: https://clearlinux.org/downloads
.. _`optimized kernel`: https://clearlinux.org/documentation/clear-linux/reference/compatible-kernels
+8 -8
View File
@@ -20,22 +20,22 @@ manually configure the proxies.
Corporate and private networks can be very complex, needing to restrict and
control network connections for security reasons. The typical side effects
are limited or blocked connectivity and requiring manual configuration of
proxies to perform the most mundane tasks such as cloning a repo or checking
are limited or blocked connectivity, and require manual configuration of
proxies to perform the most mundane tasks, such as cloning a repo or checking
for updates. With |CL|, all of the work is done behind the scenes to
effortlessly use your network and have connections “just work”.
This feature removes massive complications in network connectivity due to
proxy issues. You can automate tasks like unit testing without worrying
about the proxy not being set and you can remove unset proxies from the
This feature removes severe complications with network connectivity due to
proxy issues. You can automate tasks, such as unit testing, without worrying
about the proxy not being set, and you can remove unset proxies from the
equation when dealing with network unavailability across systems.
How it works
************
We designed Autoproxy around tools provided by most Linux
We designed Autoproxy around tools provided by most Linux\*
distributions with a few minor additions and modifications. We leveraged the
DHCP and network information provided from systemd and created a
DHCP and network information obtained from systemd and created a
PAC-discovery daemon. The daemon uses the information to resolve a URL for a
PAC file. The daemon then passes the URL into PACrunner\*. PACrunner
downloads the PAC file and uses the newly implemented Duktape\* engine to
@@ -49,7 +49,7 @@ parse it.
From that point on, any cURL\* or network requests query PACrunner for the
correct proxy to use. We modified the cURL library to communicate with
PACrunner over DBus. However, cURL will ignore PACrunner and run normally if
no PAC file is loaded or if you set any proxies manually. Thus, your
no PAC file is loaded or if you manually set any proxies. Thus, your
environment settings are respected and no time is wasted trying to resolve a
proxy. All these steps happen in the background with no user interaction.
+29 -29
View File
@@ -3,8 +3,8 @@
autospec
########
**autospec** is a tool to assist in the automated creation and maintenance of
RPM packaging in |CL-ATTR|. Where a standard RPM build process using
**autospec** is a tool used to assist with the automated creation and maintenance of
RPM packaging in |CL-ATTR|. Where a standard :abbr:`RPM (RPM Package Manager)` build process using
:command:`rpmbuild` requires a tarball and :file:`.spec` file to start, autospec
requires only a tarball and package name to start.
@@ -16,17 +16,17 @@ Description
***********
The autospec tool attempts to infer the requirements of the :file:`.spec` file
by analyzing the source code and :file:`Makefile` information. It will
continuously run updated builds based on new information discovered from build
by analyzing the source code and :file:`Makefile` information. It
continuously runs updated builds based on new information discovered from build
failures until it has a complete and valid :file:`.spec` file. If needed, you
can influence the behavior of autospec and customize the build by providing
optional `control files`_ to the autospec tool.
autospec uses mock as a sandbox to run the builds. Visit the `mock wiki`_ for
autospec uses **mock** as a sandbox to run the builds. Visit the `mock wiki`_ for
additional information on using mock.
For a general understanding of how RPMs work, visit the `rpm website`_ or the
`RPM Packaging Guide`_ .
For a general understanding of how an RPM works, visit
the `rpm website`_ or the `RPM Packaging Guide`_ .
How it works
************
@@ -44,7 +44,7 @@ The setup for building source in |CL| must be completed before using the
autospec tool.
Refer to `Setup environment to build source`_ for instructions on completing
setup.
the setup.
Create an RPM
=============
@@ -52,7 +52,7 @@ Create an RPM
The basic autospec process is described in the following steps:
#. The :command:`make autospec` command generates a :file:`.spec` file based on
analysis of code and existing control files.
the analysis of code and existing control files.
Any control files should be located in the same directory as the resulting
:file:`.spec` file.
@@ -65,13 +65,13 @@ The basic autospec process is described in the following steps:
#. autospec detects any missed declarations in the :file:`.spec`.
#. If build errors occur, autospec will scan the build log to try and detect
#. If build errors occur, autospec scans the build log to try to detect
the root cause.
#. If autospec detects the root cause and knows how to continue, it will restart
#. If autospec detects the root cause and knows how to continue, it restarts
the build automatically at step 1 with updated build instructions.
#. Otherwise, autospec will stop the build for user inspection to resolve the
#. Otherwise, autospec stops the build for user inspection to resolve the
errors. Respond to the build process output by fixing source code issues
and/or editing control files to resolve issues, which may include
dependencies or exclusions. See `autospec README`_ for more information on
@@ -79,7 +79,7 @@ The basic autospec process is described in the following steps:
The user resumes the process at step 1 after errors are resolved.
If a binary dependency doesn't exist in |CL|, you will need to build it
If a binary dependency doesn't exist in |CL|, you must build it
before running autospec again.
Following these steps, autospec continues to rebuild the package, based on
@@ -95,10 +95,10 @@ Complete `Setup environment to build source`_ before using these examples.
:local:
:depth: 1
Example 1: Build RPM with existing spec file
============================================
Example 1: Build RPM with an existing spec file
===============================================
This example shows how to build a RPM from a pre-packaged upstream package, with
This example shows how to build a RPM from a pre-packaged upstream package with
an existing spec file. The example uses the ``dmidecode`` package.
#. Navigate to the autospec workspace and clone the ``dmidecode`` package:
@@ -110,7 +110,7 @@ an existing spec file. The example uses the ``dmidecode`` package.
.. note::
You can clone all package repos at once using:
You can clone all package repos at once using the following command:
.. code-block:: bash
@@ -147,7 +147,7 @@ create a simple helloclear RPM.
cd ~/clearlinux
make autospecnew URL="https://github.com/clearlinux/helloclear/archive/helloclear-v1.0.tar.gz" NAME="helloclear"
The resulting RPMs are in :file:`./packages/helloclear/rpms`. Builde logs and
The resulting RPMs are in :file:`./packages/helloclear/rpms`. Build logs and
additional RPMs are in :file:`./packages/helloclear/results`.
Example 3: Generate a new spec file with a pre-defined package
@@ -198,8 +198,8 @@ Example 4: Provide control files to autospec
============================================
This example shows how to modify control files to correct build failures that
autospec is unable to resolve. In this example you will add a missing license
and dependencies in order for autospec to complete a successful build.
autospec is unable to resolve. In this example, you will add a missing license
and dependencies so autospec can complete a successful build.
#. Navigate to the autospec workspace:
@@ -226,7 +226,7 @@ and dependencies in order for autospec to complete a successful build.
make autospecnew URL="https://github.com/OPAE/opae-sdk/archive/0.13.0.tar.gz" NAME="opae-sdk"
This will give an error for a missing license file:
This results in an error for a missing license file:
.. code-block:: console
@@ -238,7 +238,7 @@ and dependencies in order for autospec to complete a successful build.
cd packages/opae-sdk
#. Add one or more valid license identifier from the
#. Add one or more valid license identifiers from the
`SPDX License List <https://spdx.org/licenses/>`_.
In the example below, two different licenses are appropriate based on the
opae-sdk project licensing:
@@ -253,7 +253,7 @@ and dependencies in order for autospec to complete a successful build.
make autospec
This will result in a generic error:
This results in a generic error:
.. code-block:: console
@@ -265,7 +265,7 @@ and dependencies in order for autospec to complete a successful build.
cat ./results/build.log
In the build log, you will find details for the specific failures. In this
The build log contains details for the specific failures. In this
instance, there are missing dependencies:
.. code-block:: console
@@ -279,7 +279,7 @@ and dependencies in order for autospec to complete a successful build.
linked by target "opae-c" in directory /builddir/build/BUILD/opae-sdk-0.13.0/libopae
#. Search the spec files of upstream |CL| packages to see if the json-c library
is availabe. In this case, it does exist and we'll add the json-c 'dev'
is available. In this case, it does exist and we'll add the json-c 'dev'
package into the buildreq_add:
.. code-block:: bash
@@ -302,7 +302,7 @@ and dependencies in order for autospec to complete a successful build.
grep 'libuuid\.so$' ~/clearlinux/packages/*/*.spec
echo "util-linux-dev" >> buildreq_add
#. Run autospec again and find the successfully-generated RPMs in the rpms
#. Run autospec again and find the successfully-generated RPMs in the :file:`rpms`
directory:
.. code-block:: bash
@@ -384,7 +384,7 @@ To test an autospec-created package inside a VM:
The code that makes this possible can be viewed by searching for the
*install:* target in the `Makefile.common file on GitHub`_.
#. Return back to the :file:`~/clearlinux` directory and start the |CL| VM:
#. Return to the :file:`~/clearlinux` directory and start the |CL| VM:
.. code-block:: bash
@@ -411,7 +411,7 @@ Test directly on a development machine
The |CL| development tooling also includes a method to extract
autospec-created RPMs locally onto a |CL| development system for testing.
Extracting an RPM directly onto a system offers quicker testing, however
Extracting an RPM directly onto a system offers quicker testing; however
conflicts may occur and responsibility to remove the software after testing is
up to the developer.
@@ -456,7 +456,7 @@ automated for you with a setup script. It uses tools from the
The setup script creates a workspace in the :file:`clearlinux` folder, with the
subfolders :file:`Makefile`, :file:`packages`, and :file:`projects`. The
:file:`projects` folder contains the main tools used for making packages in
|CL|: `autospec` and `common`.
|CL| :file:`autospec` and :file:`common`.
Follow these steps to setup the workspace and tooling for building source:
+11 -12
View File
@@ -3,7 +3,7 @@
ister.py image builder
######################
The `ister.py tool`_ is a template based installer used by |CL-ATTR| to produce
The `ister.py tool`_ is a template-based installer used by |CL-ATTR| to produce
images for each release. The same ister tool is available for use in |CL| to
create custom images based on an upstream image.
@@ -14,21 +14,21 @@ create custom images based on an upstream image.
Description
***********
|CL| is a rolling release and produces on average 10 releases per week using the
|CL| is a rolling release and produces an average of 10 releases per week using the
ister tool. With each release we produce multiple
`image types for different environments`_ and use cases such as installers,
Hyper-V, KVM, or VMWare.
:ref:`image types for different environments <image-types>` and use cases such
as installers, Hyper-V, KVM, or VMWare.
Each image has a JSON configuration file used by ister to generate the image.
These JSON configuration files describe the image type, partitions, version,
and which bundles will be preinstalled by default with the image. For each image
type we produce, the corresponding JSON configuration file for the image is also
Each image has a JSON configuration file that is used by ister to generate the
image. These JSON configuration files describe the image type, partitions, version,
and bundles that will be preinstalled by default with the image. For each image
type we produce, the corresponding JSON configuration file for the image also is
published.
The :ref:`mixer<mixer>` tool also uses ister to build images for your custom
mix. Like upstream images, a JSON configuration file is defined for the image,
which ister uses to generate the image. Refer to the :ref:`mixer<mixer>` guide
for instruction on using ister to build an image for a custom mix.
for instructions on using ister to build an image for a custom mix.
Examples
********
@@ -37,7 +37,7 @@ Recreate an upstream image
==========================
The published configuration files for upstream images may be used to recreate an
image, for example when you want to:
image. Here are some examples:
* Use an older version of |CL| and the image is no longer available (only after
March 2017).
@@ -64,7 +64,7 @@ configuration file:
:file:`/clear/config/image`. For example:
``https://cdn.download.clearlinux.org/releases/15700/clear/config/image/``
#. Download “PostNonChroot” script (if applicable).
#. Download the “PostNonChroot” script (if applicable).
The JSON configuration file for the image may have an accompanying
“PostNonChroot” script that is executed at the end of the image creation
@@ -88,7 +88,6 @@ Related topics
* :ref:`bulk-provision`
.. _ister.py tool: https://github.com/bryteise/ister
.. _image types for different environments: https://cdn.download.clearlinux.org/image/README-IMAGES.html
.. _Configuration files for the current release: https://cdn.download.clearlinux.org/current/config/image/
.. _Previous releases: https://cdn.download.clearlinux.org/releases/
.. _Install a bundle: https://clearlinux.org/documentation/clear-linux/guides/maintenance/swupd-guide#adding-a-bundle
+93 -102
View File
@@ -8,77 +8,68 @@ software.
.. contents::
:local:
:depth: 2
:depth: 1
Description
***********
:command:`swupd` has two main functions:
#. It manages software replacing APT or YUM, installing bundles
#. Manages software and replaces APT or YUM, by installing bundles
rather than packages.
#. It checks for system updates and installs them.
#. Checks for system updates and installs them.
:ref:`Bundles <bundles>` are the smallest granularity component that is
managed by |CL| and contain everything needed to deliver a software
capability. Rather than downloading a cascade of package dependencies when
installing a piece of software, a bundle comes with all of its dependencies.
:command:`swupd` manages overlapping dependencies behind the scenes ensuring
that all software is compatible across the system.
:command:`swupd` manages overlapping dependencies behind the scenes, ensuring
that all software is compatible across the system. It can be used to verify
the OS, clean cached files, and fix issues.
:ref:`Bundles <bundles>` contain everything needed to deliver a software
capability. They are the smallest granularity component that is
managed by |CL|. A bundle comes with all of its dependencies, rather than
downloading a cascade of package dependencies when installing a piece of
software.
Versioning
==========
In a traditional distribution, the process of describing current software
versioning usually involves:
Using package managers to keep track of software version compatibility or
compare multiple systems on many Linux distributions can be cumbersome.
- Listing and keeping track of the current OS release (generally
uninformative about any singular packages or functionality).
With |CL| :command:`swupd`, versioning happens at the individual file-level.
This means |CL| generates an entirely new OS version with any set of software
changes to the system (including software downgrades or removals). This
rolling release versioning model is similar to :command:`git` internal version
tracking, where any of the individual file commits are tracked and move the
pointer forward when changed.
- Keeping track of packages and repositories being used, and updating them
individually.
- Listing and tracking every package available and installed on the
system, none of which are directly tied to the current OS release.
This can be done effectively, but given the nearly endless combinations of
packages and versions of packages a server may have, it quickly becomes
non-trivial to define what "version" the system is and what software it
is running without explicitly going through each system and inspecting
every package.
With |CL|, we need track:
- One number
A number representing the **current** release of the OS is sufficient to
describe the versions of all the software on the OS. Each build is
composed of a specific set of bundles made from a particular version of
packages. This matters on a daily basis to system administrators, who
need to determine which of their systems do not have the latest security
fixes, or which combinations of software have been tested. Every release
of the same number is guaranteed to contain the same versions of software,
so there's no ambiguity between two systems running the same version of |CL|.
A number that represents the **current** release of the OS describes the
versions of all the software on the OS. Each build is composed of a specific
set of bundles made from a particular version of packages. This matters on a
daily basis to system administrators, who need to determine which of their
systems do not have the latest security fixes, or which combinations of
software have been tested. Every release of the same number is guaranteed to
contain the same versions of software, so there's no ambiguity between two
systems running the same version of |CL|.
Updating
========
|CL| enforces regular updating of the OS by default and will automatically
check for updates against a version server. The content server provides the
|CL| enforces regular updating of the OS by default and automatically
checks for updates against a version server. The content server provides the
file and metadata content for all versions and can be the same as the
version server. The content url server provides metadata in the form of
manifests. These Manifest files list and describe file contents, symlinks,
*manifests*, which list and describe file contents, symlinks,
and directories. Additionally, the actual content is
provided to clients in the form of archive files.
Software updates with |CL| are also efficient. Unlike package-based
distributions, :command:`swupd` only updates files that have changed rather
distributions, :command:`swupd` only updates files that have changed, rather
than entire packages. For example, it is quite common for an OS security
patch to be as small as 15 KB. Using binary deltas, the |CL| is able to
patch to be as small as 15 KB. Using binary deltas, |CL| is able to
apply only what is needed.
To get a more detailed understanding of how to generate update content for
|CL| see the :ref:`mixer <mixer>` tool.
For details on how to generate update content for |CL|, see the
:ref:`mixer <mixer>` tool.
How it works
************
@@ -93,42 +84,41 @@ Prerequisites
Updates
=======
|CL| updates are automatic by default but can be set to occur only on
|CL| updates are automatic by default, but can be set to occur only on
demand. :command:`swupd` makes sure that regular updates are simple and
secure. It can also check the validity of currently installed files and
software and correct any problems.
software, and can correct any problems.
Manifests
---------
The Clear Linux OS software update content consists of data and
metadata. The data is the files that end up in the OS. The metadata
contains relevant information to properly provision the data to the OS
file system, as well as update the system and add or remove additional
content to the OS.
The |CL| software update content consists of data and metadata. The data is
the files that end up in the OS. The metadata contains relevant information to
properly provision the data to the OS file system, as well as update the
system and add or remove additional content to the OS.
The Manifests are mostly long lists of hashes that describe content.
The manifests are mostly long lists of hashes that describe content.
Each bundle gets its own manifest file. There is a master manifest
file that describes all manifests to tie it all together.
Fullfiles, packs, and delta packs
---------------------------------
The data that an update provisions to a system can be obtained in
three different ways. There are three different methods, and they
exist to optimize the delivery of content and speed up updates.
To speed up updates and optimize content delivery, update data provisioned to
a system is obtained by one of the following methods:
Fullfiles are always generated for every file in every release. This
allows any Clear Linux OS to obtain the exact copy of the content
for each version directly. This would be used if the OS verification
needed to replace a single file, for instance.
* *Fullfiles* are always generated for every file in every release. This
allows any |CL| to obtain the exact copy of the content
for each version directly. This is used if the OS verification
needs to replace a single file, for instance.
Packs are available for some releases and combine many files to speed
up the creation of installation media and large updates. Delta packs
are an optimized version of packs that only contain updates (binary
diffs) and cannot be used without having the original file content.
* *Packs* are available for some releases. They combine many files to speed
up the creation of installation media and large updates.
Bundle Search
* *Delta packs* are an optimized version of packs that only contain updates
(binary diffs). They cannot be used without having the original file content.
Bundle search
=============
:command:`swupd` search downloads manifest data and searches for
@@ -139,21 +129,21 @@ Only the base bundle is returned. Bundles can contain other bundles via
includes. For more details, see `Bundle Definition Files`_ and its
subdirectory bundles.
Bundles that are already installed, will be marked [installed] in search
Bundles that are already installed are marked **(installed)** in search
results.
Optionally, you can review our `bundles`_ on GitHub.
Optionally, you can review our `bundles`_ on GitHub\*.
Examples
********
Example 1: Disable and Enable automatic updates
Example 1: Disable and enable automatic updates
===============================================
|CL| updates are automatic by default but can be set to occur only
|CL| updates are automatic by default, but can be set to occur only
on demand.
#. First verify your current auto-update setting.
#. Verify your current auto-update setting.
.. code-block:: bash
@@ -202,7 +192,7 @@ Example 2: Find and install Kata\* Containers
Kata Containers is a popular container implementation. Unlike other
container implementations, each Kata Container has its own
kernel instance and runs on its own :abbr:`Virtual Machine (VM)` for
kernel instance and runs on its own :abbr:`VM (Virtual Machine)` for
improved security.
|CL| makes it very easy to install, since you only need to add
@@ -211,34 +201,33 @@ number of dependencies. Also, check out our tutorial: :ref:`kata`.
#. Find the right bundle.
* To return all possible matches for the search string enter
:command:`swupd search`, followed by 'kata':
To return all possible matches for the search string, enter
:command:`swupd search`, followed by 'kata':
.. code-block:: bash
.. code-block:: bash
sudo swupd search kata
sudo swupd search kata
The output should be similar to:
The output should be similar to:
.. code-block:: console
.. code-block:: console
Bundle with the best search result:
Bundle with the best search result:
containers-virt - Run container applications from Dockerhub in lightweight virtual machines
containers-virt - Run container applications from Dockerhub in lightweight virtual machines
This bundle can be installed with:
This bundle can be installed with:
swupd bundle-add containers-virt
swupd bundle-add containers-virt
Alternative bundle options are
Alternative bundle options are
cloud-native-basic - Contains ClearLinux native software for Cloud
cloud-native-basic - Contains ClearLinux native software for Cloud
.. note::
.. note::
If your search does not produce results with a specific
term, shorten the search term. For example, use *kube* instead of
*kubernetes*.
If your search does not produce results with a specific term, shorten
the search term. For example, use *kube* instead of *kubernetes*.
#. Add the bundle.
@@ -248,7 +237,7 @@ number of dependencies. Also, check out our tutorial: :ref:`kata`.
.. note::
To add multiple bundles simply add a space followed by the bundle name.
To add multiple bundles, add a space followed by the bundle name.
The output of a successful installation should be similar to:
@@ -279,7 +268,7 @@ been added to, overwritten, removed, or modified (e.g., permissions).
sudo swupd diagnose
All directories that are watched by :command:`swupd` are verified according
to the manifest data and hash mismatches are flagged as follows:
to the manifest data. Hash mismatches are flagged as follows:
.. code-block:: console
@@ -295,47 +284,49 @@ to the manifest data and hash mismatches are flagged as follows:
423 files did not match
Verify successful
In this case, python packages that were installed on top of the default
In this case, Python\* packages that were installed on top of the default
install were flagged as mismatched. :command:`swupd` can be directed to
ignore or fix issues based on command line options.
:command:`swupd` can correct any issues it detects. Additional directives
can be added including a white list of directories that will be ignored.
can be added including a white list of directories to be ignored.
The following command will repair issues, remove unknown items, and
ignore files or directories matching `/usr/lib/python`:
The following command repairs issues, removes unknown items, and
ignores files or directories matching :file:`/usr/lib/python`:
.. code-block:: bash
sudo swupd repair --picky --picky-whitelist=/usr/lib/python
Quick Reference
Quick reference
***************
swupd info
To see the currently installed version and update servers.
Returns the currently installed version and update servers.
swupd update <version number>
To update to a specific version or with no arguments to update to latest.
Updates to a specific version or updates to latest version if no
arguments are used.
swupd bundle-list [--all]
To list installed bundles.
Lists installed bundles.
swupd bundle-add [-b] <search term>
To find a bundle that contains your search term.
Finds a bundle that contains your search term.
swupd bundle-add <bundle name>
To add a bundle.
Adds a bundle.
swupd bundle-remove <bundle name>
To remove a bundle.
Removes a bundle.
swupd --help
For additional :command:`swupd` commands.
Lists additional :command:`swupd` commands.
man swupd
To reference the :command:`swupd` man page, or see the
`source documentation`_ available on github.
Opens the :command:`swupd` man page.
Refer to :command:`swupd` `source documentation`_ on GitHub for more details.
Related topics
**************
+72 -23
View File
@@ -6,7 +6,6 @@ Deep Learning Reference Stack
This tutorial describes how to run benchmarking workloads for TensorFlow\*,
PyTorch\*, and Kubeflow in |CL-ATTR| using the Deep Learning Reference Stack.
.. contents::
:local:
:depth: 1
@@ -26,14 +25,14 @@ The Deep Learning Reference Stack is available in the following versions:
for Deep Neural Networks (Intel® MKL-DNN) primitives and introduces support
for Intel® AVX-512 Vector Neural Network Instructions (VNNI).
* `Intel MKL-DNN`_, which includes the TensorFlow framework optimized using
Intel® Math Kernel Library for Deep Neural Networks (Intel® MKL-DNN) primitives.
Intel® Math Kernel Library for Deep Neural Networks (Intel® MKL-DNN)
primitives.
* `Eigen`_, which includes `TensorFlow`_ optimized for Intel® architecture.
* `PyTorch with OpenBLAS`_, which includes PyTorch with OpenBlas.
* `PyTorch with Intel MKL-DNN`_, which includes PyTorch optimized using Intel®
Math Kernel Library (Intel® MKL) and Intel MKL-DNN.
.. note::
.. important::
To take advantage of the Intel® AVX-512 and VNNI functionality with the Deep
Learning Reference Stack, you must use the following hardware:
@@ -41,14 +40,16 @@ The Deep Learning Reference Stack is available in the following versions:
* Intel® AVX-512 images require an Intel® Xeon® Scalable Platform
* VNNI requires a 2nd generation Intel® Xeon® Scalable Platform
Stack features
==============
* Deep Learning Reference Stack `V3.0 release announcement`_.
* Deep Learning Reference Stack v2.0 including current `PyTorch benchmark results`_.
* Deep Learning Reference Stack v1.0 including current `TensorFlow benchmark results`_.
* `Release notes on Github\*`_ for the latest release of Deep Learning Reference Stack.
* Deep Learning Reference Stack v2.0 including current
`PyTorch benchmark results`_.
* Deep Learning Reference Stack v1.0 including current
`TensorFlow benchmark results`_.
* `Release notes on Github\*`_ for the latest release of Deep Learning Reference
Stack.
.. note::
@@ -58,19 +59,21 @@ Stack features
Prerequisites
=============
* :ref:`Install <bare-metal-install-desktop>` |CL| on your host system.
* :ref:`Install <bare-metal-install-desktop>` |CL| on your host system
* :command:`containers-basic` bundle
* :command:`cloud-native-basic` bundle
In |CL|, :command:`containers-basic` includes Docker\*, which is required for
TensorFlow and PyTorch benchmarking. Use the :command:`swupd` utility to
check if :command:`containers-basic` and :command:`cloud-native-basic` are present:
check if :command:`containers-basic` and :command:`cloud-native-basic` are
present:
.. code-block:: bash
sudo swupd bundle-list
To install the :command:`containers-basic` or :command:`cloud-native-basic` bundles, enter:
To install the :command:`containers-basic` or :command:`cloud-native-basic`
bundles, enter:
.. code-block:: bash
@@ -91,7 +94,7 @@ Version compatibility
We validated these steps against the following software package versions:
* |CL| 26240 (Lower version not supported.)
* |CL| 26240 (Lower version not supported)
* Docker 18.06.1
* Kubernetes 1.11.3
* Go 1.11.12
@@ -137,8 +140,8 @@ TensorFlow.
You can replace the model with one of your choice supported by the
TensorFlow benchmarks.
If you are using an FP32 based model, it can be converted to an int8 model using `Intel® quantization tools`_
If you are using an FP32 based model, it can be converted to an int8 model
using `Intel® quantization tools`_.
PyTorch single and multi-node benchmarks
****************************************
@@ -182,6 +185,11 @@ Kubeflow multi-node benchmarks
The benchmark workload runs in a Kubernetes cluster. The tutorial uses
`Kubeflow`_ for the Machine Learning workload deployment on three nodes.
.. warning::
If you choose the Intel® MKL-DNN or Intel® MKL-DNN-VNNI image, your platform must support the Intel® AVX-512 instruction set. Otherwise, an *illegal instruction* error may appear, and you won’t be able to complete this tutorial.
Kubernetes setup
================
@@ -196,12 +204,48 @@ We used `flannel`_ as the network provider for these tests. If you
prefer a different network layer, refer to the Kubernetes
`networking documentation`_ for setup.
Kubectl
=======
You can use kubectl to run commands against your Kubernetes cluster. Refer to
the `kubectl overview`_ for details on syntax and operations. Once you have a
working cluster on Kubernetes, use the following YAML script to start a pod with
a simple shell script, and keep the pod open.
#. Copy this example.yaml script to your system:
.. code-block:: console
apiVersion: v1
kind: Pod
metadata:
name: example-pod
labels:
app: ex-pod
spec:
containers:
- name: ex-pod-container
image: clearlinux/stacks-dlrs-mkl:latest
command: ['/bin/bash', '-c', '--']
args: [ "while true; do sleep 30; done" ]
#. Execute the script with kubectl:
.. code-block:: bash
kubectl apply –f <path-to-yaml-file>/example.yaml
This script opens a single pod. More robust solutions would create a deployment
or inject a python script or larger shell script into the container.
Images
======
You must add `launcher.py` to the Docker image to include the Deep
You must add `launcher.py`_ to the Docker image to include the Deep
Learning Reference Stack and put the benchmarks repo in the correct
location. From the Docker image, run the following:
location. Note that this tutorial uses Kubeflow v0.4.0, and cannot guarantee results if you use a different version.
From the Docker image, run the following:
.. code-block:: bash
@@ -210,7 +254,7 @@ location. From the Docker image, run the following:
cp launcher.py /opt
chmod u+x /opt/*
Your entry point becomes: :file:`/opt/launcher.py`
Your entry point becomes: :file:`/opt/launcher.py`.
This builds an image that can be consumed directly by TFJob from Kubeflow.
@@ -220,9 +264,9 @@ ksonnet\*
Kubeflow uses ksonnet\* to manage deployments, so you must install it
before setting up Kubeflow.
ksonnet was added to the :command:`cloud-native-basic` bundle in |CL| version 27550. If
you are using an older |CL| version (not recommended), you must manually
install ksonnet as described below.
ksonnet was added to the :command:`cloud-native-basic` bundle in |CL| version
27550. If you are using an older |CL| version (not recommended), you must
manually install ksonnet as described below.
On |CL|, follow these steps:
@@ -294,7 +338,8 @@ Run a TFJob
Replace <docker_name> with the image name you specified in previous steps.
#. Generate Kubernetes manifests for the workloads and apply them using these commands:
#. Generate Kubernetes manifests for the workloads and apply them using these
commands:
.. code-block:: bash
@@ -372,7 +417,8 @@ A new, blank notebook is displayed, with a cell ready for input.
:alt: New blank notebook
To verify that PyTorch is working, copy the following snippet into the blank cell, and run the cell.
To verify that PyTorch is working, copy the following snippet into the blank
cell, and run the cell.
.. code-block:: console
@@ -405,7 +451,6 @@ Related topics
* :ref:`kubernetes` tutorial
* `Jupyter Notebook`_
.. _TensorFlow: https://www.tensorflow.org/
.. _Kubeflow: https://www.kubeflow.org/
@@ -446,6 +491,10 @@ Related topics
.. _Jupyter Notebook: https://jupyter.org/
.. _kubectl overview: https://kubernetes.io/docs/reference/kubectl/overview/
.. _launcher.py: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dlrs/kubeflow
.. _Release notes on Github\*: https://github.com/clearlinux/dockerfiles/blob/master/stacks/dlrs/releasenote.md
.. _Intel® quantization tools: https://github.com/IntelAI/tools/blob/master/tensorflow_quantization/README.md#quantization-tools
+1
View File
@@ -28,3 +28,4 @@ Explore our tutorials to discover what you can do with |CL|!
nvidia
dars
redis
tutorial-proxy