Update index.rst

Update to the new IRC channel

requirements.txt

@@ -4,7 +4,7 @@ docutils==0.14
 sphinx_rtd_theme
 sphinx-intl==2.0.0
 sphinx-sitemap==1.0.2
-Jinja2==2.10.1
+Jinja2==2.11.3
 GitPython==3.0.8
 sphinx-tabs
 sphinx-copybutton

source/FAQ/index.rst

@@ -226,8 +226,8 @@ ZFS is not available with |CL| because of copyright and licensing
 complexities. BTRFS is an alternative filesystem that is available in |CL|
 natively.
 
-A user on GitHub notes that the `ZFS kernel module can be compiled, built, and
-installed manually <https://github.com/clearlinux/distribution/issues/631>`_.
+A community contributed tutorial has been shared on how to :ref:`manually
+install ZFS <zfs>`.
 
 |
 

source/conf.py

@@ -69,7 +69,7 @@ master_doc = 'index'
 # General information about the project.
 #project = u'Clear Linux* project'
 project = u'Clear Linux* Project Docs'
-copyright = u'2020 Intel Corporation. All Rights Reserved.'
+copyright = u'2021 Intel Corporation. All Rights Reserved.'
 author = u'many'
 
 # The version info for the project you're documenting, acts as replacement for

source/guides/clear/k8s-migration.rst

@@ -0,0 +1,321 @@
+.. _kubernetes-migration:
+
+Kubernetes\* migration
+######################
+
+This guide describes how to migrate `Kubernetes container orchestration system`_ on |CL-ATTR| from 1.17.x to 1.19.x.
+
+.. contents::
+   :local:
+   :depth: 1
+
+Background
+**********
+
+The version of Kubernetes\* was bumped from 1.17.7 to 1.19.4 in |CL-ATTR|
+release 34090. This guide and the |CL| bundle `k8s-migration` were created
+to help facilitate migration of a cluster from 1.17.x to the latest 1.19.x .
+
+The new |CL| bundle `k8s-migration` was added in |CL-ATTR| release 34270.
+
+Prerequisites
+*************
+
+* Make sure you check any updates to kubernetes upgrade doc for caveats related to the version that is running in the cluster.
+* Make sure ALL the nodes are in Ready state. Without that, the cluster cannot be upgraded.
+  Either fix the broken nodes or remove them from the cluster.
+
+.. contents::
+   :local:
+   :depth: 1
+
+Upgrade 1.17.x ---> 1.18.15
+***************************
+
+#. Upgrade Control Node to 1.18.15 first
+
+   First step would be to upgrade one of the main control node and
+   update kubernetes components on them. You will need to have a newer
+   version of :command:`kubeadm` for the upgrade to work. Please consult
+   `kubeadm upgrade guide`_
+   for any caveats from your current version to the new one.
+
+   Update |CL| to the latest release to update the kubernetes version.
+
+   .. code-block:: bash
+
+      sudo -E swupd update
+
+   .. note::
+      Note: PLEASE DO NOT REBOOT YOUR SYSTEM AT THIS TIME. |CL| is awesome and
+      your stuff will work just fine.
+
+#. Add the new Kubernetes migration bundle which contains the 1.18.15 binaries.
+
+   .. code-block:: bash
+
+      sudo -E swupd bundle-add k8s-migration
+
+#. Find the upgrade version of kubeadm that can used. This should be 1.18.15.
+
+   This command will show the command and possible jumps that can be made from the current kubernetes version.
+
+   .. code-block:: bash
+
+      sudo -E /usr/k8s-migration/bin/kubeadm upgrade plan
+
+   Sample output:
+
+   .. code-block:: console
+
+      [upgrade/config] Making sure the configuration is correct:
+      [upgrade/config] Reading configuration from the cluster...
+      [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
+      [preflight] Running pre-flight checks.
+      [upgrade] Running cluster health checks
+      [upgrade] Fetching available versions to upgrade to
+      [upgrade/versions] Cluster version: v1.17.17
+      [upgrade/versions] kubeadm version: v1.18.15
+      I0209 21:12:49.868786  832739 version.go:252] remote version is much newer: v1.20.2; falling back to: stable-1.18
+      [upgrade/versions] Latest stable version: v1.18.15
+      [upgrade/versions] Latest stable version: v1.18.15
+      [upgrade/versions] Latest version in the v1.17 series: v1.17.17
+      [upgrade/versions] Latest version in the v1.17 series: v1.17.17
+
+      Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
+      COMPONENT   CURRENT       AVAILABLE
+      Kubelet     3 x v1.17.7   v1.18.15
+
+      Upgrade to the latest stable version:
+
+      COMPONENT            CURRENT    AVAILABLE
+      API Server           v1.17.17   v1.18.15
+      Controller Manager   v1.17.17   v1.18.15
+      Scheduler            v1.17.17   v1.18.15
+      Kube Proxy           v1.17.17   v1.18.15
+      CoreDNS              1.6.5      1.6.7
+      Etcd                 3.4.3      3.4.3-0
+
+      You can now apply the upgrade by executing the following command:
+
+        kubeadm upgrade apply v1.18.15
+
+#. Upgrade the node to the intermediate 1.18.15 version of Kubernetes.
+
+   .. code-block:: bash
+
+      sudo -E /usr/k8s-migration/bin/kubeadm upgrade apply v1.18.15
+
+   .. note::
+      Note: Do **not** reboot the system yet.
+
+#. Upgrade Additional Control Nodes to 1.18.15
+
+   In multi-node control plane, verify all the control plane nodes are updated prior to upgrading the worker nodes/SUTs.
+
+#. Upgrade Other Nodes to 1.18.15
+
+   For each of the other nodes:
+
+   a. Update |CL| to the latest release to update the kubernetes version.
+
+      .. code-block:: bash
+
+         sudo -E swupd update
+
+   #. Add the new Kubernetes migration bundle which contains the 1.18.15 binaries.
+
+      .. code-block:: bash
+
+         sudo -E swupd bundle-add k8s-migration
+
+   #. On the **Admin node**, drain the Client node *FIRST*
+
+      .. code-block:: bash
+
+         /usr/k8s-migration/bin/kubectl drain <CLIENT_NODE_NAME> --ignore-daemonsets --delete-local-data
+
+   #. Back on the **Client node**, upgrade Kubernetes on the Client
+
+      .. code-block:: bash
+
+         sudo -E /usr/k8s-migration/bin/kubeadm upgrade node
+
+   #. On the **Admin node**, re-enable the Client
+
+      .. code-block:: bash
+
+         /usr/k8s-migration/bin/kubectl uncordon <CLIENT_NODE_NAME>
+
+
+   #. Back on the **Client node**, restart Kubernetes on the Client
+
+      .. code-block:: bash
+
+         sudo -E systemctl restart kubelet
+
+#. Restart Kubernetes on the Admin node(s) to finish the 1.18.x upgrade
+
+   .. code-block:: bash
+
+      sudo -E systemctl restart kubelet
+
+   .. note::
+      Note: Wait for all nodes to be Ready and showing the 1.19.x version.
+      This version will now show as it is the released version the
+      service files will see and use, but the Nodes are *not* upgraded yet.
+
+Upgrade 1.18.15 ---> 1.19.x
+***************************
+
+#. Upgrade Control Node to 1.19.x
+
+   Now that systems are upgraded to the intermediate release of 1.18.15
+   each of the nodes can be upgraded to the latest 1.19.x release.
+
+#. Find the upgrade version of kubeadm that can used. This should be 1.19.x.
+
+   This command will show the command and possible jumps that can be made from the current kubernetes version.
+
+   .. code-block:: bash
+
+      sudo -E kubeadm upgrade plan
+
+   Sample output:
+
+   .. code-block:: console
+
+      [upgrade/config] Making sure the configuration is correct:
+      [upgrade/config] Reading configuration from the cluster...
+      [upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
+      [preflight] Running pre-flight checks.
+      [upgrade] Running cluster health checks
+      [upgrade] Fetching available versions to upgrade to
+      [upgrade/versions] Cluster version: v1.18.15
+      [upgrade/versions] kubeadm version: v1.19.7
+      I0209 23:08:23.810900  925910 version.go:252] remote version is much newer: v1.20.2; falling back to: stable-1.19
+      [upgrade/versions] Latest stable version: v1.19.7
+      [upgrade/versions] Latest stable version: v1.19.7
+      [upgrade/versions] Latest version in the v1.18 series: v1.18.15
+      [upgrade/versions] Latest version in the v1.18 series: v1.18.15
+
+      Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
+      COMPONENT   CURRENT       AVAILABLE
+      kubelet     3 x v1.17.7   v1.19.7
+
+      Upgrade to the latest stable version:
+
+      COMPONENT                 CURRENT    AVAILABLE
+      kube-apiserver            v1.18.15   v1.19.7
+      kube-controller-manager   v1.18.15   v1.19.7
+      kube-scheduler            v1.18.15   v1.19.7
+      kube-proxy                v1.18.15   v1.19.7
+      CoreDNS                   1.6.7      1.7.0
+      etcd                      3.4.3-0    3.4.13-0
+
+      You can now apply the upgrade by executing the following command:
+
+        kubeadm upgrade apply v1.19.7
+
+      The table below shows the current state of component configs as understood by this version of kubeadm.
+      Configs that have a "yes" mark in the "MANUAL UPGRADE REQUIRED" column require manual config upgrade or
+      resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually
+      upgrade to is denoted in the "PREFERRED VERSION" column.
+
+      API GROUP                 CURRENT VERSION   PREFERRED VERSION   MANUAL UPGRADE REQUIRED
+      kubeproxy.config.k8s.io   v1alpha1          v1alpha1            no
+      kubelet.config.k8s.io     v1beta1           v1beta1             no
+
+#. Upgrade the node to the latest 1.19.x version of Kubernetes.
+
+   .. code-block:: bash
+
+      sudo -E /usr/bin/kubeadm upgrade apply v1.19.7
+
+   .. note::
+
+      Note: Do **not** reboot the system yet.
+
+#. Upgrade Additional Control Nodes to 1.19.x
+
+   In multi-node control plane, verify all the control plane nodes are updated prior to upgrading the worker nodes/SUTs.
+
+#. Upgrade Other Nodes to 1.19.x
+
+   For each of the other nodes:
+
+   a. On the **Admin node**, drain the Client *FIRST*
+
+      .. code-block:: bash
+
+         kubectl drain <CLIENT_NODE_NAME> --ignore-daemonsets
+
+   #. Back on the **Client node**, upgrade Kubernetes on the Client
+
+      .. code-block:: bash
+
+         sudo -E kubeadm upgrade node
+
+   #. On the **Admin node**, re-enable the Client
+
+      .. code-block:: bash
+
+         kubectl uncordon <CLIENT_NODE_NAME>
+
+   #. Back on the **Client node**, if you wish reboot the Client, it is now safe to do so.
+
+      .. code-block:: bash
+
+         sudo reboot
+
+#. Reboot the Control Node (optional)
+
+  *If you wish reboot the nodes, it is now safe to do so.*
+
+   .. code-block:: bash
+
+      sudo reboot
+
+**Congratulations!**
+
+You've successfully installed and set up Kubernetes in |CL| using CRI-O and kata-runtime. You are now ready to follow on-screen instructions to deploy a pod network to the cluster and join worker nodes with the displayed token and IP information.
+
+Clean up: Remove the migration bundle for each node
+
+.. code-block:: bash
+
+   sudo -E swupd bundle-remove k8s-migration
+
+Related topics
+**************
+
+Read the Kubernetes documentation to learn more about:
+
+*  `Kubernetes tutorial <tutorials/kubernetes>`_
+
+*  `Kubernetes best practices <tutorials/kubernetes-bp>`_
+
+*  Deploying Kubernetes with a `cloud-native-setup`_
+
+* `Understanding basic Kubernetes architecture`_
+
+* `Deploying an application to your cluster`_
+
+* Installing a `pod network add-on`_
+
+* `Joining your nodes`_
+
+
+.. _kubeadm upgrade guide: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/
+
+.. _Kubernetes container orchestration system: https://kubernetes.io/
+
+.. _Understanding basic Kubernetes architecture: https://kubernetes.io/docs/user-journeys/users/application-developer/foundational/#section-3
+
+.. _Deploying an application to your cluster: https://kubernetes.io/docs/user-journeys/users/application-developer/foundational/#section-2
+
+.. _pod network add-on: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
+
+.. _Joining your nodes: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#join-nodes
+
+.. _cloud-native-setup: https://github.com/clearlinux/cloud-native-setup/tree/master/clr-k8s-examples

source/guides/clear/performance.rst

@@ -91,7 +91,7 @@ optimize software builds for runtime performance. Some significant flags that
 	Options used to tune generated code with optimized instructions for specific
 	CPU types instead of creating generic code for maximum compatibility.
 
-	|CL| defines it's minimum hardware requirements to be second-generation
+	|CL| defines its minimum hardware requirements to be second-generation
 	Intel® microarchitecture code name Westmere (released in 2010) or later.
 	This enables compiler optimizations that are available only on newer
 	architectures. Whenever possible, |CL| tunes code for the Haswell generation
@@ -238,4 +238,4 @@ Related topics
 * `Boosting Python* from profile-guided to platform-specific optimizations <https://clearlinux.org/news-blogs/boosting-python-profile-guided-platform-specific-optimizations>`_
 * `Transparent use of library packages optimized for Intel® architecture <https://clearlinux.org/news-blogs/transparent-use-library-packages-optimized-intel-architecture>`_
 
-*Intel and the Intel logo are trademarks of Intel Corporation or its subsidiaries.*
+*Intel and the Intel logo are trademarks of Intel Corporation or its subsidiaries.*

source/guides/kernel/kernel-modules-dkms.rst

@@ -64,8 +64,10 @@ The :command:`kernel-native-dkms` bundle also:
    kernel. This is especially important for systems where a successful boot
    relies on a kernel module.
 
+.. _kernel-modules-dkms-install-begin-alt:
+
 Install the :command:`kernel-native-dkms` or :command:`kernel-lts-dkms`
-bundle:
+bundle.
 
 #. Determine which kernel variant is running on |CL|. Only the *native*
    and *lts* kernels are enabled to build and load out-of-tree kernel modules
@@ -93,7 +95,8 @@ bundle:
       sudo swupd bundle-add kernel-lts-dkms
 
 
-#. Update the |CL| bootloader and reboot.
+#. Update the |CL| bootloader and reboot, and 
+   ensure that you can start the new kernel.
 
    .. code-block:: bash
 

source/guides/maintenance/configure-hugepages.rst

@@ -23,7 +23,7 @@ to add hugepages to the system and how to change the default hugepage size.
       
       The active option is enclosed in brackets. In this case, always is active, 
       which means hugepages are enabled for every process. The `madvise` 
-      option means that hugepages are enabled for processes that explicitely 
+      option means that hugepages are enabled for processes that explicitly 
       call `madvise`_. 
 
 #. To check the size of hugepages, run the below command.
@@ -71,4 +71,4 @@ to add hugepages to the system and how to change the default hugepage size.
       sudo clr-boot-manager update
       sudo reboot
 
-   .. _madvise: https://linux.die.net/man/2/madvise
+   .. _madvise: https://linux.die.net/man/2/madvise

source/index.rst

@@ -57,7 +57,7 @@
 
       | `Ask the Clear Linux experts <https://clearlinux.org/community/mailing-list>`_
       | `Clear Linux Forum <https://community.clearlinux.org/>`_
-      | `Freenode IRC: #clearlinux <https://webchat.freenode.net/#clearlinux>`_
+      | `irc.libera.chat IRC: #clearlinux <https://kiwiirc.com/client/irc.libera.chat/?nick=web_guest|?#clearlinux>`_
 
 
 .. container:: video
@@ -75,4 +75,4 @@
    tutorials/index
    reference/index
    FAQ/index
-   collaboration/collaboration
+   collaboration/collaboration

source/reference/system-requirements.rst

@@ -39,7 +39,9 @@ System requirements
 *******************
 
 |CL| requires an x86 64-bit processor which supports Intel® Streaming SIMD
-Extensions 4.2 (Intel® SSE 4.2), and it requires a system that supports UEFI.
+Extensions 4.2 (Intel® SSE 4.2). 
+
+For information on the boot loader, see the `clr-boot-manager readme`_ .
 
 The |CL| installer performs a system compatibility check upon booting. To
 manually verify system compatibility with |CL|, run the :ref:`compatibility
@@ -88,4 +90,6 @@ Network      Active Internet connection
 
 
 *Intel, Intel Core, Xeon, Intel Atom, and the Intel logo are trademarks of
-Intel Corporation or its subsidiaries.*
+Intel Corporation or its subsidiaries.*
+
+.. _clr-boot-manager readme: https://github.com/clearlinux/clr-boot-manager

source/tutorials/apache-spark.rst

@@ -1,142 +0,0 @@
-  .. _spark:
-
-Apache\* Spark
-##############
-
-This tutorial describes how to install, configure, and run Apache Spark on
-|CL-ATTR| on a single machine running the master daemon and a worker daemon.
-
-.. contents::
-   :local:
-   :depth: 1
-
-Description
-***********
-
-Apache Spark is a fast, general-purpose cluster computing system with
-the following features:
-
-*  Provides high-level APIs in Java\*, Scala\*, Python\*, and R\*.
-*  Includes an optimized engine that supports general execution graphs.
-*  Supports high-level tools including Spark SQL, MLlib, GraphX, and Spark
-   Streaming.
-
-
-Prerequisites
-*************
-
-* |CL| installed on your host system.
-
-  For detailed instructions on installing |CL| on a bare metal system, visit
-  the :ref:`bare metal installation guide <bare-metal-install-desktop>`.
-
-* Before installing any new packages, update |CL| with the following command:
-
-  .. code-block:: bash
-
-     sudo swupd update
-
-Install Apache Spark
-********************
-
-Apache Spark is included in the :command:`big-data-basic` bundle. To install the
-framework, run the following command:
-
-.. code-block:: bash
-
-   sudo swupd bundle-add big-data-basic
-
-Configure Apache Spark
-**********************
-
-#. Create the configuration directory:
-
-   .. code-block:: bash
-
-      sudo mkdir /etc/spark
-
-#. Copy the default templates from :file:`/usr/share/defaults/spark` to
-   :file:`/etc/spark`:
-
-   .. code-block:: bash
-
-      sudo cp /usr/share/defaults/spark/* /etc/spark
-
-   .. note:: Since |CL| is a stateless system, you should never modify the
-      files under the :file:`/usr/share/defaults` directory. The software
-      updater overwrites those files.
-
-#. Copy the template files shown below to create custom configuration files:
-
-   .. code-block:: bash
-
-      sudo cp /etc/spark/spark-defaults.conf.template /etc/spark/spark-defaults.conf
-      sudo cp /etc/spark/spark-env.sh.template /etc/spark/spark-env.sh
-      sudo cp /etc/spark/log4j.properties.template /etc/spark/log4j.properties
-
-#. Edit the :file:`/etc/spark/spark-env.sh` file and add the
-   :envvar:`SPARK_MASTER_HOST` variable. Replace the example address below
-   with your localhost IP address. View your IP address using the
-   :command:`hostname -I` command.
-
-   .. code-block:: bash
-
-      SPARK_MASTER_HOST="10.300.200.100"
-
-   .. note:: This optional step enables the master's web user interface to
-      view information needed later in this tutorial.
-
-#. Edit the :file:`/etc/spark/spark-defaults.conf` file and update the
-   :envvar:`spark.master` variable with the `SPARK_MASTER_HOST` address and port
-   `7077`.
-
-   .. code-block:: bash
-
-      spark.master    spark://10.300.200.100:7077
-
-Start the master server and a worker daemon
-*******************************************
-
-#. Start the master server:
-
-   .. code-block:: bash
-
-      sudo /usr/share/apache-spark/sbin/./start-master.sh
-
-#. Start one worker daemon and connect it to the master using the
-   :envvar:`spark.master` variable defined earlier:
-
-   .. code-block:: bash
-
-      sudo /usr/share/apache-spark/sbin/./start-slave.sh spark://10.300.200.100:7077
-
-#. Open an internet browser and view the worker daemon information using
-   the master's IP address and port `8080`:
-
-   .. code-block:: bash
-
-      http://10.300.200.100:8080
-
-Run the Spark wordcount example
-*******************************
-
-#. Run the wordcount example using a file on your local host and output the
-   results to a new file with the following command:
-
-   .. code-block:: bash
-
-      sudo spark-submit /usr/share/apache-spark/examples/src/main/python/wordcount.py ~/Documents/example_file > ~/Documents/results
-
-#. Open an internet browser and view the application information using
-   the master's IP address and port `8080`:
-
-   .. code-block:: bash
-
-      http://10.300.200.100:8080
-
-#. View the results of the wordcount application in the :file:`~/Documents/results` file.
-
-**Congratulations!**
-
-You have successfully installed and set up a standalone Apache Spark cluster,
-and ran a simple wordcount example.

source/tutorials/index.rst

@@ -31,7 +31,6 @@ sorted by difficulty level! Learn about :ref:`how we evaluate tutorials <tutoria
       Moderate
 
       -  :ref:`hadoop`
-      -  :ref:`spark`
       -  :ref:`broadcom`
       -  :ref:`fmv`
       -  :ref:`hpc`

source/tutorials/kubernetes.rst

@@ -115,6 +115,14 @@ running quickly.
 
       sudo systemctl stop docker
       sudo systemctl stop containerd
+      
+
+#. Install git as it's a dependency of the :file:`create_stack.sh`.
+
+   .. code-block:: bash
+
+      sudo swupd bundle-add git
+
 
 #. Run the :file:`create_stack.sh` script to initialize the Kubernetes node
    and setup a container network plugin.

source/tutorials/nvidia.rst

@@ -121,7 +121,7 @@ and sustainable on |CL|.
     
          sudo systemctl daemon-reload
        
-   c. Add the service as a depndency to the |CL| updates trigger causing the
+   c. Add the service as a dependency to the |CL| updates trigger causing the
       service to run after every update.
   
       .. code-block:: bash  

source/tutorials/zfs.rst

@@ -32,29 +32,9 @@ Prerequisites
 Install the DKMS kernel
 =======================
 
-If you do not currently use a DKMS kernel, install it by using one of the options below.
-
-#.  Check whether you have an LTS or native kernel:
-
-    .. code-block:: bash
-  
-       uname -r
-
-    a. If 'native' appears in the kernel name, then install a native
-       kernel with DKMS support:
-
-       .. code-block:: bash
-
-          sudo swupd bundle-add kernel-native-dkms
-
-    #. If 'lts' appears in the kernel name, then install the latest LTS
-       kernel with DKMS support:
-
-       .. code-block:: bash
-
-          sudo swupd bundle-add kernel-lts-dkms
-
-#. Reboot and make sure you can start the new kernel.
+.. include:: ../guides/kernel/kernel-modules-dkms.rst
+   :start-after: kernel-modules-dkms-install-begin-alt:
+   :end-before: kernel-modules-dkms-install-end:
 
 Bundles
 =======