clearlinux/clear-linux-documentation
2
0
Fork 0
mirror of https://github.com/clearlinux/clear-linux-documentation.git synced 2026-08-19 05:37:18 +00:00
Code Issues Packages Projects Releases Wiki Activity

Remove guides/stacks; no longer supported. (#1238)

Browse Source 
Signed-off-by: michael vincerra <michael.vincerra@intel.com>
This commit is contained in:
michael vincerra
2022-02-04 13:46:27 -08:00
committed by GitHub
parent f1411470c9
commit ec0b823a91
7 changed files with 0 additions and 4873 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/guides/stacks/dars.rst
-703
View File
@@ -1,703 +0,0 @@
.. _dars:
Data Analytics Reference Stack
##############################
This guide explains how to use the :abbr:`DARS (Data Analytics Reference Stack)`, and to optionally build your own DARS container image.
Any system that supports Docker\* containers can be used with DARS. The steps
in this guide use |CL-ATTR| as the host system.
.. contents::
:local:
:depth: 1
Overview
********
The Data Analytics Reference Stack (DARS) provides developers and enterprises a straightforward, highly optimized software stack for storing and processing large amounts of data. More detail is available on the `DARS architecture and performance benchmarks`_.
Stack Features
==============
The Data Analytics Reference Stack provides two pre-built Docker images,
available on `Docker Hub`_:
* A |CL|-derived `DARS with OpenBlas`_ stack optimized for `OpenBLAS`_
* A |CL|-derived `DARS with Intel® MKL`_ stack optimized for
`Intel® Math Kernel Library`_ (Intel® MKL)
We recommend you view the latest component versions for each image in the
:file:`releasenote` found in the `Data Analytics Reference Stack`_ GitHub\*
repository. Because |CL| is a rolling distribution, the package version numbers
in the |CL|-based containers may not be the latest released by |CL|.
.. note::
The Data Analytics Reference Stack is a collective work, and each piece
of software within the work has its own license. Please see the
`DARS Terms of Use`_ for more details about licensing and usage of the Data
Analytics Reference Stack.
Using the Docker images
***********************
Launching the Image
===================
#. To use the latest stable DARS images, pull an image
directly from `Docker Hub`_. This example uses the
`DARS with Intel® MKL`_ Docker image.
.. code-block:: bash
docker pull clearlinux/stacks-dars-mkl
#. Once you have downloaded the image, you can run it with this command, which will launch the image and drop you into a bash shell inside the container.
.. code-block:: bash
docker run -it --ulimit nofile=1000000:1000000 --name mkl --network host --rm -i -t <name-of-image>
Command Flags
:command:`--ulimit nofile=` is required in order to increase the allowed number of open files for the Apache Spark\* engine.
:command:`--name` can be any name of your choice. This guide is using `mkl`
:command:`--network host` enables the host machine's IP address to be used to access the container.
If you need to verify the name of the DARS image for the <name-of-image> flag, you can use the :command:`docker image ls` command to see which images reside on your system.
.. code-block:: bash
docker image ls
.. code-block:: console
REPOSITORY TAG IMAGE ID CREATED SIZE
clearlinux/stacks-dars-mkl test-img 49a70a22231f 23 hours ago 2.66GB
ubuntu latest 2ca708c1c9cc 7 days ago 64.2MB
katadocker/kata-deploy latest bd6dc92f8060 7 days ago 673MB
clearlinux/stacks-dars-mkl latest 2c9555536d5f 4 weeks ago 2.62GB
.. note::
All of the DARS components are compiled on Open JDK11\*. The container will have preinstalled JDK11 at :file:`/usr/lib/jvm/java-1.11.0-openjdk/` and it has been set as the default Java version. While the DARS containers also contain Open JDK8, it is not covered in this guide.
Building DARS images
====================
If you choose to build your own DARS container images, you can customize them as needed. Use the :file:`Dockerfile` included in the Github\* repository as your baseline.
To construct images with |CL|, start with a |CL| development platform that has the :command:`containers-basic-dev` bundle installed. Learn more about bundles and installing them by using :ref:`swupd-guide`.
#. The `Data Analytics Reference Stack`_ is part of the Intel stacks GitHub\* repository. Clone the :file:`stacks` repository.
.. code-block:: bash
git clone https://github.com/intel/stacks.git
#. Inside the :file:`stacks/dars/clearlinux/mkl` directory, use docker with the :file:`Dockerfile` to build the MKL image.
.. code-block:: bash
cd ./stacks/dars/clearlinux/mkl
docker build --no-cache -t clearlinux/stacks-dars-mkl .
#. Once completed, check the resulting images with :command:`Docker`
.. code-block:: bash
docker images | grep dars
#. You can use any of the resulting images to launch fully functional containers. If you need to customize the containers, you can edit the provided :file:`Dockerfile`.
.. note::
The environment variables for Apache Hadoop* and Apache Spark have been configured in the Dockerfile for the DARS container. For Apache Hadoop\* use :file:`/etc/hadoop` as `HADOOP_CONF_DIR` folder. For Apache Spark use :file:`/etc/spark` as `SPARK_CONF_DIR` folder.
Using Apache Spark\* in DARS
****************************
After launching the container, you can start Apache Spark with either the Scala or PySpark environment. For these examples we will use PySpark, which is the Python\* API for Apache Spark.
.. code-block:: bash
pyspark
Launching is as simple as this. Depending on your system configuration and capabilities, you may need to define proxy or memory allocation settings on the command line or in a config file for optimal performance. Refer to the `Apache Spark documentation`_ for more detail.
After executing :command:`pyspark`, you will see output similar to this.
.. code-block:: console
root@fd5155b89857 /root # pyspark
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
Using Python version 3.7.4 (default, Jul 13 2019 06:59:17)
SparkSession available as 'spark'.
>>>
Execute code directly in PySpark
================================
A simple example for verifying that pyspark is working correctly is to run a small python function from a `PySpark getting started guide`_ to estimate the value of Pi. Run these lines in the PySpark shell.
.. code-block:: console
import random
NUM_SAMPLES = 100000000
def inside(p):
x, y = random.random(), random.random()
return x*x + y*y < 1
count = sc.parallelize(range(0, NUM_SAMPLES)).filter(inside).count()
pi = 4 * count / NUM_SAMPLES
print(“Pi is roughly”, pi)
Run Python programs with spark-submit
=====================================
You can also run python scripts in Apache Spark from the command line. We'll use the Apache Spark example found in the :file:`/usr/share/apache-spark/examples/src/main/python/pi.py` file. Note that we have turned off the INFO and WARN messages in Apache Spark for this example.
.. code-block:: console
#spark-submit /usr/share/apache-spark/examples/src/main/python/pi.py
Config directory: /usr/share/defaults/spark/
Pi is roughly 3.134700
DARS Usecase example
====================
The DARS container is used in conjunction with the Deep Learning Reference Stack container to implement a real world use case. Refer to the `Github Issue Classification`_ Usecase found in the `stacks-usecase`_ repository for a walkthrough. This usecase is implemented using the Scala environment, rather than PySpark.
Using Apache Hadoop in DARS
***************************
Apache Hadoop is an open source framework allowing for distributed processing of large data sets across clusters of computers using simple programming models. This framework is designed to scale up from a few servers to thousands of machines, each offering local computation and storage.
Single Node Hadoop Cluster Setup
================================
In this mode, all the daemons involved (e.g., the DataNode, NameNode, TaskTracker, JobTracker) run as Java processes on the same machine. This setup is useful for developing and testing Apache Hadoop applications.
The components of an Apache Hadoop Cluster are described below:
* NameNode manages HDFS storage. HDFS exposes a filesystem namespace and allows user data to be stored in files. Internally a file is split into one or more blocks and these blocks are stored in a set of DataNodes.
* DataNode is also known as Slave node. It is responsible for storing and managing the data in that node and responds to the NameNode for all filesystem operations.
* JobTracker is a master which creates and runs the job through tasktrackers. It also tracks resource availability and task lifecycle management.
* TaskTracker manages the processing resources on each worker node and send status updates to the JobTracker periodically.
Configuration
=============
#. To setup a single node cluster, run a DARS container with the following flags:
.. code-block:: bash
docker run --ulimit nofile=1000000:1000000 -ti --rm --network host clearlinux/stacks-dars-mkl cp -r -n /usr/share/defaults/hadoop/* /etc/hadoop
#. In the running container, set configuration in the :file:`/etc/hadoop/mapred-site.xml` file
.. code-block:: xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=${HADOOP_HOME}</value>
</property>
</configuration>
#. Set up the :file:`/etc/hadoop/yarn-site.xml` as follows
.. code-block:: xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
Start the Apache Hadoop daemons
===============================
#. Format the NameNode server using this command:
.. code-block:: bash
hdfs namenode -format
#. Start the Apache Hadoop services
HDFS Namenode service :
.. code-block:: bash
hdfs --daemon start namenode
HDFS Datanode service :
.. code-block:: bash
hdfs --daemon start datanode
Yarn ResourceManager :
.. code-block:: bash
yarn --daemon start resourcemanager
Yarn NodeManager :
.. code-block:: bash
yarn --daemon start nodemanager
jobhistory service :
.. code-block:: bash
mapred --daemon start historyserver
#. Verify the nodes are alive with this command:
.. code-block:: bash
yarn node -list 2
Your output will look similar to:
.. code-block:: console
Total Nodes:1
Node-Id Node-State Node-Http-Address Number-of-Running-Containers
<hostname>:43489 RUNNING <hostname>:8042 0
Example application
===================
Apache Hadoop comes packages with a set of example applications. In this example we will show how to use the cluster to calculate Pi. The JAR file containing the compiled class can be found on your running DARS container at :file:`/usr/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar`
.. code-block:: bash
hadoop jar /usr/share/hadoop/mapreduce/hadoop-mapreduce-examples-$(hadoop version | grep Hadoop | cut -d ' ' -f2).jar pi 16 100
Deploy DARS on Kubernetes\*
***************************
Many containerized workloads are deployed in clusters managed by orchestration software like Kubernetes.
Prerequisites
=============
* A running Kubernetes cluster at version >= 1.6 with access configured to it using kubectl.
* You must have appropriate permissions to list, create, edit and delete pods in your cluster.
* The service account credentials used by the driver pods must be allowed to create pods, services and configmaps.
* You must have Kubernetes DNS configured in your cluster.
.. note::
To ensure that Kubernetes is correctly installed and configured for |CL|, follow the instructions in :ref:`kubernetes`.
#. For this example we will create the following Dockerfile
.. code-block:: bash
cat > $(pwd)/Dockerfile << 'EOF'
ARG DERIVED_IMAGE
FROM ${DERIVED_IMAGE}
RUN mkdir -p /etc/passwd /etc/pam.d /opt/spark/conf /opt/spark/work-dir
RUN set -ex && \
rm /bin/sh && \
ln -sv /bin/bash /bin/sh && \
touch /etc/pam.d/su \
echo "auth required pam_wheel.so use_uid" >> /etc/pam.d/su && \
chgrp root /etc/passwd && chmod ug+rw /etc/passwd
RUN ln -s /usr/share/apache-spark/jars/ /opt/spark/ && \
ln -s /usr/share/apache-spark/bin/ /opt/spark/ && \
ln -s /usr/share/apache-spark/sbin/ /opt/spark/ && \
ln -s /usr/share/apache-spark/examples/ /opt/spark/ && \
ln -s /usr/share/apache-spark/kubernetes/tests/ /opt/spark/ && \
ln -s /usr/share/apache-spark/data/ /opt/spark/ && \
ln -s /etc/spark/* /opt/spark/conf/
COPY entrypoint.sh /opt/
ENV JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk
ENV PATH="${JAVA_HOME}/bin:${PATH}"
ENV SPARK_HOME /opt/spark
WORKDIR /opt/spark/work-dir
ENTRYPOINT [ "/opt/entrypoint.sh" ]
EOF
#. Create the :file:`entrypoint.sh` file. The Dockerfile requires an entrypoint script, to allow spark-submit to interact with the container.
.. code-block:: bash
cat > $(pwd)/entrypoint.sh << 'EOF'
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# echo commands to the terminal output
set -ex
# Check whether there is a passwd entry for the container UID
myuid=$(id -u)
mygid=$(id -g)
# turn off -e for getent because it will return error code in anonymous uid case
set +e
uidentry=$(getent passwd $myuid)
set -e
# If there is no passwd entry for the container UID, attempt to create one
if [ -z "$uidentry" ] ; then
if [ -w /etc/passwd ] ; then
echo "$myuid:x:$myuid:$mygid:anonymous uid:$SPARK_HOME:/bin/false" >> /etc/passwd
else
echo "Container ENTRYPOINT failed to add passwd entry for anonymous UID"
fi
fi
SPARK_K8S_CMD="$1"
case "$SPARK_K8S_CMD" in
driver | driver-py | driver-r | executor)
shift 1
;;
"")
;;
*)
echo "Non-spark-on-k8s command provided, proceeding in pass-through mode..."
exec /sbin/tini -s -- "$@"
;;
esac
SPARK_CLASSPATH="$SPARK_CLASSPATH:${SPARK_HOME}/jars/*"
env | grep SPARK_JAVA_OPT_ | sort -t_ -k4 -n | sed 's/[^=]*=\(.*\)/\1/g' > /tmp/java_opts.txt
readarray -t SPARK_EXECUTOR_JAVA_OPTS < /tmp/java_opts.txt
if [ -n "$SPARK_EXTRA_CLASSPATH" ]; then
SPARK_CLASSPATH="$SPARK_CLASSPATH:$SPARK_EXTRA_CLASSPATH"
fi
if [ -n "$PYSPARK_FILES" ]; then
PYTHONPATH="$PYTHONPATH:$PYSPARK_FILES"
fi
PYSPARK_ARGS=""
if [ -n "$PYSPARK_APP_ARGS" ]; then
PYSPARK_ARGS="$PYSPARK_APP_ARGS"
fi
R_ARGS=""
if [ -n "$R_APP_ARGS" ]; then
R_ARGS="$R_APP_ARGS"
fi
if [ "$PYSPARK_MAJOR_PYTHON_VERSION" == "2" ]; then
pyv="$(python -V 2>&1)"
export PYTHON_VERSION="${pyv:7}"
export PYSPARK_PYTHON="python"
export PYSPARK_DRIVER_PYTHON="python"
elif [ "$PYSPARK_MAJOR_PYTHON_VERSION" == "3" ]; then
pyv3="$(python3 -V 2>&1)"
export PYTHON_VERSION="${pyv3:7}"
export PYSPARK_PYTHON="python3"
export PYSPARK_DRIVER_PYTHON="python3"
fi
case "$SPARK_K8S_CMD" in
driver)
CMD=(
"$SPARK_HOME/bin/spark-submit"
--conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS"
--deploy-mode client
"$@"
)
;;
driver-py)
CMD=(
"$SPARK_HOME/bin/spark-submit"
--conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS"
--deploy-mode client
"$@" $PYSPARK_PRIMARY $PYSPARK_ARGS
)
;;
driver-r)
CMD=(
"$SPARK_HOME/bin/spark-submit"
--conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS"
--deploy-mode client
"$@" $R_PRIMARY $R_ARGS
)
;;
executor)
CMD=(
${JAVA_HOME}/bin/java
"${SPARK_EXECUTOR_JAVA_OPTS[@]}"
-Xms$SPARK_EXECUTOR_MEMORY
-Xmx$SPARK_EXECUTOR_MEMORY
-cp "$SPARK_CLASSPATH"
org.apache.spark.executor.CoarseGrainedExecutorBackend
--driver-url $SPARK_DRIVER_URL
--executor-id $SPARK_EXECUTOR_ID
--cores $SPARK_EXECUTOR_CORES
--app-id $SPARK_APPLICATION_ID
--hostname $SPARK_EXECUTOR_POD_IP
)
;;
*)
echo "Unknown command: $SPARK_K8S_CMD" 1>&2
exit 1
esac
# Execute the container CMD
exec "${CMD[@]}"
EOF
#. Make :file:`entrypoint.sh` executable
.. code-block:: bash
sudo chmod +x $(pwd)/entrypoint.sh
#. Build the Docker image, for this example we will use dars_k8s_spark for the name of the image.
.. code-block:: bash
docker build . --build-arg DERIVED_IMAGE=clearlinux/stacks-dars-mkl -t dars_k8s_spark
#. Verify your built image. Execute the following command looking for the given name dars_k8s_spark
.. code-block:: bash
docker images | grep "dars_k8s_spark"
You should see something like:
.. code-block:: console
dars_k8s_spark latest 1fa3278a3421 1 minutes ago 6.56GB
#. Use a variable to store the image's given name:
.. code-block:: bash
DARS_K8S_IMAGE=dars_k8s_spark
Configure RBAC
==============
Create the Spark service account and cluster role binding to allow Spark on Kubernetes to create Executors as required. For this example use the default namespace.
.. code-block:: bash
kubectl create serviceaccount spark-serviceaccount --namespace default
kubectl create clusterrolebinding spark-rolebinding --clusterrole=edit --serviceaccount=default:spark-serviceaccount --namespace=default
Prepare to Submit the Spark Job
===============================
#. Determine the Kubernetes master address:
.. code-block:: bash
kubectl cluster-info
You should see something like:
.. code-block:: console
Kubernetes master is running at https://192.168.39.127:8443
#. Use a variable to store the master address:
.. code-block:: bash
MASTER_ADDRESS='https://192.168.39.127:8443'
#. Submit the Spark Job on Minikube using the MASTER_ADDRESS and DARS_K8S variables. The driver pod will be called spark-pi-driver.
.. code-block:: bash
spark-submit \
--master k8s://${MASTER_ADDRESS} \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=2 \
--conf spark.kubernetes.container.image=${DARS_K8S_IMAGE} \
--conf spark.kubernetes.driver.pod.name=spark-pi-driver \
--conf spark.kubernetes.namespace=default \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-serviceaccount \
local:///usr/share/apache-spark/examples/jars/spark-examples_2.12-2.4.0.jar
#. Check the Job. Read the logs and look for the Pi result:
.. code-block:: bash
kubectl logs spark-pi-driver | grep "Pi is roughly"
You should see something like:
.. code-block:: console
Pi is roughly 3.1418957094785473
More information about spark-submit configuration is available in the `running-on-kubernetes`_ documentation.
Troubleshooting
***************
Dropped or refused connection
=============================
If Pyspark / Spark-shell warns of a dropped connection exception or Connection refused, check if the `HADOOP_CONF_DIR` environment variable is set. These APIs assume they will use Hadoop Distributed File System.
You can unset `HADOOP_CONF_DIR` and use Spark RDDs, or start Hadoop services and then create your directories and files as required using hdfs.
It is also possible to change the file system to local without unsetting `HADOOP_CONF_DIR` using one of these commands.
.. code-block:: bash
pyspark --conf "spark.hadoop.fs.defaultFS=file:///"
.. code-block:: bash
spark-shell --conf "spark.hadoop.fs.defaultFS=file:///"
Using Spark with proxy settings
===============================
There are two ways to work with proxies:
#. Add the following line to :file:`$SPARK_CONF_DIR/spark-defaults.conf` for both `spark.executor.extraJavaOptions` and `spark.driver.extraJavaOptions` variables:
.. code-block:: console
-Dhttp.proxyHost=<URL> -Dhttp.proxyPort=<PORT> -Dhttps.proxyHost=<URL> -Dhttps.proxyPort=<PORT>
#. Give the proxies URL and Port as a configuration parameter
.. code-block:: bash
pyspark --conf "spark.hadoop.fs.defaultFS=file:///" --conf "spark.driver.extraJavaOptions=-Dhttp.proxyHost=example.proxy -Dhttp.proxyPort=111 -Dhttps.proxyHost=example.proxy -Dhttps.proxyPort=112"
.. code-block:: bash
spark-shell --conf "spark.hadoop.fs.defaultFS=file:///" --conf "spark.driver.extraJavaOptions=-Dhttp.proxyHost=example.proxy -Dhttp.proxyPort=111 -Dhttps.proxyHost=example.proxy -Dhttps.proxyPort=112"
Known issues
============
#. There is an exception message `Unrecognized Hadoop major version number: 3.2.0 at org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion.`
This exception can be disregarded because DARS does not use hadoop.hive.shims. Hive binaries installed from Apache on |CL| with JDK11 does not work at the time of this writing.
#. There is an exception message `Exception in thread "Thread-3" java.lang.ExceptionInInitializerError at org.apache.hadoop.hive.conf.HiveConf` This is related to the same issue with |CL| and JDK11 noted above, and does not affect DARS for the same reason.
*Intel and the Intel logo are trademarks of Intel Corporation or its subsidiaries.*
.. _Data Analytics Reference Stack: https://github.com/intel/stacks/tree/master/dars/clearlinux
.. _Docker Hub: https://hub.docker.com/
.. _OpenBLAS: http://www.openblas.net/
.. _Intel® Math Kernel Library: https://software.intel.com/en-us/mkl
.. _CentOS: https://www.centos.org/
.. _DARS with OpenBLAS: https://hub.docker.com/r/clearlinux/stacks-dars-openblas/
.. _DARS with Intel® MKL: https://hub.docker.com/r/clearlinux/stacks-dars-mkl/
.. _DARS architecture and performance benchmarks: https://clearlinux.org/stacks/data-analytics-stack-v1
.. _DARS Terms of Use: https://clearlinux.org/stacks/data-analytics/terms-of-use
.. _PySpark getting started guide: https://towardsdatascience.com/how-to-get-started-with-pyspark-1adc142456ec
.. _Apache Spark documentation: https://spark.apache.org/docs/latest/
.. _stacks-usecase: https://github.com/intel/stacks-usecase
.. _Github Issue Classification: https://github.com/intel/stacks-usecase/tree/master/github-issue-classification
.. _running-on-kubernetes: https://spark.apache.org/docs/latest/running-on-kubernetes.html#configuration
source/guides/stacks/dbrs.rst
-714
View File
@@ -1,714 +0,0 @@
.. _dbrs:
Database Reference Stack
########################
This guide describes the hardware and installation requirements for using the
:abbr:`DBRS (Database Reference Stack)`, along with getting started configuration examples, using |CL-ATTR| as the host system.
.. contents::
:local:
:depth: 1
Overview
********
The Database Reference Stack is integrated, highly-performant, open source,
and optimized for 2nd generation Intel® Xeon® Scalable processors and Intel®
Optane™ persistent memory. This open source community release is part of
an effort to ensure developers have easy access to the features and
functionality of Intel Platforms.
Stack Features
==============
Current supported database applications are Apache Cassandra* and Redis*, which
have been enabled for `Intel Optane PMM`_.
DBRS with Apache Cassandra can be deployed as a standalone container or inside a
Kubernetes* cluster.
The Redis stack application is enabled for a multinode Kubernetes
environment, using AEP PMem DIMM in fsdax mode for storage.
Releases
********
Refer to the `Database Reference Stack website`_ for information and download links for the different versions and offerings of the stack.
The release announcement for each release provides more detail about the stack features, as well as benchmark results.
* `DBRS V2.0`_ release announcement.
* `DBRS V1.0`_ release announcement.
.. note::
The Database Reference Stack is a collective work, and each piece
of software within the work has its own license. Please see the
`DBRS Terms of Use`_ for more details about licensing and usage of the Database Reference Stack.
Hardware Requirements
*********************
* Intel Xeon Scalable platform with Intel® C620 series chipset
* 2nd Gen Intel Xeon Scalable processor CPU (Intel® Optane™ PMem-enabled stepping). Provides cache & memory control. Intel Optane PMem works only on systems powered by 2nd Generation Intel® Xeon® Platinum or Intel® Xeon® Gold processors.
* BIOS with Reference Code
* Intel Optane PMem
Hardware configuration used in stacks development
=================================================
* Intel® Server System R2208WFTZSR
* BIOS with Reference Code
* BIOS ID: SE5C620.86B.0D.01.0438.032620191658
* BMC Firmware: 1.94.6b42b91d
* Intel Optane PMem Firmware: 1.2.0.5310
* 2x Intel Xeon Platinum 8268 Processor
* Intel® SSD Data Center Family S5600 Series 960GB 2.5in SATA Drive
* 64 GB RAM - Distributed in 4x 16 GB DDR4 DIMM's
* 2x Intel Optane PMem 256GB Module
* 1-1-1 Layout 8 Intel Optane : 1 RAM ratio
.. list-table:: **Table 1. IMC**
:widths: 16,16,16,16,16,16
:header-rows: 1
* - Channel 2
- Channel 2
- Channel 1
- Channel 1
- Channel 0
- Channel 0
* - Slot 1
- Slot 0
- Slot 1
- Slot 0
- Slot 1
- Slot 0
* -
- 256 GB DCPMM
-
- 16 GB DRAM
-
- 16 GB DRAM
Firmware configuration
**********************
.. important::
When updating DCPMM Firmware, all DCPMM parts must be in the same mode (you cannot mix 1LM and 2LM parts).
The latest firmware download for the Intel® Server Board S2600WF Family is available at the `Intel Download Center`_
Firmware Update Steps
=====================
#. Unzip the contents of the update package and copy all files to the root directory of a removable media (USB flash drive).
#. Insert the USB flash drive to any available USB port on the system to be updated.
#. Boot to EFI shell.
#. Input "fsx(x:0,1,...):" to enter into your usb disk
#. Run "startup.nsh"
#. After update BMC firmware, system BIOS, ME firmware,FD, FRUSDR, system will reboot automatically.
If Intel Optane PMem is installed, run startup.nsh a second time after the first reboot to upgrade Intel Optane PMem Firmware:
* Boot to EFI shell.
* Input "fsx(x:0,1,...):" to enter into your usb disk
* Run "startup.nsh" again to update the corresponding AEP FW.
.. _dbrs-hardware-configuration:
Hardware Configuration
**********************
Online Resources
================
Before going through the configuration steps, we strongly recommend visiting the following resources and wikis to have a broader understanding of what is being done
* `Quick Start Guide`_ Configure Intel Optane PMem Modules on Linux
* `Managing NVDIMMs`_
* `Configure, Manage, and Profile`_ Intel Optane PMem Modules
Optane™ DIMM Configuration
==========================
The PMem DIMMs can be configured in devdax or fsdax mode. The use case to enable database stack on a kubernetes environment currently only support fsdax mode.
Configuration Steps
===================
.. important::
Run the following steps with root privileges (sudo) as shown in the examples
#. To configure Optane™ DIMMs for App direct mode run this command
.. code-block:: bash
sudo ipmctl create -goal PersistentMemoryType=AppDirect
#. Verify the Optane™ Configuration by showing the defined region, then reboot the system for your changes to take effect
.. code-block:: bash
sudo ipmctl show -region
#. Next, list the defined namespaces for the pmem devices in the system. If they are not defined, create them as shown in the following step.
.. code-block:: bash
sudo ndctl list -N
#. Create namespaces based on the regions and set mode as fsdax -- use the names of the regions listed in previous step as the –-region parameter (default is region0 and region1; one for each CPU socket)
.. code-block:: bash
sudo ndctl create-namespace --region=region0 --mode=fsdax
sudo ndctl create-namespace --region=region1 --mode=fsdax
#. Create the filesystem and mount it. We are using /mnt/dax{#} as a convention in this guide to mount our devices
.. code-block:: bash
sudo mkfs.ext4 /dev/pmem0
sudo mount -o dax /dev/pmem0 /mnt/dax0
sudo mkfs.ext4 /dev/pmem1
sudo mount -o dax /dev/pmem1 /mnt/dax1
Running DBRS with Apache Cassandra*
***********************************
DBRS with Apache Cassandra can be deployed as a standalone container or inside
Kubernetes\*. Instructions for both cases is included here. Note that you can
use the released `Docker image with Apache Cassandra`_ (Docker\* examples below).
These instructions provide a baseline for creating your own container image.
If you are using the released image, skip this section.
.. important::
At the initial release of DBRS, Apache Cassandra is considered to be Engineering Preview release quality and may not be suitable for production release. Please take this into consideration when planning your project.
Build the DBRS with Apache Cassandra container
==============================================
To build the container with Apache Cassandra, you must build cassandra-pmem, and then build the container using the :command:`docker build` command. We are using |CL| as our container host as well as the OS in the container.
Build cassandra-pmem
====================
.. important::
At the initial release of DBRS, the pmem-csi driver is considered to be Engineering Preview release quality and may not be suitable for production release. Please take this into consideration when planning your project.
In the `DBRS github repository`_, there is a file called `build-cassandra-pmem.sh`_, which handles all the requirements for compiling cassandra-pmem for Dockerfile usage. The dependencies for this build can be installed with :command:`swupd`.
.. code-block:: bash
sudo swupd bundle-add c-basic java-basic devpkg-pmdk pmdk
Once installed, we run the script
.. code-block:: bash
./build-cassandra-pmem.sh
At the completion of the build you will have a file called :file:`cassandra-pmem-build.tar.gz`. Place this file in the same directory with the Dockerfile to build the Docker image.
Build the Docker container
==========================
To build the Docker image, run the Dockerfile in the same directory with the :file:`cassandra-pmem-build.tar.gz`
.. code-block:: bash
docker build --force-rm --no-cache -f Dockerfile -t $build_image_name .
Once it completes, the Docker image is ready to be used.
Deploy Apache Cassandra PMEM as a standalone container
======================================================
Requirements
------------
To deploy Apache Cassandra PMEM, you must meet the following requirements
* PMEM memory must be configured in `devdax` or `fsdax` mode. The container image is able to handle both modes and depending on the PMEM mode, the mount points inside the container must be different.
* In order to make available `devdax` pmem devices inside the container you must use the `--device` directive. Internally the container always uses :command:`/dev/dax0.0`, so the mapping should be: :command:`--device=/dev/<host-device>:/dev/dax0.0`
* In a similar fashion for `fsdax` we need the device to be mapped to :command:`/mnt/pmem` inside the container: :command:`--mount type=bind,source=<source-mount-point>,target=/mnt/pmem`
Preparing PMEM for container use
--------------------------------
The cassandra-pmem image is capable of using both `fsdax` and `devdax`, the necessary steps to configure the PMEM to work with cassandra are documented here.
.. tabs::
.. group-tab:: devdax
We need to verify the device we want to use is in `devdax` mode
.. code-block:: bash
sudo ndctl create-namespace -fe namespace0.0 --mode=devdax
.. code-block:: console
{
"dev":"namespace0.0",
"mode":"devdax",
"map":"dev",
"size":"3.94 GiB (4.23 GB)",
"uuid":"cb738cc7-711d-4578-bebf-1f7ba02ca169",
"daxregion":{
"id":0,
"size":"3.94 GiB (4.23 GB)",
"align":2097152,
"devices":[
{
"chardev":"dax0.0",
"size":"3.94 GiB (4.23 GB)"
}
]
},
"align":2097152
}
If needed, we can reconfigure it using :command:`ndctl create-namespace -fe <namespace-name> --mode=devdax`.
Before using a `devdax` device we need to clear the device:
.. code-block:: bash
sudo pmempool rm -vaf /dev/dax0.0
The `jvm.options` configuration for Apache Cassandra should look like the following:
.. code-block:: console
-Dpmem_path=/dev/dax0.0
-Dpool_size=0
Where
* pmem_path is the `devdax` device.
* pool_size=0 indicates to use the entire `devdax` device.
When using the `Docker image with Apache Cassandra`_, the file `jvm.options` is automatically populated.
.. group-tab:: fsdax
Verify that the PMEM is in `fsdax` mode
.. code-block:: bash
sudo ndctl list -u
.. code-block:: console
{
"dev":"namespace0.0",
"mode":"fsdax",
"map":"mem",
"size":"4.00 GiB (4.29 GB)",
"sector_size":512,
"blockdev":"pmem0"
}
If for some reason the device is not in `fsdax` mode you can reconfigure the namespace as follows:
.. code-block:: bash
sudo `ndctl create-namespace -fe <namespace-name> --mode=fsdax`
Once the PMEM namespace is configured, you will see a device named :file:`/dev/pmem{0-9}`. We will create a filesystem on that device. The filesystem could be `ext4` or `xfs`, for this example we are going to use `ext4`.
.. code-block:: bash
sudo mkfs.ext4 /dev/pmem0
.. code-block:: console
mke2fs 1.45.2 (27-May-2019)
Creating filesystem with 1031680 4k blocks and 258048 inodes
Filesystem UUID: 303c03f5-ac4e-4462-8bf9-bc6b0fae53fe
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736
Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done
Once the filesystem is created, we mount it with the dax option
.. code-block:: bash
sudo mount /dev/pmem0 /mnt/pmem -o dax
When using `fsdax` mode cassandra-pmem creates a pool file on the pmem mountpoint, so the `jvm.options` configuration should look like the output below:
.. code-block:: console
-Dpmem_path=/mnt/pmem/cassandra_pool
-Dpool_size=3221225472
Where
* `pmem_path` is the path to the pool file, which should include the path itself and the file name
* `pool_size` is the size of the pool file in bytes. If you are using the `Docker image with Apache Cassandra`_ you can pass this value as an environment variable to the container runtime in Gb and the calculation is done automatically.
Is important to note that when creating the filesystem in the pmem device certain amount of space of the device is used by the filesystem metadata so the pool_size should be smaller than the total pmem namespace size.
When using the `Docker image with Apache Cassandra`_, the file `jvm.options` is automatically populated with the environment variables `CASSANDRA_PMEM_POOL_NAME` and `CASSANDRA_FSDAX_POOL_SIZE_GB`.
Run the DBRS Container
======================
Replace `<image-id>` in the following commands with the name of the image you are using.
.. tabs::
.. group-tab:: devdax
In `devdax` mode:
.. code-block:: bash
docker run --device=/<devdax-device>:/dev/dax0.0 --ulimit nofile=262144:262144 -p 9042:9042 -p 7000:7000 -it --name cassandra-test <image-id>
.. group-tab:: fsdax
In `fsdax` mode:
.. code-block:: bash
docker run --mount type=bind,source=/<fsdax-mountpoint>,target=/mnt/pmem --ulimit nofile=262144:262144 -p 9042:9042 -p 7000:7000 -it -e 'CASSANDRA_FSDAX_POOL_SIZE_GB=<fsdax-pool-size-in-gb>' --name cassandra-test <image-id>
Container Configuration
=======================
Using environment variables
---------------------------
The container listens on the primary container IP address, but if required, some parameters can be provided as environment variables using `--env`.
* `CASSANDRA_CLUSTER_NAME` Cassandra cluster name, by default `Cassandra Cluster`
* `CASSANDRA_LISTEN_ADDRESS` Cassandra listen address
* `CASSANDRA_RPC_ADDRESS` Cassandra RPC address
* `CASSANDRA_SEED_ADDRESSES` A comma separated list of hosts in the cluster, if not provided, cassandra is going to run as a single node.
* `CASSANDRA_SNITCH` The snitch type for the cluster, by default it is `SimpleSnitch`, for more complex snitches you can mount your own `cassandra-rackdc.properties` file.
* `LOCAL_JMX` If set to `no` the JMX service will listen on all IP addresses, the default is `yes` and listens just on localhost 127.0.0.1
* `JVM_OPTS` When set you can pass additional arguments to the JVM for cassandra execution, for example for specifying memory heap sizes `JVM_OPTS=-Xms16G -Xmx16G -Xmn12G`
When using PMEM in `fsdax` mode, there are some parameters to control the allocation of memory:
* `CASSANDRA_FSDAX_POOL_SIZE_GB` The size of the fsdax pool in GB, if it is not specified the pool size is `1`
* `CASSANDRA_PMEM_POOL_NAME` The filename of the pool created in PMEM, by default `cassandra_pool`
Using custom files
------------------
For more complex deployments it is also possible to provide custom `cassandra.yaml` and `jvm.options` files as shown below:
.. code-block:: bash
docker run --mount type=bind,source=/<fsdax-mountpoint>,target=/mnt/pmem -it --ulimit nofile=262144:262144 --mount type=bind,source=/<path-to-file>/cassandra.yaml,target=/workspace/cassandra/conf/cassandra.yaml --mount type=bind,source=/path-to-file>/jvm.options,target=/workspace/cassandra/conf/jvm.options --name cassandra-custom-files
Clustering
==========
For a simple two node cluster using PMEM in `fsdax` mode on both containers:
Node 1
------
* IP: 172.17.0.2
* PMEM mountpoint: /mnt/pmem1
.. code-block:: bash
docker run --mount type=bind,source=/mnt/pmem1,target=/mnt/pmem --ulimit nofile=262144:262144 -it -e 'CASSANDRA_FSDAX_POOL_SIZE_GB=2' -e 'CASSANDRA_SEED_ADDRESSES=172.17.0.2:7000,172.17.0.3:7000' --name cassandra-node1 <image-id>
Node 2
------
* IP: 172.17.0.3
* PMEM mountpoint: /mnt/pmem2
.. code-block:: bash
docker run --mount type=bind,source=/mnt/pmem2,target=/mnt/pmem --ulimit nofile=262144:262144 -it -e 'CASSANDRA_FSDAX_POOL_SIZE_GB=2' -e 'CASSANDRA_SEED_ADDRESSES=172.17.0.2:7000,172.17.0.3:7000' --name cassandra-node2 <image-id>
Once both nodes are running, eventually the gossip is settled and we can use `nodetool` on either container to check cluster status.
.. code-block:: bash
docker exec -it <container-id> bash /workspace/cassandra/bin/nodetool status
The output should look similar to this:
.. code-block:: console
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 172.17.0.3 0 bytes 256 100.0% 22387159-8192-41cf-8b6c-8bf0e1049eb7 rack1
UN 172.17.0.2 0 bytes 256 100.0% 219b56ba-c07c-400b-a018-a5dc20edeb09 rack1
Persistence
===========
By default you can access the data written to Apache Cassandra as long as the container exists. In order to persist the data past that, you can mount volumes or bind mounts on :file:`/workspace/cassandra/data` and :file:`/workspace/cassandra/logs` and in this way the data can still be accessed once the container is deleted.
Deploy An Apache Cassandra-PMEM cluster on Kubernetes*
******************************************************
Many containerized workloads are deployed in clusters and orchestration software like Kubernetes can be useful. We will use the `cassandra-pmem-helm`_ Helm* chart in this example.
Requirements
============
* Kubectl* must be configured to access the Kubernetes Cluster
* A Kubernetes cluster with `pmem-csi`_ enabled
* The Kubernetes cluster must have `helm`_ and tiller installed
* PMEM hardware
.. important::
When selecting the `fsdax` pool file size, it is important to consider that when requesting a volume, certain amount of space is used by the filesystem metadata on that volume and the available space turns out to be less than total amount specified. Taking this into consideration the size of the fsdax pool file should be ~2G less than the total volume size requested.
Configuration
=============
In order to configure the Apache Cassandra PMEM cluster some variables and values are provided. These values are set in :file:`test/cassandra-pmem-helm/values.yaml`, and can be modified according to your specific needs. A summary of those parameters is shown below:
* clusterName: The cluster Name set across all deployed nodes
* replicaCount: The number of nodes in the cluster to be deployed
* image.repository: The address of the container registry where the cassandra-pmem image should be pulled
* image.tag: The tag of the image to be pulled during deployment
* image.name: The name of the image to be pulled during deployment
* pmem.containerPmemAllocation: The size of the persistent volume claim to be used as heap, it uses the storage class `pmem-csi-sc-ext4` from pmem-csi The size of the fsdax pool to be created inside the persistent volume claim, in practice it should be `1G` less than pmem.containerPmemAllocation
* pmem.fsdaxPoolSizeInGB: The size of the fsdax pool to be created inside the persistent volume claim, in practice it should be 1G less than pmem.containerPmemAllocation
* enablePersistence: If set to `true`, K8s persistent volumes are deployed to store data and logs
* persistentVolumes.logsVolumeSize: The size of the persistent volume used for storing logs on each node, the default is `4G`
* persistentVolumes.dataVolumeSize: The size of the persistent volume used for storing data on each node, the default is `4G`
* persistentVolumes.logsStorageClass: Storage class used by the logs pvc, by default it uses `pmem-csi-sc-ext4`
* persistentVolumes.dataStorageClass: Storage class used by the data pvc, by default it uses `pmem-csi-sc-ext4`
* provideCustomConfig: If set to `true`, it mounts all the files located on `<helm-chart-dir>/files/conf` on `/workspace/cassandra/conf` inside each container in order to provide a way to customize the deployment beyond the options provided here
* exposeJmxPort: When set to `true` it exposes the JMX port as part of the Kubernetes headless service. It should be used together with `enableAdditionalFilesConfigMap` in order to provide authentication files needed for JMX when the remote connections are allowed. When set to `false` only local access through 127.0.0.1 is granted and no additional authentication is needed.
* enableClientToolsPod: If set to `true`, an additional pod independent from the cluster is deployed, this pod contains various Cassandra client tools and mounts test profiles located under `<helm-chart-dir>/files/testProfiles` to `/testProfiles` inside the pod. This pod is useful to test and launch benchmarks
* enableAdditionalFilesConfigMap: When set to true, it takes the files located in `<helm-chart-dir>/files/additionalFiles` and mount them in `/etc/cassandra` inside the pods, some additional files for cassandra can be stored here, such as JMX auth files
* jvmOpts.enabled: If set to `true` the environment variable `JVM_OPTS` is overridden with the value provided on jvmOpts.value
* jvmOpts.value: Sets the value of the environment variable `JVM_OPTS`, in this way some java runtime configurations can be provided such as RAM heap usage
* resources.enabled: if set to `true`, the resource constraints are set on each pod using the values under resources.requests and resources.limits
* resources.requests.memory: Initial resource allocation for each pod in the cluster
* resources.request.cpu: Initial resource allocation for each pod in the cluster
* resources.limits.memory: Limits for memory allocation for each pod in the cluster
* resources.limits.cpu: Limits for cpu allocation for each pod in the cluster
Installation
============
Once all the configurations are set, to install the chart inside a given Kubernetes cluster you must run:
.. code-block:: bash
helm install ./cassandra-pmem-helm
Eventually all the given nodes will be shown as running using :command:`kubectl get pods`.
Running DBRS with Redis
***********************
The Redis stack application is enabled for a multinode Kubernetes environment using Intel Optane DCPMM PMem DIMMs in fsdax mode for storage.
The source code used for this application can be found in the `Github repository`_
The following examples will use the `Docker image with Redis`_. You can also build your own image with Docker by using the :file:`Dockerfile` and running with this command
.. code-block:: bash
docker build --force-rm --no-cache -f Dockerfile -t ${DOCKER_IMAGE} .
Single node
===========
Prior to starting the container, you will need to have the Intel Optane DCPMM module in fsdax with a file system and mounted in `/mnt/dax0` as shown above.
Use the following to start the container, replacing ${DOCKER_IMAGE} with the name of the image you are using.
.. code-block:: bash
docker run --mount type=bind,source=/mnt/dax0,target=/mnt/pmem0 -i -d --name pmem-redis ${DOCKER_IMAGE} --nvm-maxcapacity 200 --nvm-dir /mnt/pmem0 --nvm-threshold 64 --protected-mode no
Redis Operator in a Kubernetes cluster
======================================
After setting up :ref:`kubernetes` in |CL|, you will need to enable it to support DCPMM using the pmem-cls driver. To install the driver follow the instructions in the `pmem-csi`_ repository.
We are using source code from the `Redis operator`_ .
.. note::
If you already have a redis-operator, you will need to delete it before installing a new one.
After installing the operator you are ready to deploy redisfailover instances using a yaml file, like this `example for persistent memory`_. You can download it and change the source of the image to reflect your environment. We have named our yaml `redis-failover.yml`
To start a redisfailover instance in Kubernetes run the following
.. code-block:: bash
kubectl create -f redis-failover.yml
.. important::
There is a `known issue`_ in which the sentinels do not have enough memory to create the InitContainer. The current workaround is to build the image increasing the limits for the InitContainer memory to 32Mb
Running DBRS with Memcached
***************************
With DBRS V2.0 you can use the DBRS stack with `Memcached`_, a free and open source, high performance, distributed meory object caching system. This stack is ready to use DCPMM in fsdax for storage. The source for this application can be found in the `Memcached`_ repository.
.. note::
The DBRS v2.0 release does not support Redis or Cassandra.
Build the DBRS Memcached image
==============================
To build the Memcached enabled image, use the Dockerfile with this command:
.. code-block:: bash
docker build --force-rm --no-cache -f Dockerfile -t ${DOCKER_IMAGE} .
Run DBRS with Memcached as a standalone container
=================================================
Prior to launching the container, you will need to configure the DCPMM in fsdax mode with a file system, and have it mounted in :file:`/mnt/dax0`. Instructions for configuration can be found in :ref:`dbrs-hardware-configuration`.
To launch the container run this command:
.. code-block:: bash
docker run --mount type=bind,source=/mnt/dax0,target=/mnt/pmem0 -i -d --name pmem-memchached ${DOCKER_IMAGE} -e /mnt/pmem0/memcached.file -m 64 -c 1024 -p 11211
where:
:command:`-m` is the maximum memory limit to use in megabytes
:command:`-e` is the mmap path for external memory (DCPMM storage). For this container the DCPMM sould be mounted inside the container on :file:`/mnt/pmem0`
:command:`-c` is the number of concurrent connections
:command:`-p` is the TCP connection port.
For more information please refer to this `blog post`_ from `Memcached`_
*Intel, Xeon, Intel Optane, and the Intel logo are trademarks of Intel Corporation or its subsidiaries.*
.. _Intel Download Center: https://downloadcenter.intel.com/download/28695/Intel-Server-Board-S2600WF-Family-BIOS-and-Firmware-Update-Package-for-UEFI
.. _Quick Start Guide: https://software.intel.com/en-us/articles/quick-start-guide-configure-intel-optane-dc-persistent-memory-on-linux
.. _Managing NVDIMMs: https://docs.pmem.io/ndctl-user-guide/managing-nvdimms
.. _Configure, Manage, and Profile: https://software.intel.com/en-us/articles/configure-manage-and-profile-intel-optane-dc-persistent-memory-modules
.. _DBRS github repository: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dbrs
.. _build-cassandra-pmem.sh: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dbrs/cassandra/scripts/
.. _cassandra-pmem-helm: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dbrs/cassandra/cassandra-pmem-helm
.. _helm: https://helm.sh/
.. _Github repository: https://github.com/pmem/pmem-redis
.. _Redis operator: https://github.com/spotahome/redis-operator
.. _example for persistent memory: https://github.com/spotahome/redis-operator/blob/master/example/redisfailover/pmem.yaml
.. _known issue: https://github.com/spotahome/redis-operator/issues/176
.. _Docker image with Apache Cassandra: https://hub.docker.com/r/clearlinux/stacks-dbrs-cassandra
.. _Docker image with Redis: https://hub.docker.com/r/clearlinux/stacks-dbrs-redis
.. _Intel Optane PMM: https://www.intel.com/content/www/us/en/architecture-and-technology/optane-technology/optane-for-data-centers.html
.. _pmem-csi: https://github.com/intel/pmem-csi/blob/release-0.6/README.md
.. _DBRS Terms of Use: https://clearlinux.org/stacks/database/terms-of-use
.. _Database Reference Stack website: https://clearlinux.org/stacks/database-reference
.. _DBRS V1.0: https://clearlinux.org/news-blogs/database-reference-stack-dbrs-v10-now-available
.. _DBRS V2.0: https://clearlinux.org/blogs-news/database-reference-stack-dbrs-v2-now-available
.. _Memcached: https://memcached.org
.. _blog post: https://memcached.org/blog/persistent-memory/
source/guides/stacks/dlrs-inference.rst
-1047
View File
File diff suppressed because it is too large Load Diff
source/guides/stacks/dlrs.rst
-1173
View File
File diff suppressed because it is too large Load Diff
source/guides/stacks/greengrass.rst
-430
View File
@@ -1,430 +0,0 @@
.. _greengrass:
Enable AWS Greengrass\* and OpenVINO™ toolkit
#############################################
This guide explains how to enable AWS Greengrass\* and OpenVINO™ toolkit.
Specifically, the guide demonstrates how to:
* Set up the Intel edge device with |CL-ATTR|
* Install the OpenVINO™ toolkit and Amazon Web Services\* (AWS\*)
Greengrass\* software stacks
* Use AWS Greengrass\* and AWS Lambda\* to deploy the FaaS samples from
the cloud
.. contents::
:local:
:depth: 1
Overview
********
Hardware accelerated Function-as-a-Service (FaaS) enables cloud developers to
deploy inference functionalities [1] on Intel IoT edge devices with
accelerators (CPU, Integrated GPU, Intel® FPGA, and Intel® Movidius™
technology). These functions provide a great developer experience and
seamless migration of visual analytics from cloud to edge in a secure manner
using a containerized environment. Hardware-accelerated FaaS provides the
best-in-class performance by accessing optimized deep learning libraries on
Intel IoT edge devices with accelerators.
Supported platforms
*******************
* Operating System: |CL| latest release
* Hardware: Intel® Core™ processors (that support inference on CPU only)
Sample description
==================
The AWS Greengrass samples are located at `Edge-Analytics-FaaS`_. This
guide uses the 1.0 version of the source code.
|CL| provides the following AWS Greengrass samples:
* `greengrass_classification_sample.py`_
This AWS Greengrass sample classifies a video stream using classification
networks such as AlexNet and GoogLeNet and publishes top-10 results on AWS\*
IoT Cloud every second.
* `greengrass_object_detection_sample_ssd.py`_
This AWS Greengrass sample detects objects in a video stream and
classifies them using single-shot multi-box detection (SSD) networks such
as SSD Squeezenet, SSD Mobilenet, and SSD300. This sample publishes
detection outputs such as class label, class confidence, and bounding box
coordinates on AWS IoT Cloud every second.
Install the OS on the edge device
*********************************
Start with a clean installation of |CL| on a new system, using the
:ref:`bare-metal-install-desktop`, found in :ref:`get-started`.
Create user accounts
====================
After |CL| is installed, create two user accounts. Create an administrative
user in |CL| and create a user account for the Greengrass services to use (
see Greengrass user below).
#. Create a new user and set a password for that user. Enter the following
commands as ``root``:
.. code-block:: bash
useradd <userid>
passwd <userid>
#. Next, enable the :command:`sudo` command for your new <userid>. Add
<userid> to the `wheel` group:
.. code-block:: bash
usermod -G wheel -a <userid>
#. Create a :file:`/etc/fstab` file.
.. code-block:: bash
touch /etc/fstab
.. note::
By default, |CL| does not create an :file:`/etc/fstab` file.
You must create this file before the Greengrass service runs.
Add required bundles
====================
Use the :command:`swupd` software updater utility to add the prerequisite bundles
for the OpenVINO software stack:
.. code-block:: bash
swupd bundle-add os-clr-on-clr desktop-autostart computer-vision-basic
.. note::
Learn more about how to :ref:`swupd-guide`.
The :command:`computer-vision-basic` bundle installs the OpenVINO™ toolkit,
and the sample models optimized for Intel edge platforms.
.. _convert-dl-models:
Convert deep learning models
============================
Locate sample models
--------------------
There are two types of provided models that can be used in conjunction with
AWS Greengrass for this guide: classification or object detection.
To complete this guide using an image classification model,
download the BVLC AlexNet model files `bvlc_alexnet.caffemodel`_ and
`deploy.prototxt`_ to the default model_location at
:file:`/usr/share/openvino/models`. Any custom pre-trained classification models
can be used with the classification sample.
For object detection, the sample models optimized for Intel edge platforms
are included with the computer-vision-basic bundle installation at
:file:`/usr/share/openvino/models`. These models are provided as an example;
you may also use a custom SSD model with the Greengrass object detection sample.
Run model optimizer
-------------------
Follow the instructions in the `Model Optimizer Developer Guide`_ for converting
deep learning models to Intermediate Representation using Model Optimizer. To
optimize either of the sample models described above, run one of the following commands.
For classification using BVLC AlexNet model:
.. code-block:: bash
python3 mo.py --framework caffe --input_model
<model_location>/bvlc_alexnet.caffemodel --input_proto
<model_location>/deploy.prototxt --data_type <data_type> --output_dir
<output_dir> --input_shape [1,3,227,227]
For object detection using SqueezeNetSSD-5Class model:
.. code-block:: bash
python3 mo.py --framework caffe --input_model
<model_location>/'SqueezeNet 5-Class detection'/SqueezeNetSSD-5Class.caffemodel
--input_proto <model_location>/'SqueezeNet 5-Class detection'/SqueezeNetSSD-5Class.prototxt
--data_type <data_type> --output_dir <output_dir>
In these examples:
* `<model_location>` is :file:`/usr/share/openvino/models`.
* `<data_type>` is FP32 or FP16, depending on target device.
* `<output_dir>` is the directory where the Intermediate Representation
(IR) is stored. IR contains .xml format corresponding to the network
structure and .bin format corresponding to weights. This .xml file should be
passed to :command:`<PARAM_MODEL_XML>`.
* In the BVLC AlexNet model, the prototxt defines the input shape with
batch size 10 by default. In order to use any other batch size, the
entire input shape must be provided as an argument to the model
optimizer. For example, to use batch size 1, you must provide:
`--input_shape [1,3,227,227]`
Configure AWS Greengrass group
******************************
For each Intel edge platform, you must create a new AWS Greengrass group
and install AWS Greengrass core software to establish the connection between
cloud and edge.
#. To create an AWS Greengrass group, follow the instructions in
`Configure AWS IoT Greengrass on AWS IoT`_.
#. To install and configure AWS Greengrass core on edge platform, follow
the instructions in `Start AWS Greengrass on the Core Device`_. In
step 8(b), download the x86_64 Ubuntu\* configuration of the AWS Greengrass
core software.
.. note::
You do not need to run the :file:`cgroupfs-mount.sh` script in step #6
of Module 1 of the `AWS Greengrass Developer Guide`_ because this is
enabled already in |CL|.
#. Be sure to download both the security resources and the AWS Greengrass
core software.
.. note::
Security certificates are linked to your AWS account.
Create and package Lambda function
**********************************
#. Complete steps 1-4 of the AWS Greengrass guide at
`Create and Package a Lambda Function`_.
.. note::
This creates the tarball needed to create the AWS Greengrass
environment on the edge device.
#. In step 5, replace :file:`greengrassHelloWorld.py` with the classification or
object detection Greengrass sample from `Edge-Analytics-Faas`_:
* Classification: `greengrass_classification_sample.py`_
* Object Detection: `greengrass_object_detection_sample_ssd.py`_
#. Zip the selected Greengrass sample with the extracted Greengrass SDK folders
from the previous step into :file:`greengrass_sample_python_lambda.zip`.
The zip should contain:
* greengrasssdk
* greengrass classification or object detection sample
For example:
.. code-block:: bash
zip -r greengrass_lambda.zip greengrasssdk
greengrass_object_detection_sample_ssd.py
#. Return to the AWS documentation section called
`Create and Package a Lambda Function`_ and complete the procedure.
.. note::
In step 9(a) of the AWS documentation, while uploading the zip file,
make sure to name the handler to one of the following, depending on the
AWS Greengrass sample you are using:
* greengrass_object_detection_sample_ssd.function_handler
* greengrass_classification_sample.function_handler
Configure Lambda function
*************************
After creating the Greengrass group and the Lambda function, start
configuring the Lambda function for AWS Greengrass.
#. Follow steps 1-8 in `Configure the Lambda Function for AWS IoT Greengrass`_
in the AWS documentation.
#. In addition to the details mentioned in step 8, change the Memory limit
to 2048 MB to accommodate large input video streams.
#. Add the following environment variables as key-value pairs when editing
the Lambda configuration and click on update:
.. list-table:: **Table 1. Environment variables: Lambda configuration**
:widths: 20 80
:header-rows: 1
* - Key
- Value
* - PARAM_MODEL_XML
- <MODEL_DIR>/<IR.xml>, where <MODEL_DIR> is user specified and
contains IR.xml, the Intermediate Representation file from the
OpenVINO™ Model Optimizer.
For this guide, <MODEL_DIR> should be set to '/usr/share/openvino/models'
or one of its subdirectories.
* - PARAM_INPUT_SOURCE
- <DATA_DIR>/input.webm to be specified by user. Holds both input and
output data. For webcam, set PARAM_INPUT_SOURCE to ‘/dev/video0’
* - PARAM_DEVICE
- "CPU"
* - PARAM_CPU_EXTENSION_PATH
- /usr/lib64/libcpu_extension.so
* - PARAM_OUTPUT_DIRECTORY
- <DATA_DIR> to be specified by user. Holds both input and output
data
* - PARAM_NUM_TOP_RESULTS
- User specified for classification sample.
(e.g. 1 for top-1 result, 5 for top-5 results)
#. Add subscription to subscribe, or publish messages from AWS Greengrass
Lambda function by completing the procedure in `Configure the Lambda Function for AWS IoT Greengrass`_.
.. note::
The optional topic filter field is the topic mentioned inside the Lambda function. In this guide, sample topics include the following:
:command:`openvino/ssd` or :command:`openvino/classification`
Add local resources
===================
Refer to the AWS documentation `Access Local Resources with Lambda Functions and Connectors`_
for details about local resources and access privileges.
The following table describes the local resources needed for the CPU:
.. list-table:: **Local resources**
:widths: 20, 20, 20, 20
:header-rows: 1
* - Name
- Resource type
- Local path
- Access
* - ModelDir
- Volume
- <MODEL_DIR> to be specified by user
- Read-Only
* - Webcam
- Device
- /dev/video0
- Read-Only
* - DataDir
- Volume
- <DATA_DIR> to be specified by user. Holds both input and output
data.
- Read and Write
Deploy Lambda function
**********************
Refer to the AWS documentation `Deploy Cloud Configurations to an AWS IoT Greengrass Core Device`_ for instructions on how to deploy the lambda function to AWS
Greengrass core device. Select *Deployments* on the group page and follow the instructions.
Output consumption
==================
There are four options available for output consumption. These options are
used to report, stream, upload, or store inference output at an interval
defined by the variable :command:`reporting_interval` in the AWS Greengrass samples.
#. IoT cloud output:
This option is enabled by default in the AWS Greengrass samples using the
:command:`enable_iot_cloud_output` variable. You can use it to verify the lambda
running on the edge device. It enables publishing messages to IoT cloud
using the subscription topic specified in the lambda. (For example, topics
may include :command:`openvino/classification` for classification and :command:`openvino/ssd`
for object detection samples.) For classification, top-1 result with class
label are published to IoT cloud. For SSD object detection, detection
results such as bounding box coordinates of objects, class label, and
class confidence are published.
Refer to the AWS documentation
`Verify the Lambda Function Is Running on the Device`_ for instructions on
how to view the output on IoT cloud.
#. Kinesis streaming:
This option enables inference output to be streamed from the edge device
to cloud using Kinesis [3] streams when :command:`enable_kinesis_output` is set
to True. The edge devices act as data producers and continually push
processed data to the cloud. You must set up and specify
Kinesis stream name, Kinesis shard, and AWS region in the AWS Greengrass
samples.
#. Cloud storage using AWS S3 bucket:
When the :command:`enable_s3_jpeg_output` variable is set to True, it enables
uploading and storing processed frames (in jpeg format) in an AWS S3
bucket. You must set up and specify the S3 bucket name in the AWS
Greengrass samples to store the JPEG images. The images are named using the
timestamp and uploaded to S3.
#. Local storage:
When the :command:`enable_s3_jpeg_output` variable is set to True, it enables
storing processed frames (in jpeg format) on the edge device. The images
are named using the timestamp and stored in a directory specified by
:command:`PARAM_OUTPUT_DIRECTORY`.
References
**********
#. AWS Greengrass: https://aws.amazon.com/greengrass/
#. AWS Lambda: https://aws.amazon.com/lambda/
#. AWS Kinesis: https://aws.amazon.com/kinesis/
*Intel, OpenVINO, and the Intel logo are trademarks of Intel Corporation or its subsidiaries.*
.. _Edge-Analytics-FaaS: https://github.com/intel/Edge-Analytics-FaaS/tree/v1.0/AWS%20Greengrass
.. _bvlc_alexnet.caffemodel: http://dl.caffe.berkeleyvision.org/bvlc_alexnet.caffemodel
.. _deploy.prototxt: https://github.com/BVLC/caffe/blob/master/models/bvlc_alexnet/deploy.prototxt
.. _greengrass_classification_sample.py: https://github.com/intel/Edge-Analytics-FaaS/blob/v1.0/AWS%20Greengrass/greengrass_classification_sample.py
.. _greengrass_object_detection_sample_ssd.py: https://github.com/intel/Edge-Analytics-FaaS/blob/v1.0/AWS%20Greengrass/greengrass_object_detection_sample_ssd.py
.. _Model Optimizer Developer Guide: https://software.intel.com/en-us/articles/OpenVINO-ModelOptimizer
.. _AWS Greengrass Developer Guide: https://docs.aws.amazon.com/greengrass/latest/developerguide/what-is-gg.html
.. _Configure AWS IoT Greengrass on AWS IoT: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-config.html
.. _Start AWS Greengrass on the Core Device: https://docs.aws.amazon.com/greengrass/latest/developerguide/gg-device-start.html
.. _Configure the Lambda Function for AWS IoT Greengrass: https://docs.aws.amazon.com/greengrass/latest/developerguide/config-lambda.html
.. _Access Local Resources with Lambda Functions and Connectors: https://docs.aws.amazon.com/greengrass/latest/developerguide/access-local-resources.html
.. _Deploy Cloud Configurations to an AWS IoT Greengrass Core Device: https://docs.aws.amazon.com/greengrass/latest/developerguide/configs-core.html
.. _Verify the Lambda Function Is Running on the Device: https://docs.aws.amazon.com/greengrass/latest/developerguide/lambda-check.html
.. _Create and Package a Lambda Function: https://docs.aws.amazon.com/greengrass/latest/developerguide/create-lambda.html
source/guides/stacks/hpcrs.rst
-242
View File
@@ -1,242 +0,0 @@
.. _hpcrs:
High Performance Computing Reference Stack
##########################################
This guide gives examples for converting Docker* containers, such as those provided by the :ref:`dlrs` into Singularity* containers suited for HPC, and then walking through a multi-node benchmarking example with TensorFlow*.
.. contents::
:local:
:depth: 1
Overview
********
The High Performance Computing Reference Stack (HPCRS) meets the needs of deploying HPC and AI workloads on the same system. This software solution reduces the complexities associated with integrating software components for High Performance Computing (HPC) Platforms. `Singularity`_ is an open source container platform to package entire scientific workflows, software and libraries, and even data.
Installing Singularity
**********************
The installation instructions are for Linux* systems, and have been enabled for installation on |CL-ATTR|.
.. note::
The steps for installation can also be found on the `Singularity quick-start`_ https://sylabs.io/guides/3.0/user-guide/quick_start.html#quick-installation
#. Install Go*.
This guide requires version 1.13 of Go, for compatibility with Singularity v3.0.0. Please use these steps to ensure the correct version of Go is installed:
.. code-block:: bash
$ export VERSION=1.13 OS=linux ARCH=amd64 && \
wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
rm go$VERSION.$OS-$ARCH.tar.gz
#. Setup the environment for Go.
.. code-block:: bash
echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
source ~/.bashrc
#. Install :command:`dep` for dependency resolution with Singularity v3.0.0.
.. code-block:: bash
go get -u github.com/golang/dep/cmd/dep
#. Download Singularity.
.. code-block:: bash
go get -d github.com/sylabs/singularity
.. note::
Go will complain that there are no Go files, but it will still download the Singularity source code to the appropriate directory within the $GOPATH.
#. Checkout version 3.0.0 of Singularity.
.. code-block:: bash
export VERSION=v3.0.3 # or another tag or branch if you like && \
cd $GOPATH/src/github.com/sylabs/singularity && \
git fetch && \
git checkout $VERSION # omit this command to install the latest bleeding edge code from master
#. Build Singularity.
Singularity uses a custom build system called makeit. mconfig is called to generate a Makefile and then make is used to compile and install.
The devpkg-openssl, devpkg-util-linux package may be required and can be installed using the :command:`sudo swupd bundle-add <pkg-name>`.
.. code-block:: bash
./mconfig && \
make -C ./builddir && \
sudo make -C ./builddir install
#. Configure bash completion (optional).
To enjoy bash completion with Singularity commands and options, source the bash completion file. Add this command to your ~/.bashrc file so that bash completion continues to work in new shells
.. code-block:: bash
. /usr/local/etc/bash_completion.d/singularity
Converting Docker images to Singularity Images
**********************************************
#. Download d2s.
:command:`d2s` os an open source tool to convert Docker images to
Singularity images. You can use the script in the location where it is
downloaded, or install it using the included :file:`setup.py` file with
the :command:`python setup.py install`
.. code-block:: bash
git clone https://github.com/intel/stacks.git
cd stacks/hpcrs/d2s
#. List local Docker images.
.. code-block:: bash
python d2s.py --list_docker_images
Your output can appear like this:
.. code-block:: console
==============================
Docker images present locally
==============================
ID NAME
0: clearlinux/stacks-dlrs-mkl
1: clearlinux/stacks-dlrs_2-mkl
==============================
#. Convert to Singularity images.
To convert the Docker images to Singularity images, use the :command:`d2s`
script with the ID numbers of the images you wish to convert.
We strongly recommend using one of the :file:`clearlinux/stacks-dlrs-mkl`
or :file:`sysstacks/stacks-dlrs-mkl` based images for this guide. Other
images may be incompatible with expected configuration or filesystem
options.
.. code-block:: bash
python d2s.py --convert_docker_images <ID_1> <ID_2>
#. Use the Singularity image.
To use the container shell to run workloads, launch the image and you
will be dropped into the shell. The Singularity image name will be the
same as the name of the Docker image, with slashes converted to
underscores.
.. code-block:: bash
singularity shell <singularity image>
Using the example output above, after conversion you could launch the clearlinux/stacks-dlrs-mkl Singularity image with
:command:`singularity shell clearlinux_stacks-dlrs-mkl`
Execute a multi-node benchmark on an HPC cluster
************************************************
The following example was executed on an Intel(r) Xeon(r) Processor-based
HPC infrastructure. The following steps may need to be adjusted for
different environments. See this `Intel Whitepaper`_ for more information.
Running a ResNet50 workload multi-node
--------------------------------------
#. Download the TensorFlow benchmark.
.. code-block:: bash
git clone http://github.com/tensorflow/benchmarks -b cnn_tf_v1.13_compatible
#. Copy the Singularity image and the benchmark files to the HPC cluster
environment.
#. Install OpenMPI* if needed.
.. note::
If the HPC host does not have OpenMPI installed, install a custom
local version in the user's home directory. This version must be the
same as the version installed in the DLRS container. Follow the steps
for `building OpenMPI`_ from their documentation.
#. Adjust PATH variables.
Include the OpenMPI install locations in the PATH and LD_LIBRARY_PATH
environment variables.
.. code-block:: bash
export PATH="$PATH:<openmpi install path>/bin"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:<openmpi install path>/lib/"
#. Execute the TF benchmark script on single or multiple nodes using OpenMPI
through the :command:`mpirun` command. Replace variables in {} braces to
reflect your environment.
.. code-block:: bash
mpirun --np ${NUM_COPIES} \
-bind-to none \
-map-by slot \
--display-map \
-host ${HOSTNAMES} \
--report-bindings \
--oversubscribe \
-x LD_LIBRARY_PATH \
-x PATH \
-x HOROVOD_FUSION_THRESHOLD \
-x OMP_NUM_THREADS=${OMP_NUM_THREADS} \
singularity exec ${PATH_TO_SING_IMAGE} \
python ${PATH_TO_TF_BENCH}/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
--batch_size=128 \
--model=resnet50 \
--num_intra_threads=${NUM_INTRA_THREADS} \
--num_inter_threads=${NUM_INTER_THREADS} \
--data_format=NHWC \
--device=cpu \
--variable_update=horovod \
--horovod_device=cpu
.. note::
Refer to the `DLRS script`_ for recommended values for setting environment variables in the :command:`mpirun` command.
.. note::
You may see an error regarding a missing library while executing the DLRS container.
“tensorflow.python.framework.errors_impl.NotFoundError: libnuma.so.1: cannot open shared object file: No such file or directory”
A workaround for this error is to bind the path to the library from the host.
.. code-block:: bash
--bind /usr/lib64/libnuma.so.1:/usr/lib64/libnuma.so.1
.. _Singularity: https://sylabs.io/
.. _Singularity quick-start: https://sylabs.io/guides/3.0/user-guide/quick_start.html#quick-installation
.. _Intel Whitepaper: https://www.intel.com/content/www/us/en/artificial-intelligence/solutions/best-known-methods-for-scaling-deep-learning-with-tensorflow-on-xeon-processor-based-clusters.html
.. _building OpenMPI: https://www.open-mpi.org/faq/?category=building#easy-build
.. _DLRS script: https://github.com/intel/stacks/blob/master/dlrs/clearlinux/tensorflow_2/mkl/scripts/set_env.sh
source/guides/stacks/mers.rst
-564
View File
@@ -1,564 +0,0 @@
.. _mers:
Media Reference Stack
#####################
The Media Reference Stack (MeRS) is a highly optimized software stack for
Intel® Architecture Processors (the CPU) and Intel® Processor Graphics (the
GPU) to enable media prioritized workloads, such as transcoding and analytics.
This guide explains how to use the pre-built |MERS| container image, build
your own |MERS| container image, and use the reference stack.
.. contents::
:local:
:depth: 1
Overview
********
Developers face challenges due to the complexity of software integration for
media tasks that require investing time and engineering effort.
For example:
* Finding the balance between quality and performance.
* Understanding available standard-compliant encoders.
* Optimizing across the hardware-software stack for efficiency.
|MERS| abstracts away the complexity of integrating multiple software
components and specifically tunes them for Intel platforms. |MERS| enables
media and visual cloud developers to deliver experiences using a simple
containerized solution.
Releases
********
Refer to the `System Stacks for Linux* OS repository
<https://github.com/intel/stacks>`_ for information and download links for the
different versions and offerings of the stack.
* MeRS V0.2.0 release announcement including media processing on GPU and
analytics on CPU.
* MeRS V0.1.0 including media processing and analytics CPU.
* `MeRS Release notes on Github*
<https://github.com/intel/stacks/blob/master/mers/NEWS.md>`_ for the
latest release of Deep Learning Reference Stack
Prerequisites
=============
|MERS| can run on any host system that supports Docker\*. This guide uses
|CL-ATTR| as the host system.
- To install |CL| on a host system, see how to
:ref:`install Clear Linux* OS from the live desktop
<bare-metal-install-desktop>`.
- To install Docker* on a |CL| host system, see
the :ref:`instructions for installing Docker* <docker>`.
.. important::
For optimal media analytics performance, a processor with Vector Neural
Network Instructions (VNNI) should be used. VNNI is an extension of Intel®
Advanced Vector Extensions 512 (Intel® AVX-512) and is available starting
with the 2nd generation of Intel® Xeon® Scalable processors, providing AI
inference acceleration.
Stack features
==============
The |MERS| provides a `pre-built Docker image available on DockerHub
<https://hub.docker.com/r/sysstacks/mers-clearlinux>`_, which includes
instructions on building the image from source. |MERS| is open-sourced to
make sure developers have easy access to the source code and are able to
customize it. |MERS| is built using the latest *clearlinux/os-core* Docker
image and aims to support the latest |CL| version.
|MERS| provides the following libraries and drivers:
.. list-table::
:widths: 15 85
* - SVT-HEVC
- Scalable Video Technology for HEVC encoding, also known as H.265
* - SVT-AV1
- Scalable Video Technology for AV1 encoding
* - x264
- x264 for H.264/MPEG-4 AVC encoding
* - dav1d
- `dav1d <https://code.videolan.org/videolan/dav1d>`_ for AV1 decoding
* - libVA
- `VAAPI (Video Acceleration API) open-source library (LibVA),
<https://github.com/intel/libva>`_ which provides access to graphics
hardware acceleration capabilities.
* - media-driver
- `Intel® Media Driver for VAAPI <https://github.com/intel/media-driver/>`_
for supporting hardware acceleration on Intel® Gen graphics hardware
platforms.
* - gmmlib
- `Intel® Graphics Memory Management Library
<https://github.com/intel/gmmlib>`_ provides device specific and buffer
management for the Intel® Graphics Compute Runtime for oneAPI Level Zero
and OpenCL™ Driver and the Intel Media Driver for VAAPI.
Components of the |MERS| include:
* |CL| as a base for performance and security.
* `OpenVINO™ toolkit
<https://01.org/openvinotoolkit>`_ for inference.
* `FFmpeg* <https://www.ffmpeg.org>`_ with plugins for:
- `Scalable Video Technology (SVT)
<https://01.org/svt>`_
* `GStreamer* <https://gstreamer.freedesktop.org/>`_ with plugins for:
- `Scalable Video
Technology (SVT) <https://01.org/svt>`_
- `OpenVINO™ toolkit
<https://01.org/openvinotoolkit>`_
- `VAAPI <https://github.com/GStreamer/gstreamer-vaapi>`_
* `Intel® Media SDK <https://github.com/Intel-Media-SDK/MediaSDK>`_
.. note::
The |MERS| is validated on 11th generation Intel Processor Graphics and
newer. Older generations should work but are not tested against.
.. note::
The pre-built |MERS| container image configures FFmpeg without certain
elements (specific encoder, decoder, muxer, etc.) that you may require. If
you require changes to FFmpeg we suggest starting at
:ref:`building-the-mers-container-image`.
.. note::
The Media Reference Stack is a collective work, and each piece of software
within the work has its own license. Please see the `MeRS Terms of Use
<https://clearlinux.org/stacks/media/terms-of-use>`_ for more details about
licensing and usage of the Media Reference Stack.
Get the pre-built |MERS| container image
****************************************
Pre-built |MERS| Docker images are available on DockerHub* at
https://hub.docker.com/r/sysstacks/mers-clearlinux
To use the |MERS|:
#. Pull the image directly from `Docker Hub
<https://hub.docker.com/r/sysstacks/mers-clearlinux>`_.
.. code-block:: bash
docker pull sysstacks/mers-clearlinux
.. note ::
The |MERS| docker image is large in size and will take some time to
download depending on your Internet connection.
If you are on a network with outbound proxies, be sure to configure
Docker to allow access. See the `Docker service proxy
<https://docs.docker.com/config/daemon/systemd/#httphttps-proxy>`_ and
`Docker client proxy
<https://docs.docker.com/network/proxy/#configure-the-docker-client>`_
documentation for more details.
#. Once you have downloaded the image, run it using the following command:
.. code-block:: bash
docker run -it sysstacks/mers-clearlinux
This will launch the image and drop you into a bash shell inside the
container. GStreamer and FFmpeg programs are installed in the container
image and accessible in the default $PATH. Use these programs as you would
outside of |MERS|.
Paths to media files and video devices, such as cameras, can be shared from
the host to the container with the :command:`--volume` switch `using Docker
volumes <https://docs.docker.com/storage/volumes/>`_.
.. _building-the-mers-container-image:
Build the |MERS| container image from source
********************************************
If you choose to build your own MeRS container image, you can optionally add
customizations as needed. The :file:`Dockerfile` for the MeRS is available on
`GitHub <https://github.com/intel/stacks/tree/master/mers>`_ and can be used
as a reference when creating your own container image.
#. The |MERS| image is part of the dockerfiles repository inside the |CL|
organization on GitHub. Clone the :file:`stacks` repository.
.. code-block:: bash
git clone https://github.com/intel/stacks.git
#. Navigate to the :file:`stacks/mers/clearlinux` directory which contains
the Dockerfile for the |MERS|.
.. code-block:: bash
cd ./stacks/mers/clearlinux
#. Use the :command:`docker build` command with the :file:`Dockerfile` to
build the MeRS container image.
.. code-block:: bash
docker build --no-cache -t sysstacks/mers-clearlinux .
Use the |MERS| container image
******************************
This section shows examples of how the |MERS| container image can be used to
process media files.
The models and video source can be substituted from your use-case. Some
publicly licensed sample videos are available at `sample-videos repository
<https://github.com/intel-iot-devkit/sample-videos>`_ for testing.
Media Transcoding
=================
The examples below show transcoding using the GPU or CPU for processing.
#. On the host system, setup a workspace for data and models:
.. code:: bash
mkdir ~/ffmpeg
mkdir ~/ffmpeg/input
mkdir ~/ffmpeg/output
#. Copy a video file to :file:`~/ffmpeg/input`.
.. code:: bash
cp </path/to/video> ~/ffmpeg/input
#. Run the *sysstacks/mers-clearlinux* Docker image, allowing shared access to
the workspace on the host:
.. code:: bash
docker run -it \
--volume ~/ffmpeg:/home/mers-user:ro \
--device=/dev/dri \
--env QSV_DEVICE=/dev/dri/renderD128 \
sysstacks/mers-clearlinux:latest
.. note::
The :command:`--device` parameter and the **GSV_DEVICE** environment
variable allow shared access to the GPU on the host system. The values
needed may be different depending on host's graphics configuration.
After running the :command:`docker run` command, you enter a bash shell
inside the container.
#. From the container shell, you can run FFmpeg and
GStreamer commands against the videos in :file:`/home/mers-user/input` as
you would normally outside of |MERS|.
Some sample commands are provided for reference.
For more information on using the *FFmpeg* commands, refer to the `FFmpeg
documentation <https://ffmpeg.org/documentation.html>`_.
For more information on using the *GStreamer* commands, refer to the
`GStreamer documentation
<https://gstreamer.freedesktop.org/documentation>`_.
Example: Transcoding using GPU
-------------------------------
The examples below show transcoding using the GPU for processing.
Using a FFmpeg to transcode raw content to SVT-HEVC and mp4:
.. code:: bash
ffmpeg -y -vaapi_device /dev/dri/renderD128 -f rawvideo -video_size 320x240 -r 30 -i </home/mers-user/input/test.yuv> -vf 'format=nv12, hwupload' -c:v h264_vaapi -y </home/mers-user/output/test.mp4>
Using a GStreamer to transcode H264 to H265:
.. code:: bash
gst-launch-1.0 filesrc location=</home/mers-user/input/test.264> ! h264parse ! vaapih264dec ! vaapih265enc rate-control=cbr bitrate=5000 ! video/x-h265,profile=main ! h265parse ! filesink location=</home/mers-user/output/test.265>
|MERS| builds FFmpeg with `HWAccel
<https://trac.ffmpeg.org/wiki/HWAccelIntro>`_ enabled which supports VAAPI.
Refer to the `FFmpeg wiki on VAAPI
<https://trac.ffmpeg.org/wiki/Hardware/VAAPI>`_ and `GStreamer with Media-SDK
wiki
<https://github.com/Intel-Media-SDK/MediaSDK/wiki/Build-and-use-GStreamer-with-MediaSDK#usage-examples>`_
for more usage examples and compatibility information.
Example: Transcoding using CPU
------------------------------
The example below shows transcoding of raw yuv420 content to SVT-HEVC and mp4,
using the CPU for processing.
.. code:: bash
ffmpeg -f rawvideo -vcodec rawvideo -s 320x240 -r 30 -pix_fmt yuv420p -i </home/mers-user/input/test.yuv> -c:v libsvt_hevc -y </home/mers-user/output/test.mp4>
Additional generic examples of FFmpeg commands can be found in the
`OpenVisualCloud repository
<https://github.com/OpenVisualCloud/Dockerfiles/blob/master/doc/ffmpeg.md>`_
and used for reference with |MERS|.
Media Analytics
===============
This example shows how to perform analytics and inferences with GStreamer
using the CPU for processing.
The steps here are referenced from the `gst-video-analytics Getting Started
Guide <https://github.com/opencv/gst-video-analytics/wiki>`_ except simply
substituting the *gst-video-analytics* docker image for the
*sysstacks/mers-clearlinux* image.
The example below shows how to use the |MERS| container image to perform video
with object detection and attributes recognition of a video using GStreamer
using pre-trained models and sample video files.
#. On the host system, setup a workspace for data and models:
.. code:: bash
mkdir ~/gva
mkdir ~/gva/data
mkdir ~/gva/data/models
mkdir ~/gva/data/models/intel
mkdir ~/gva/data/models/common
mkdir ~/gva/data/video
#. Clone the opencv/gst-video-analytics repository into the workspace:
.. code:: bash
git clone https://github.com/opencv/gst-video-analytics ~/gva/gst-video-analytics
cd ~/gva/gst-video-analytics
git submodule init
git submodule update
#. Clone the Open Model Zoo repository into the workspace:
.. code:: bash
git clone https://github.com/opencv/open_model_zoo.git ~/gva/open_model_zoo
#. Use the Model Downloader tool of Open Model Zoo to download ready to use
pre-trained models in IR format.
.. note::
If you are on a network with outbound proxies, you will need to
configure set environment variables with the proxy server.
Refer to the documentation on :ref:`proxy` for detailed steps.
On |CL| systems you will need the *python-extras* bundle.
Use :command:`sudo swupd bundle-add python-extras` for the downloader script to work.
.. code:: bash
cd ~/gva/open_model_zoo/tools/downloader
python3 downloader.py --list ~/gva/gst-video-analytics/samples/model_downloader_configs/intel_models_for_samples.LST -o ~/gva/data/models/intel
#. Copy a video file in h264 or mp4 format to :file:`~/gva/data/video`. Any
video with cars, pedestrians, human bodies, and/or human faces can be used.
.. code:: bash
git clone https://github.com/intel-iot-devkit/sample-videos.git ~/gva/data/video
This example simply clones all the video files from the `sample-videos
repsoitory <https://github.com/intel-iot-devkit/sample-videos>`_.
#. From a desktop terminal, allow local access to the X host display.
.. code:: bash
xhost local:root
export DATA_PATH=~/gva/data
export GVA_PATH=~/gva/gst-video-analytics
export MODELS_PATH=~/gva/data/models
export INTEL_MODELS_PATH=~/gva/data/models/intel
export VIDEO_EXAMPLES_PATH=~/gva/data/video
#. Run the *sysstacks/mers-clearlinux* docker image, allowing shared access
to the X server and workspace on the host:
.. code:: bash
docker run -it --runtime=runc --net=host \
-v ~/.Xauthority:/root/.Xauthority \
-v /tmp/.X11-unix:/tmp/.X11-unix \
-e DISPLAY=$DISPLAY \
-e HTTP_PROXY=$HTTP_PROXY \
-e HTTPS_PROXY=$HTTPS_PROXY \
-e http_proxy=$http_proxy \
-e https_proxy=$https_proxy \
-v $GVA_PATH:/home/mers-user/gst-video-analytics \
-v $INTEL_MODELS_PATH:/home/mers-user/intel_models \
-v $MODELS_PATH:/home/mers-user/models \
-v $VIDEO_EXAMPLES_PATH:/home/mers-user/video-examples \
-e MODELS_PATH=/home/mers-user/intel_models:/home/mers-user/models \
-e VIDEO_EXAMPLES_DIR=/home/mers-user/video-examples \
sysstacks/mers-clearlinux:latest
.. note::
In the :command:`docker run` command above:
- :command:`--runtime=runc` specifies the container runtime to be
*runc* for this container. It is needed for correct interaction with X
server.
- :command:`--net=host` provides host network access to the container.
It is needed for correct interaction with X server.
- Files :file:`~/.Xauthority` and :file:`/tmp/.X11-unix` mapped to the
container are needed to ensure smooth authentication with X server.
- :command:`-v` instances are needed to map host system directories
inside the Docker container.
- :command:`-e` instances set the Docker container environment
variables. Some examples need these variables set correctly in order
to operate correctly. Proxy variables are needed if host is behind a
firewall.
After running the :command:`docker run` command, it will drop you into a
bash shell inside the container.
#. From the container shell, run a sample analytics program in
:file:`~/gva/gst-video-analytics/samples` against your video source.
Below are sample analytics that can be run against the sample videos.
Choose one to run:
- Samples with *face detection and classification*:
.. code:: bash
./gst-video-analytics/samples/shell/face_detection_and_classification.sh $VIDEO_EXAMPLES_DIR/face-demographics-walking-and-pause.mp4
./gst-video-analytics/samples/shell/face_detection_and_classification.sh $VIDEO_EXAMPLES_DIR/face-demographics-walking.mp4
./gst-video-analytics/samples/shell/face_detection_and_classification.sh $VIDEO_EXAMPLES_DIR/head-pose-face-detection-female-and-male.mp4
./gst-video-analytics/samples/shell/face_detection_and_classification.sh $VIDEO_EXAMPLES_DIR/head-pose-face-detection-male.mp4
./gst-video-analytics/samples/shell/face_detection_and_classification.sh $VIDEO_EXAMPLES_DIR/head-pose-face-detection-female.mp4
When running, a video with object detection and attributes recognition
(bounding boxes around faces with recognized attributes) should be
played.
.. figure:: /_figures/stacks/mers-fig-1.png
:scale: 60%
:align: center
:alt: Face detection with the Clear Linux* OS Media Reference Stack
Figure 1: Screenshot of |MERS| running face detection with GSTreamer
and OpenVINO.
- Sample with *vehicle detection*:
.. code:: bash
./gst-video-analytics/samples/shell/vehicle_detection_2sources_cpu.sh $VIDEO_EXAMPLES_DIR/car-detection.mp4
When running, a video with object detection and attributes recognition
(bounding boxes around vehicles with recognized attributes) should be
played.
.. figure:: /_figures/stacks/mers-fig-2.png
:scale: 60%
:align: center
:alt: Vehicle detection with the Clear Linux* OS Media Reference Stack
Figure 2: Screenshot of |MERS| running vehicle detection with
GSTreamer and OpenVINO.
- Sample with *FPS measurement*:
.. code:: bash
./gst-video-analytics/samples/shell/console_measure_fps_cpu.sh $VIDEO_EXAMPLES_DIR/bolt-detection.mp4
Add AOM support
***************
The current version of |MERS| does not include the `Alliance for Open Media
<https://aomedia.org/>`_ Video Codec (AOM). AOM can be built from source on an
individual basis.
To add AOM support to the |MERS| image:
#. The following programs are needed to add AOM support to |MERS|: **docker,
git, patch**. On |CL| these can be installed with the commands below. For
other operating systems, install the appropriate packages.
.. code:: bash
sudo swupd bundle-add containers-basic dev-utils
#. Clone the Intel Stacks repository from GitHub.
.. code:: bash
git clone https://github.com/intel/stacks.git
#. Navigate to the directory for the |MERS| image.
.. code:: bash
cd stacks/mers/clearlinux/
#. Apply the patch to the :file:`Dockerfile`.
.. code:: bash
patch -p1 < aom-patches/stacks-mers-v2-include-aom.diff
#. Use the :command:`docker build` command to build a local copy of the
MeRS container image tagged as *aom*.
.. code-block:: bash
docker build --no-cache -t sysstacks/mers-clearlinux:aom .
Once the build has completed successfully, the local image can be used
following the same steps in this tutorial by substituting the image name with
*sysstacks/mers-clearlinux:aom*.
*Intel, Xeon, OpenVINO, and the Intel logo are trademarks of Intel
Corporation or its subsidiaries. OpenCL and the OpenCL logo are trademarks of
Apple Inc. used by permission by Khronos.*