clearlinux/clear-linux-documentation
2
0
Fork 0
mirror of https://github.com/clearlinux/clear-linux-documentation.git synced 2026-08-19 13:46:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add useage examples to DARS guide (#823)

Browse Source 
* Add useage examples to DARS guide

* Add detail on docker commands, reference custom-app-container doc

* Back out reference to custom-app-containers.
This commit is contained in:
Beth Dean
2019-09-26 13:41:32 -07:00
committed by michael vincerra
parent 1f82cf1dab
commit 4707f1d3e4
1 changed files with 114 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
source/guides/stacks/dars.rst
+114 -46
View File
@@ -6,25 +6,26 @@ Data Analytics Reference Stack
This guide explains how to use the :abbr:`DARS (Data Analytics Reference Stack)`,
and to optionally build your own DARS container image.
Any system that supports Docker\* containers can be used with DARS. This steps
Any system that supports Docker\* containers can be used with DARS. The steps
in this guide use |CL-ATTR| as the host system.
.. contents::
:local:
:depth: 1
The Data Analytics Reference Stack release
******************************************
Overview
********
The Data Analytics Reference Stack (DARS) provides developers and enterprises a straightforward, highly optimized software stack for storing and processing large
amounts of data. More detail is available on the
`DARS architecture and performance benchmarks`_.
The Data Analytics Reference Stack (DARS) provides developers and enterprises a straightforward, highly optimized software stack for storing and processing large amounts of data. More detail is available on the `DARS architecture and performance benchmarks`_.
Stack Features
==============
The Data Analytics Reference Stack provides two pre-built Docker images,
available on `Docker Hub`_:
* A |CL|-derived `DARS with OpenBlas`_ stack optimized for `OpenBLAS`_
* A |CL|-derived `DARS with Intel® MKL`_ stack optimized for `MKL`_
* A |CL|-derived `DARS with Intel® MKL`_ stack optimized for `MKL`_ (Intel® Math Kernel Library)
We recommend you view the latest component versions for each image in the
:file:`README` found in the `Data Analytics Reference Stack`_ GitHub\*
@@ -41,78 +42,137 @@ in the |CL|-based containers may not be the latest released by |CL|.
Using the Docker images
***********************
#. To immediately start using the latest stable DARS images, pull an image
Launching the Image
===================
#. To use the latest stable DARS images, pull an image
directly from `Docker Hub`_. This example uses the
`DARS with Intel® MKL`_ Docker image.
.. code-block:: bash
docker pull clearlinux/stacks-dars-mkl
#. Once you have downloaded the image, you can run it with
.. code-block:: bash
docker run -it --ulimit nofile=1000000:1000000 --name mkl <name of image>
This will launch the image and drop you into a bash shell inside the
container. You will see output similar to the following:
This will launch the image and drop you into a bash shell inside the container. The :command:`--ulimit nofile=` parameter is required in order to increase the allowed number of open files for the Apache Spark engine.
If you need to verify the name of the DARS image, you can use the :command:`docker image ls` command to see which images reside on your system.
.. code-block:: bash
docker image ls
.. code-block:: console
root@fd5155b89857 /root # spark-shell
spark-shell
Config directory: /usr/share/defaults/spark/
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
REPOSITORY TAG IMAGE ID CREATED SIZE
clearlinux/stacks-dars-mkl test-img 49a70a22231f 23 hours ago 2.66GB
ubuntu latest 2ca708c1c9cc 7 days ago 64.2MB
katadocker/kata-deploy latest bd6dc92f8060 7 days ago 673MB
clearlinux/stacks-dars-mkl latest 2c9555536d5f 4 weeks ago 2.62GB
Using Scala version 2.12.7 (OpenJDK 64-Bit Server VM, Java 1.8.0-internal)
Type in expressions to have them evaluated.
Type :help for more information.
scala>
The :command:`--ulimit nofile` parameter is currently required in order to
increase the number of open files opened at certain point by the spark
engine.
Building DARS images
********************
====================
If you choose to build your own DARS container images, you can customize
them as needed. Use the provided Dockerfile as a baseline.
If you choose to build your own DARS container images, you can customize them as needed. Use the :file:`Dockerfile` included in the Github\* repository as your baseline.
To construct images with |CL|, start with a |CL| development platform that
has the :command:`containers-basic-dev` bundle installed. Learn more about
bundles and installing them by using :ref:`swupd-guide`.
To construct images with |CL|, start with a |CL| development platform that has the :command:`containers-basic-dev` bundle installed. Learn more about bundles and installing them by using :ref:`swupd-guide`.
#. Clone the `Data Analytics Reference Stack`_ GitHub\* repository.
#. The `Data Analytics Reference Stack`_ is part of the |CL| Project GitHub\* repository. Clone the :file:`dockerfiles` repository.
.. code-block:: bash
git clone https://github.com/clearlinux/dockerfiles/tree/master/stacks/dars -b master
git clone https://github.com/clearlinux/dockerfiles.git
#. Inside the DARS directory, run :command:`make` to build OpenBLAS and MKL images.
#. Inside the :file:`stacks/dars/mkl` directory, use docker with the :file:`Dockerfile` to build the MKL image.
.. code-block:: bash
make
cd ./dockerfiles/stacks/dlrs/mkl
docker build --no-cache -t clearlinux/stacks-dars-mkl .
Run :command:`make baseline` to build the baseline CentOS image. Depending on
the system, it may take a while to finish building.
.. code-block:: bash
make baseline
#. Once completed, check the resulting images with :command:`Docker`
.. code-block:: bash
docker images | grep dars
docker images | grep dars
#. You can use any of the resulting images to launch fully functional containers. If you need to customize the containers, you can edit the provided :file:`Dockerfile`.
Using Apache Spark\* in DARS
****************************
After launching the container, you can start Apache Spark with either the Scala or PySpark environment. For these examples we will use PySpark, which is the Python\* API for Apache Spark.
.. code-block:: bash
pyspark
Launching is as simple as this. Depending on your system configuration and capabilities, you may need to define proxy or memory allocation settings on the command line or in a config file for optimal performance. Refer to the `Apache Spark documentation`_ for more detail.
After executing :command:`pyspark`, you will see output similar to this.
.. code-block:: console
root@fd5155b89857 /root # pyspark
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
Using Python version 3.7.4 (default, Jul 13 2019 06:59:17)
SparkSession available as 'spark'.
>>>
Execute code directly in PySpark
================================
A simple example for verifying that pyspark is working correctly is to run a small python function from a `PySpark getting started guide`_ to estimate the value of Pi. Run these lines in the PySpark shell.
.. code-block:: console
import random
NUM_SAMPLES = 100000000
def inside(p):
x, y = random.random(), random.random()
return x*x + y*y < 1
count = sc.parallelize(range(0, NUM_SAMPLES)).filter(inside).count()
pi = 4 * count / NUM_SAMPLES
print(“Pi is roughly”, pi)
Run Python programs with spark-submit
=====================================
You can also run python scripts in Apache Spark from the command line. We'll use the Apache Spark example found in the :file:`/usr/share/apache-spark/examples/src/main/python/pi.py` file. Note that we have turned off the INFO and WARN messages in Apache Spark for this example.
.. code-block:: console
#spark-submit /usr/share/apache-spark/examples/src/main/python/pi.py
Config directory: /usr/share/defaults/spark/
Pi is roughly 3.134700
DARS Usecase example
====================
The DARS container is used in conjunction with the Deep Learning Reference Stack container to implement a real world use case. Refer to the `Github Issue Classification`_ Usecase found in the `stacks-usecase`_ repository for a walkthrough. This usecase is implemented using the Scala environment, rather than PySpark.
#. You can use any of the resulting images to launch fully functional containers.
If you need to customize the containers, you can edit the provided :file:`Dockerfile`.
.. _Data Analytics Reference Stack: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dars
@@ -131,3 +191,11 @@ bundles and installing them by using :ref:`swupd-guide`.
.. _DARS architecture and performance benchmarks: https://clearlinux.org/stacks/data-analytics-stack-v1
.. _DARS Terms of Use: https://clearlinux.org/stacks/data-analytics/terms-of-use
.. _PySpark getting started guide: https://towardsdatascience.com/how-to-get-started-with-pyspark-1adc142456ec
.. _Apache Spark documentation: https://spark.apache.org/docs/latest/
.. _stacks-usecase: https://github.com/intel/stacks-usecase
.. _Github Issue Classification: https://github.com/intel/stacks-usecase/tree/master/github-issue-classification