diff --git a/source/guides/stacks/dars.rst b/source/guides/stacks/dars.rst index 396dbefe..ab163113 100644 --- a/source/guides/stacks/dars.rst +++ b/source/guides/stacks/dars.rst @@ -6,25 +6,26 @@ Data Analytics Reference Stack This guide explains how to use the :abbr:`DARS (Data Analytics Reference Stack)`, and to optionally build your own DARS container image. -Any system that supports Docker\* containers can be used with DARS. This steps +Any system that supports Docker\* containers can be used with DARS. The steps in this guide use |CL-ATTR| as the host system. .. contents:: :local: :depth: 1 -The Data Analytics Reference Stack release -****************************************** +Overview +******** -The Data Analytics Reference Stack (DARS) provides developers and enterprises a straightforward, highly optimized software stack for storing and processing large -amounts of data. More detail is available on the -`DARS architecture and performance benchmarks`_. +The Data Analytics Reference Stack (DARS) provides developers and enterprises a straightforward, highly optimized software stack for storing and processing large amounts of data. More detail is available on the `DARS architecture and performance benchmarks`_. + +Stack Features +============== The Data Analytics Reference Stack provides two pre-built Docker images, available on `Docker Hub`_: * A |CL|-derived `DARS with OpenBlas`_ stack optimized for `OpenBLAS`_ -* A |CL|-derived `DARS with Intel® MKL`_ stack optimized for `MKL`_ +* A |CL|-derived `DARS with Intel® MKL`_ stack optimized for `MKL`_ (Intel® Math Kernel Library) We recommend you view the latest component versions for each image in the :file:`README` found in the `Data Analytics Reference Stack`_ GitHub\* @@ -41,78 +42,137 @@ in the |CL|-based containers may not be the latest released by |CL|. Using the Docker images *********************** -#. To immediately start using the latest stable DARS images, pull an image +Launching the Image +=================== + +#. To use the latest stable DARS images, pull an image directly from `Docker Hub`_. This example uses the `DARS with Intel® MKL`_ Docker image. + .. code-block:: bash + + docker pull clearlinux/stacks-dars-mkl + + #. Once you have downloaded the image, you can run it with .. code-block:: bash docker run -it --ulimit nofile=1000000:1000000 --name mkl - This will launch the image and drop you into a bash shell inside the - container. You will see output similar to the following: + This will launch the image and drop you into a bash shell inside the container. The :command:`--ulimit nofile=` parameter is required in order to increase the allowed number of open files for the Apache Spark engine. + + If you need to verify the name of the DARS image, you can use the :command:`docker image ls` command to see which images reside on your system. + + .. code-block:: bash + + docker image ls + .. code-block:: console - root@fd5155b89857 /root # spark-shell - spark-shell - Config directory: /usr/share/defaults/spark/ - Welcome to - ____ __ - / __/__ ___ _____/ /__ - _\ \/ _ \/ _ `/ __/ '_/ - /___/ .__/\_,_/_/ /_/\_\ version 2.4.0 - /_/ + REPOSITORY TAG IMAGE ID CREATED SIZE + clearlinux/stacks-dars-mkl test-img 49a70a22231f 23 hours ago 2.66GB + ubuntu latest 2ca708c1c9cc 7 days ago 64.2MB + katadocker/kata-deploy latest bd6dc92f8060 7 days ago 673MB + clearlinux/stacks-dars-mkl latest 2c9555536d5f 4 weeks ago 2.62GB - Using Scala version 2.12.7 (OpenJDK 64-Bit Server VM, Java 1.8.0-internal) - Type in expressions to have them evaluated. - Type :help for more information. - scala> - - The :command:`--ulimit nofile` parameter is currently required in order to - increase the number of open files opened at certain point by the spark - engine. Building DARS images -******************** +==================== -If you choose to build your own DARS container images, you can customize -them as needed. Use the provided Dockerfile as a baseline. +If you choose to build your own DARS container images, you can customize them as needed. Use the :file:`Dockerfile` included in the Github\* repository as your baseline. -To construct images with |CL|, start with a |CL| development platform that -has the :command:`containers-basic-dev` bundle installed. Learn more about -bundles and installing them by using :ref:`swupd-guide`. +To construct images with |CL|, start with a |CL| development platform that has the :command:`containers-basic-dev` bundle installed. Learn more about bundles and installing them by using :ref:`swupd-guide`. -#. Clone the `Data Analytics Reference Stack`_ GitHub\* repository. +#. The `Data Analytics Reference Stack`_ is part of the |CL| Project GitHub\* repository. Clone the :file:`dockerfiles` repository. .. code-block:: bash - git clone https://github.com/clearlinux/dockerfiles/tree/master/stacks/dars -b master + git clone https://github.com/clearlinux/dockerfiles.git -#. Inside the DARS directory, run :command:`make` to build OpenBLAS and MKL images. +#. Inside the :file:`stacks/dars/mkl` directory, use docker with the :file:`Dockerfile` to build the MKL image. .. code-block:: bash - make + cd ./dockerfiles/stacks/dlrs/mkl + docker build --no-cache -t clearlinux/stacks-dars-mkl . - Run :command:`make baseline` to build the baseline CentOS image. Depending on - the system, it may take a while to finish building. - - .. code-block:: bash - - make baseline #. Once completed, check the resulting images with :command:`Docker` .. code-block:: bash - docker images | grep dars + docker images | grep dars + +#. You can use any of the resulting images to launch fully functional containers. If you need to customize the containers, you can edit the provided :file:`Dockerfile`. + +Using Apache Spark\* in DARS +**************************** + +After launching the container, you can start Apache Spark with either the Scala or PySpark environment. For these examples we will use PySpark, which is the Python\* API for Apache Spark. + +.. code-block:: bash + + pyspark + + +Launching is as simple as this. Depending on your system configuration and capabilities, you may need to define proxy or memory allocation settings on the command line or in a config file for optimal performance. Refer to the `Apache Spark documentation`_ for more detail. + +After executing :command:`pyspark`, you will see output similar to this. + +.. code-block:: console + + root@fd5155b89857 /root # pyspark + Welcome to + ____ __ + / __/__ ___ _____/ /__ + _\ \/ _ \/ _ `/ __/ '_/ + /__ / .__/\_,_/_/ /_/\_\ version 2.4.0 + /_/ + + Using Python version 3.7.4 (default, Jul 13 2019 06:59:17) + SparkSession available as 'spark'. + >>> + + +Execute code directly in PySpark +================================ + +A simple example for verifying that pyspark is working correctly is to run a small python function from a `PySpark getting started guide`_ to estimate the value of Pi. Run these lines in the PySpark shell. + +.. code-block:: console + + import random + NUM_SAMPLES = 100000000 + def inside(p): + x, y = random.random(), random.random() + return x*x + y*y < 1 + + count = sc.parallelize(range(0, NUM_SAMPLES)).filter(inside).count() + pi = 4 * count / NUM_SAMPLES + print(“Pi is roughly”, pi) + + +Run Python programs with spark-submit +===================================== + +You can also run python scripts in Apache Spark from the command line. We'll use the Apache Spark example found in the :file:`/usr/share/apache-spark/examples/src/main/python/pi.py` file. Note that we have turned off the INFO and WARN messages in Apache Spark for this example. + +.. code-block:: console + + #spark-submit /usr/share/apache-spark/examples/src/main/python/pi.py + Config directory: /usr/share/defaults/spark/ + Pi is roughly 3.134700 + +DARS Usecase example +==================== + +The DARS container is used in conjunction with the Deep Learning Reference Stack container to implement a real world use case. Refer to the `Github Issue Classification`_ Usecase found in the `stacks-usecase`_ repository for a walkthrough. This usecase is implemented using the Scala environment, rather than PySpark. + + -#. You can use any of the resulting images to launch fully functional containers. - If you need to customize the containers, you can edit the provided :file:`Dockerfile`. .. _Data Analytics Reference Stack: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dars @@ -131,3 +191,11 @@ bundles and installing them by using :ref:`swupd-guide`. .. _DARS architecture and performance benchmarks: https://clearlinux.org/stacks/data-analytics-stack-v1 .. _DARS Terms of Use: https://clearlinux.org/stacks/data-analytics/terms-of-use + +.. _PySpark getting started guide: https://towardsdatascience.com/how-to-get-started-with-pyspark-1adc142456ec + +.. _Apache Spark documentation: https://spark.apache.org/docs/latest/ + +.. _stacks-usecase: https://github.com/intel/stacks-usecase + +.. _Github Issue Classification: https://github.com/intel/stacks-usecase/tree/master/github-issue-classification