From e4a917545d6c813eab468718e856016e145db27b Mon Sep 17 00:00:00 2001 From: Jose Lamego Date: Mon, 12 Feb 2018 22:47:20 +0000 Subject: [PATCH 1/5] spark.rst: Add standalone-Spark use case tutorial Signed-off-by: Jose Lamego --- source/clear-linux/tutorials/spark.rst | 110 +++++++++++++++++++++ source/clear-linux/tutorials/tutorials.rst | 2 +- 2 files changed, 111 insertions(+), 1 deletion(-) create mode 100644 source/clear-linux/tutorials/spark.rst diff --git a/source/clear-linux/tutorials/spark.rst b/source/clear-linux/tutorials/spark.rst new file mode 100644 index 00000000..83cf18c6 --- /dev/null +++ b/source/clear-linux/tutorials/spark.rst @@ -0,0 +1,110 @@ +.. _spark: + +Set up a standalone cluster system using Spark*\ +################################################ + +This tutorial walks you through the process of installing, configuring, and running Apache* Spark on Clear Linux OS for Intel(r) Architecture. Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming. + +Prerequisites +************* + +Before following this tutorial, you should follow the :ref:`bare-metal-install` to ensure you have installed |CLOSIA|. + +Before you install any new packages, update |CL| with the following command: + +.. code-block:: bash + + $ sudo swupd update + +For the purposes of this tutorial, we will install Spark in a single machine running both the master and a worker daemons. + +Install Apache Spark +******************** + +Apache Spark is included in the *big-data-basic* bundle. To install the framework, enter the following command: + +.. code-block:: bash + + $ sudo swupd bundle-add big-data-basic + +Configure Apache Spark +********************** + +#. To create the configuration directory, enter the following command: + + .. code-block:: bash + + $ sudo mkdir /etc/spark + +#. Copy the default templates from */usr/share/defaults/spark* to */etc/spark* with the following command: + + .. code-block:: bash + + $ sudo cp /usr/share/defaults/spark/* /etc/spark + + .. note:: Since |CL| is a stateless system, you should never modify the files under the */usr/share/defaults* directory. The software updater will overwrite those files. + +Once all the configuration templates are in */etc/spark*, we must edit them to fit our needs. + +#. Create the basic configuration files from the default templates with the following commands: + + .. code-block:: bash + + $ sudo cp /etc/spark/spark-defaults.conf.template /etc/spark/spark-defaults.conf + $ sudo cp /etc/spark/spark-env.sh.template /etc/spark/spark-env.sh + $ sudo cp /etc/spark/log4j.properties.template /etc/spark/log4j.properties + +#. Open the */etc/spark/spark-env.sh* file using the editor of your choice and add the *SPARK_MASTER_HOST* variable with your localhost IP address as follows (substitute the address with your actual value. You can view this with the command *hostname -I*): + + .. code-block:: xml + + SPARK_MASTER_HOST="10.300.200.100" + + .. note:: Although setting the *SPARK_MASTER_HOST* variable to an IP address + is not required to run Spark in standalone mode, by doing so now you + will be able to navigate through the master's web UI pages without + issues and view information there from both the worker and the + application that will be started in next steps of this tutorial. + +#. Open the */etc/spark/spark-defaults.conf* file using the editor of your choice and add the master host's IP Address and port *7077* to the variable *spark.master* (substitute the address with the value you entered in previous step): + + .. code-block:: xml + + spark.master spark://10.300.200.100:7077 + +Start the master server and a worker +************************************ + +#. Start the master server by executing: + + .. code-block:: bash + + $ sudo /usr/share/apache-spark/sbin/./start-master.sh + +#. Start one worker and connect it to the master through port *7077* via the following command (substitute the IP address with the one you entered in step 2 from the previous section): + + .. code-block:: bash + + $ sudo /usr/share/apache-spark/sbin/./start-slave.sh spark://10.300.200.100:7077 + + +You should now be able to view the worker information in the master's web UI by using the internet browser of your choice and entering the master's IP address, followed by the port *8080* in the address bar (substitute the address with the value entered in step 4 from previous section): + + .. code-block:: xml + + http://10.300.200.100:8080 + +Run the Spark worcount example +****************************** + +#. Run the wordcount example using a file in your local host and output the results to a new file by using the following command (substitute the file path and name with actual values in your host): + + .. code-block:: bash + + $ sudo spark-submit /usr/share/apache-spark/examples/src/main/python/wordcount.py ~/Documents/example_file > ~/Documents/results + +You should now be able to view the application information in the master's web UI by using the internet browser of your choice and entering the master's IP address, followed by the port *8080* in the address bar (substitute the address with the value entered in previous sections). The results of the *wordcount* application on the input file can be viewed in the output file. + +Congratulations! + +You successfully installed and setup a standalone Apache Spark cluster. Additionally, you ran a simple wordcount example. diff --git a/source/clear-linux/tutorials/tutorials.rst b/source/clear-linux/tutorials/tutorials.rst index ad1b3227..b6a11ed3 100644 --- a/source/clear-linux/tutorials/tutorials.rst +++ b/source/clear-linux/tutorials/tutorials.rst @@ -18,4 +18,4 @@ specific |CLOSIA| use cases. fmv aws-web/aws-web telemetry-backend/telemetry-backend - + spark From 2ec956e5a7f5c505ef6b645c38dc9f4f25182da0 Mon Sep 17 00:00:00 2001 From: MCamp859 Date: Thu, 22 Feb 2018 14:01:13 -0500 Subject: [PATCH 2/5] Edited for grammar, formatting, and text flow. Signed-off-by: MCamp859 --- source/clear-linux/tutorials/spark.rst | 106 +++++++++++++++---------- 1 file changed, 66 insertions(+), 40 deletions(-) diff --git a/source/clear-linux/tutorials/spark.rst b/source/clear-linux/tutorials/spark.rst index 83cf18c6..eb261cdf 100644 --- a/source/clear-linux/tutorials/spark.rst +++ b/source/clear-linux/tutorials/spark.rst @@ -1,14 +1,26 @@ -.. _spark: + .. _spark: -Set up a standalone cluster system using Spark*\ -################################################ +Set up a standalone cluster system using Apache\* Spark\* +######################################################### -This tutorial walks you through the process of installing, configuring, and running Apache* Spark on Clear Linux OS for Intel(r) Architecture. Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming. +This tutorial describes how to install, configure, and run Apache Spark on +Clear Linux OS for Intel(r) Architecture. Apache Spark is a fast +general-purpose cluster computing system with the following features: + +* Provides high-level APIs in Java\*, Scala\*, Python\*, and R\*. +* Includes an optimized engine that supports general execution graphs. +* Supports high-level tools including Spark SQL, MLlib, GraphX, and Spark + Streaming. + +In this tutorial, you will install Spark on a single machine running the +master daemon and a worker daemon. Prerequisites ************* -Before following this tutorial, you should follow the :ref:`bare-metal-install` to ensure you have installed |CLOSIA|. +This tutorial assumes you have installed |CL| on your host system. +For detailed instructions on installing |CL| on a bare metal system, visit +the :ref:`bare metal installation tutorial`. Before you install any new packages, update |CL| with the following command: @@ -16,12 +28,11 @@ Before you install any new packages, update |CL| with the following command: $ sudo swupd update -For the purposes of this tutorial, we will install Spark in a single machine running both the master and a worker daemons. - Install Apache Spark ******************** -Apache Spark is included in the *big-data-basic* bundle. To install the framework, enter the following command: +Apache Spark is included in the :file:`big-data-basic` bundle. To install the +framework, enter: .. code-block:: bash @@ -30,23 +41,25 @@ Apache Spark is included in the *big-data-basic* bundle. To install the framewor Configure Apache Spark ********************** -#. To create the configuration directory, enter the following command: +#. Create the configuration directory with the command: .. code-block:: bash $ sudo mkdir /etc/spark -#. Copy the default templates from */usr/share/defaults/spark* to */etc/spark* with the following command: +#. Copy the default templates from :file:`/usr/share/defaults/spark` to + :file:`/etc/spark` with the command: .. code-block:: bash $ sudo cp /usr/share/defaults/spark/* /etc/spark - .. note:: Since |CL| is a stateless system, you should never modify the files under the */usr/share/defaults* directory. The software updater will overwrite those files. + .. note:: Since |CL| is a stateless system, you should never modify the + files under the :file:`/usr/share/defaults` directory. The software + updater overwrites those files. -Once all the configuration templates are in */etc/spark*, we must edit them to fit our needs. -#. Create the basic configuration files from the default templates with the following commands: +#. Copy the template files below to create custom configuration files: .. code-block:: bash @@ -54,57 +67,70 @@ Once all the configuration templates are in */etc/spark*, we must edit them to f $ sudo cp /etc/spark/spark-env.sh.template /etc/spark/spark-env.sh $ sudo cp /etc/spark/log4j.properties.template /etc/spark/log4j.properties -#. Open the */etc/spark/spark-env.sh* file using the editor of your choice and add the *SPARK_MASTER_HOST* variable with your localhost IP address as follows (substitute the address with your actual value. You can view this with the command *hostname -I*): +#. Edit the :file:`/etc/spark/spark-env.sh` file and add the + :option:`SPARK_MASTER_HOST` variable. Replace the example + address below with your localhost IP address. (View your IP address with + the :command:`hostname -I` command.) .. code-block:: xml SPARK_MASTER_HOST="10.300.200.100" - - .. note:: Although setting the *SPARK_MASTER_HOST* variable to an IP address - is not required to run Spark in standalone mode, by doing so now you - will be able to navigate through the master's web UI pages without - issues and view information there from both the worker and the - application that will be started in next steps of this tutorial. - -#. Open the */etc/spark/spark-defaults.conf* file using the editor of your choice and add the master host's IP Address and port *7077* to the variable *spark.master* (substitute the address with the value you entered in previous step): + + .. note:: This optional step enables the master's web user interface to + view information needed later in this tutorial. + +#. Edit the :file:`/etc/spark/spark-defaults.conf` file and update the + :option:`spark.master` variable with the :option:`SPARK_MASTER_HOST` address + from the previous step and port :option:`7077`. .. code-block:: xml spark.master spark://10.300.200.100:7077 - -Start the master server and a worker -************************************ - -#. Start the master server by executing: + +Start the master server and a worker daemon +******************************************* + +#. Start the master server using: .. code-block:: bash $ sudo /usr/share/apache-spark/sbin/./start-master.sh - -#. Start one worker and connect it to the master through port *7077* via the following command (substitute the IP address with the one you entered in step 2 from the previous section): + +#. Start one worker daemon and connect it to the master using the + :option:`spark.master` variable defined earlier: .. code-block:: bash $ sudo /usr/share/apache-spark/sbin/./start-slave.sh spark://10.300.200.100:7077 - -You should now be able to view the worker information in the master's web UI by using the internet browser of your choice and entering the master's IP address, followed by the port *8080* in the address bar (substitute the address with the value entered in step 4 from previous section): +#. Open an internet browser and view the worker daemon information using + the master's IP address and port :option:`8080`: .. code-block:: xml http://10.300.200.100:8080 - -Run the Spark worcount example -****************************** -#. Run the wordcount example using a file in your local host and output the results to a new file by using the following command (substitute the file path and name with actual values in your host): +Run the Spark wordcount example +******************************* + +#. Run the wordcount example using a file on your local host and output the + results to a new file with the following command: .. code-block:: bash $ sudo spark-submit /usr/share/apache-spark/examples/src/main/python/wordcount.py ~/Documents/example_file > ~/Documents/results - -You should now be able to view the application information in the master's web UI by using the internet browser of your choice and entering the master's IP address, followed by the port *8080* in the address bar (substitute the address with the value entered in previous sections). The results of the *wordcount* application on the input file can be viewed in the output file. - -Congratulations! -You successfully installed and setup a standalone Apache Spark cluster. Additionally, you ran a simple wordcount example. +#. Open an internet browser and view the application information using + the master's IP address and port :option:`8080`: + + .. code-block:: xml + + http://10.300.200.100:8080 + +#. View the results of the wordcount application in the results file specified + in step 1. + +**Congratulations!** + +You successfully installed and set up a standalone Apache Spark cluster. +Additionally, you ran a simple wordcount example. From f817a58066bcc762af47a70d86eb695311972194 Mon Sep 17 00:00:00 2001 From: MCamp859 Date: Wed, 28 Feb 2018 11:38:19 -0500 Subject: [PATCH 3/5] Edited for grammar, format, and text flow. Signed-off-by: MCamp859 --- source/clear-linux/tutorials/spark.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/source/clear-linux/tutorials/spark.rst b/source/clear-linux/tutorials/spark.rst index eb261cdf..3b5b318f 100644 --- a/source/clear-linux/tutorials/spark.rst +++ b/source/clear-linux/tutorials/spark.rst @@ -68,9 +68,9 @@ Configure Apache Spark $ sudo cp /etc/spark/log4j.properties.template /etc/spark/log4j.properties #. Edit the :file:`/etc/spark/spark-env.sh` file and add the - :option:`SPARK_MASTER_HOST` variable. Replace the example - address below with your localhost IP address. (View your IP address with - the :command:`hostname -I` command.) + :option:`SPARK_MASTER_HOST` variable. Replace the example address below + with your localhost IP address. (View your IP address using the + :command:`hostname -I` command.) .. code-block:: xml From 62265b552ca48bd1db04e7c7b11021ddf4cc2bf3 Mon Sep 17 00:00:00 2001 From: MCamp859 Date: Thu, 1 Mar 2018 15:33:11 -0500 Subject: [PATCH 4/5] Removed prompt character. Signed-off-by: MCamp859 --- source/clear-linux/tutorials/spark.rst | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/source/clear-linux/tutorials/spark.rst b/source/clear-linux/tutorials/spark.rst index 3b5b318f..e090e6ac 100644 --- a/source/clear-linux/tutorials/spark.rst +++ b/source/clear-linux/tutorials/spark.rst @@ -26,7 +26,7 @@ Before you install any new packages, update |CL| with the following command: .. code-block:: bash - $ sudo swupd update + sudo swupd update Install Apache Spark ******************** @@ -36,7 +36,7 @@ framework, enter: .. code-block:: bash - $ sudo swupd bundle-add big-data-basic + sudo swupd bundle-add big-data-basic Configure Apache Spark ********************** @@ -45,14 +45,14 @@ Configure Apache Spark .. code-block:: bash - $ sudo mkdir /etc/spark + sudo mkdir /etc/spark #. Copy the default templates from :file:`/usr/share/defaults/spark` to :file:`/etc/spark` with the command: .. code-block:: bash - $ sudo cp /usr/share/defaults/spark/* /etc/spark + sudo cp /usr/share/defaults/spark/* /etc/spark .. note:: Since |CL| is a stateless system, you should never modify the files under the :file:`/usr/share/defaults` directory. The software @@ -63,9 +63,9 @@ Configure Apache Spark .. code-block:: bash - $ sudo cp /etc/spark/spark-defaults.conf.template /etc/spark/spark-defaults.conf - $ sudo cp /etc/spark/spark-env.sh.template /etc/spark/spark-env.sh - $ sudo cp /etc/spark/log4j.properties.template /etc/spark/log4j.properties + sudo cp /etc/spark/spark-defaults.conf.template /etc/spark/spark-defaults.conf + sudo cp /etc/spark/spark-env.sh.template /etc/spark/spark-env.sh + sudo cp /etc/spark/log4j.properties.template /etc/spark/log4j.properties #. Edit the :file:`/etc/spark/spark-env.sh` file and add the :option:`SPARK_MASTER_HOST` variable. Replace the example address below @@ -94,14 +94,14 @@ Start the master server and a worker daemon .. code-block:: bash - $ sudo /usr/share/apache-spark/sbin/./start-master.sh + sudo /usr/share/apache-spark/sbin/./start-master.sh #. Start one worker daemon and connect it to the master using the :option:`spark.master` variable defined earlier: .. code-block:: bash - $ sudo /usr/share/apache-spark/sbin/./start-slave.sh spark://10.300.200.100:7077 + sudo /usr/share/apache-spark/sbin/./start-slave.sh spark://10.300.200.100:7077 #. Open an internet browser and view the worker daemon information using the master's IP address and port :option:`8080`: @@ -118,7 +118,7 @@ Run the Spark wordcount example .. code-block:: bash - $ sudo spark-submit /usr/share/apache-spark/examples/src/main/python/wordcount.py ~/Documents/example_file > ~/Documents/results + sudo spark-submit /usr/share/apache-spark/examples/src/main/python/wordcount.py ~/Documents/example_file > ~/Documents/results #. Open an internet browser and view the application information using the master's IP address and port :option:`8080`: From 510b40fc3d8b6799d07b6be31ff16f9390b2172b Mon Sep 17 00:00:00 2001 From: MCamp859 Date: Mon, 5 Mar 2018 14:30:09 -0500 Subject: [PATCH 5/5] Completed requested edits. Added |CLOSIA|, removed incorrect option roles, added envvar role, removed XML language in code blocks, and minor text edits. Signed-off-by: MCamp859 --- source/clear-linux/tutorials/spark.rst | 30 ++++++++++++-------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/source/clear-linux/tutorials/spark.rst b/source/clear-linux/tutorials/spark.rst index e090e6ac..7941de1d 100644 --- a/source/clear-linux/tutorials/spark.rst +++ b/source/clear-linux/tutorials/spark.rst @@ -4,8 +4,8 @@ Set up a standalone cluster system using Apache\* Spark\* ######################################################### This tutorial describes how to install, configure, and run Apache Spark on -Clear Linux OS for Intel(r) Architecture. Apache Spark is a fast -general-purpose cluster computing system with the following features: +|CLOSIA|. Apache Spark is a fast general-purpose cluster computing system with +the following features: * Provides high-level APIs in Java\*, Scala\*, Python\*, and R\*. * Includes an optimized engine that supports general execution graphs. @@ -68,11 +68,11 @@ Configure Apache Spark sudo cp /etc/spark/log4j.properties.template /etc/spark/log4j.properties #. Edit the :file:`/etc/spark/spark-env.sh` file and add the - :option:`SPARK_MASTER_HOST` variable. Replace the example address below - with your localhost IP address. (View your IP address using the - :command:`hostname -I` command.) + :envvar:`SPARK_MASTER_HOST` variable. Replace the example address below + with your localhost IP address. View your IP address using the + :command:`hostname -I` command. - .. code-block:: xml + .. code-block:: SPARK_MASTER_HOST="10.300.200.100" @@ -80,10 +80,9 @@ Configure Apache Spark view information needed later in this tutorial. #. Edit the :file:`/etc/spark/spark-defaults.conf` file and update the - :option:`spark.master` variable with the :option:`SPARK_MASTER_HOST` address - from the previous step and port :option:`7077`. + `spark.master` variable with the `SPARK_MASTER_HOST` address and port `7077`. - .. code-block:: xml + .. code-block:: spark.master spark://10.300.200.100:7077 @@ -97,16 +96,16 @@ Start the master server and a worker daemon sudo /usr/share/apache-spark/sbin/./start-master.sh #. Start one worker daemon and connect it to the master using the - :option:`spark.master` variable defined earlier: + `spark.master` variable defined earlier: .. code-block:: bash sudo /usr/share/apache-spark/sbin/./start-slave.sh spark://10.300.200.100:7077 #. Open an internet browser and view the worker daemon information using - the master's IP address and port :option:`8080`: + the master's IP address and port `8080`: - .. code-block:: xml + .. code-block:: http://10.300.200.100:8080 @@ -121,14 +120,13 @@ Run the Spark wordcount example sudo spark-submit /usr/share/apache-spark/examples/src/main/python/wordcount.py ~/Documents/example_file > ~/Documents/results #. Open an internet browser and view the application information using - the master's IP address and port :option:`8080`: + the master's IP address and port `8080`: - .. code-block:: xml + .. code-block:: http://10.300.200.100:8080 -#. View the results of the wordcount application in the results file specified - in step 1. +#. View the results of the wordcount application in the :file:`~/Documents/results` file. **Congratulations!**