From c1f75816927e7397fa35c1feda51986b014108e2 Mon Sep 17 00:00:00 2001 From: Rodrigo Caballero Date: Thu, 21 Sep 2017 16:26:08 -0500 Subject: [PATCH 1/4] Add Hadoop use case tutorial. Signed-off-by: Rodrigo Caballero --- source/clear-linux/tutorials/hadoop.rst | 177 +++++++++++++++++++++ source/clear-linux/tutorials/tutorials.rst | 1 + 2 files changed, 178 insertions(+) create mode 100644 source/clear-linux/tutorials/hadoop.rst diff --git a/source/clear-linux/tutorials/hadoop.rst b/source/clear-linux/tutorials/hadoop.rst new file mode 100644 index 00000000..676384a1 --- /dev/null +++ b/source/clear-linux/tutorials/hadoop.rst @@ -0,0 +1,177 @@ +.. _hadoop: + +Set up a single node cluster with Hadoop\* +########################################## + +This tutorial walks you through the process of installing, configuring, and +running Apache\* Hadoop on |CLOSIA|. The Apache Hadoop software library is a +framework for distributed processing of large data sets across clusters of +computers using simple programming models. It is designed to scale up from +single servers to thousands of machines, with each machine offering local +computation and storage. + +Prerequisites +************* + +Before following this tutorial, you should follow the +:ref:`bare-metal-install` to ensure you have installed |CLOSIA|. + +Before you install any new packages, update |CL| with the following command: + +.. code-block:: bash + + sudo swupd update + +For the purposes of this tutorial, we will install Hadoop in a single machine +running both the master and slave daemons. + +Install Apache Hadoop +********************* + +Apache Hadoop is included in the `big-data-basic` bundle. To install the +framework, enter the following command: + +.. code-block:: bash + + sudo swupd bundle-add big-data-basic + +Configure Apache Hadoop +*********************** + +#. To create the configuration directory, enter the following command: + + .. code-block:: bash + + sudo mkdir /etc/hadoop + +#. Copy the defaults from :file:`/usr/share/defaults/hadoop` to + :file:`/etc/hadoop` with the following command: + + .. code-block:: bash + + $ sudo cp /usr/share/defaults/hadoop/* /etc/hadoop + + .. note:: Remember, you should never modify the files under the + :file:`/usr/share/defaults` directory since swupd will overwrite them. + +Once all configuration files are in :file:`/etc/hadoop`, we must edit them +according to our needs. The first file, :file:`/etc/hadoop/core-site.xml`, is +responsible to inform the Hadoop daemon of where `NameNode` running. The +`NameNode` server is the master server managing the files system namespace +and regulating the clients' access to files. + +In this tutorial, our `NameNode` runs in our `localhost`. Follow these steps +to set it up correctly: + +#. Open the :file:`/etc/hadoop/core-site.xml` using the editor of your + choice and modify the file as follows: + + .. code-block:: xml + + + + + + fs.default.name + hdfs://localhost:9000 + + + +#. Edit :file:`/etc/hadoop/hdfs-site.xml`. This file configures the + :abbr:`HDFS (Hadoop Distributed File System)` daemons. This includes + things like the list of permitted and excluded data nodes or the size of + the blocks. In this example, we are setting the number of block + replication to 1 from the default of 3 as follows: + + .. code-block:: xml + :emphasize-lines: 6 + + + + + + dfs.replication + 1 + + + dfs.permission + false + + + +#. Edit :file:`/etc/hadoop/mapred-site.xml`. This file configures all daemons + related to MapReduce: `JobTracker` and `TaskTrackers`. With MapReduce, + Hadoop can process big amounts of data in multiple systems. In our + example, we set :abbr:`YARN (Yet Another Resource Manager)` as our runtime + framework for executing MapReduce jobs as follows: + + .. code-block:: xml + :emphasize-lines: 5,6 + + + + + + mapreduce.framework.name + yarn + + + +#. Edit :file:`/etc/hadoop/yarn-site.xml`. This file configures all daemons + related to YARN: `ResourceManager` and `NodeManager`. In our example, we + implement the `mapreduce_shuffle` service, which is the default as follows: + + .. code-block:: xml + :emphasize-lines: 4,5,8,9 + + + + + yarn.nodemanager.aux-services + mapreduce_shuffle + + + yarn.nodemanager.auxservices.mapreduce.shuffle.class + org.apache.hadoop.mapred.ShuffleHandler + + + +Run the Hadoop daemons +********************** + +With all the configuration files properly edited, we are ready to start the +daemons. + +When we format `NameNode`, it formats the meta-data related to data-nodes. +Thus, all the information on the data nodes is lost and the nodes can be +reused for new data. + +#. Format `NameNode` with the following command: + + .. code-block:: bash + + hdfs namenode -format + +#. Start the DFS daemons `NameNode` and `DataNodes` with the following command: + + .. code-block:: bash + + start-dfs.sh + +#. Start the YARN daemons `ResourceManager` and `NodeManager` with the + following command: + + .. code-block:: bash + + start-yarn.sh + +#. Ensure everything is running as expected with the following command: + + .. code-block:: bash + + jps + +**Congratulations!** + +Your single node Hadoop cluster is up and running! + diff --git a/source/clear-linux/tutorials/tutorials.rst b/source/clear-linux/tutorials/tutorials.rst index f0cef09f..e1a660fd 100644 --- a/source/clear-linux/tutorials/tutorials.rst +++ b/source/clear-linux/tutorials/tutorials.rst @@ -14,3 +14,4 @@ specific |CLOSIA| use cases. machine-learning/machine-learning azure multi-boot/multi-boot + hadoop From b57e08b19b8d3fa526d3665998236adef687865f Mon Sep 17 00:00:00 2001 From: Rodrigo Caballero Date: Tue, 26 Sep 2017 14:30:08 -0500 Subject: [PATCH 2/4] Update Hadoop tutorial based on review feedback. Signed-off-by: Rodrigo Caballero --- source/clear-linux/tutorials/hadoop.rst | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/source/clear-linux/tutorials/hadoop.rst b/source/clear-linux/tutorials/hadoop.rst index 676384a1..76dc52c6 100644 --- a/source/clear-linux/tutorials/hadoop.rst +++ b/source/clear-linux/tutorials/hadoop.rst @@ -51,12 +51,13 @@ Configure Apache Hadoop $ sudo cp /usr/share/defaults/hadoop/* /etc/hadoop - .. note:: Remember, you should never modify the files under the - :file:`/usr/share/defaults` directory since swupd will overwrite them. + .. note:: Since |CL| is a stateless system, you should never modify the + files under the :file:`/usr/share/defaults` directory. The software + updater will overwrite those files. Once all configuration files are in :file:`/etc/hadoop`, we must edit them according to our needs. The first file, :file:`/etc/hadoop/core-site.xml`, is -responsible to inform the Hadoop daemon of where `NameNode` running. The +responsible to inform the Hadoop daemon of where `NameNode` is running. The `NameNode` server is the master server managing the files system namespace and regulating the clients' access to files. @@ -78,9 +79,9 @@ to set it up correctly: #. Edit :file:`/etc/hadoop/hdfs-site.xml`. This file configures the - :abbr:`HDFS (Hadoop Distributed File System)` daemons. This includes - things like the list of permitted and excluded data nodes or the size of - the blocks. In this example, we are setting the number of block + :abbr:`HDFS (Hadoop Distributed File System)` daemons. This configuration + includes the list of permitted and excluded data nodes and the size of + said blocks. In this example, we are setting the number of block replication to 1 from the default of 3 as follows: .. code-block:: xml @@ -158,6 +159,18 @@ reused for new data. start-dfs.sh +#. The console outputs: + + .. code-block:: console + + Starting namenodes on [localhost] + The authenticity of host 'localhost (::1)' can't be established. + ECDSA key fingerprint is + SHA256:97e+7TnomsS9W7GjFPjzY75HGBp+f1y6sA+ZFcOPIPU. + Are you sure you want to continue connecting (yes/no)? + + Enter `yes` to continue. + #. Start the YARN daemons `ResourceManager` and `NodeManager` with the following command: From 2d6c8e87642c7f577b8c93e32478456df739c0fc Mon Sep 17 00:00:00 2001 From: Rodrigo Caballero Date: Wed, 18 Oct 2017 15:08:16 -0500 Subject: [PATCH 3/4] Add SSH connectivity and wordcount example to Hadoop tutorial. Beyond adding those two sections, some commands were updated and other minor issues were fixed. Signed-off-by: Rodrigo Caballero --- source/clear-linux/tutorials/hadoop.rst | 73 +++++++++++++++++++++++-- 1 file changed, 69 insertions(+), 4 deletions(-) diff --git a/source/clear-linux/tutorials/hadoop.rst b/source/clear-linux/tutorials/hadoop.rst index 76dc52c6..cf0c7bba 100644 --- a/source/clear-linux/tutorials/hadoop.rst +++ b/source/clear-linux/tutorials/hadoop.rst @@ -137,6 +137,29 @@ to set it up correctly: +Configure your SSH key +********************** + +#. Create a SSH key. If you already have one, just skip this step. + + .. code-block:: bash + + sudo ssh-keygen -t rsa + + +#. Copy the key to your authorized keys. + + .. code-block:: bash + + sudo cat /root/.ssh/id_rsa.pub | sudo tee -a /root/.ssh/authorized_keys + +#. Log into the localhost. If no password prompt appears, you are ready to + run the Hadoop daemons. + + .. code-block:: bash + + sudo ssh localhost + Run the Hadoop daemons ********************** @@ -151,13 +174,13 @@ reused for new data. .. code-block:: bash - hdfs namenode -format + sudo hdfs namenode -format #. Start the DFS daemons `NameNode` and `DataNodes` with the following command: .. code-block:: bash - start-dfs.sh + sudo start-dfs.sh #. The console outputs: @@ -176,15 +199,57 @@ reused for new data. .. code-block:: bash - start-yarn.sh + sudo start-yarn.sh #. Ensure everything is running as expected with the following command: .. code-block:: bash - jps + sudo jps + +#. The console output should be similar to: + + .. code-block:: console + + 22674 DataNode + 26228 Jps + 22533 NameNode + 23046 ResourceManager + 22854 SecondaryNameNode + 23150 NodeManager + +Run the MapReduce wordcount example +*********************************** + +#. Create the input directory. + + .. code-block:: bash + + sudo hdfs dfs -mkdir -p /user/root/input + +#. Copy a file from the local file system to the HDFS. + + .. code-block:: bash + + sudo hdfs dfs -copyFromLocal local-file /user/root/input + +#. Run the wordcount example. + + .. code-block:: bash + + sudo hadoop jar /usr/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar wordcount input output + +#. Read output file "part-r-00000". This file contains the number of times + each word appears in the file. + + .. code-block:: bash + + sudo hdfs dfs -cat /user/root/output/part-r-00000 **Congratulations!** +You successfully installed and setup a single node Hadoop cluster. +Additionally, you ran a simple wordcount example. + Your single node Hadoop cluster is up and running! From 1817bd8cdb8d1812a0d43c8283633408875815fa Mon Sep 17 00:00:00 2001 From: Rodrigo Caballero Date: Mon, 30 Oct 2017 10:41:13 -0600 Subject: [PATCH 4/4] Edited for clarity, added missing articles and nouns. Signed-off-by: Rodrigo Caballero --- source/clear-linux/tutorials/hadoop.rst | 49 +++++++++++++------------ 1 file changed, 25 insertions(+), 24 deletions(-) diff --git a/source/clear-linux/tutorials/hadoop.rst b/source/clear-linux/tutorials/hadoop.rst index cf0c7bba..a489c855 100644 --- a/source/clear-linux/tutorials/hadoop.rst +++ b/source/clear-linux/tutorials/hadoop.rst @@ -55,16 +55,16 @@ Configure Apache Hadoop files under the :file:`/usr/share/defaults` directory. The software updater will overwrite those files. -Once all configuration files are in :file:`/etc/hadoop`, we must edit them -according to our needs. The first file, :file:`/etc/hadoop/core-site.xml`, is -responsible to inform the Hadoop daemon of where `NameNode` is running. The -`NameNode` server is the master server managing the files system namespace -and regulating the clients' access to files. +Once all the configuration files are in :file:`/etc/hadoop`, we must edit +them to fit our needs. The `NameNode` server is the master server. It manages +the namespace of the files system and regulates the clients' access to files. +The first file we edit, :file:`/etc/hadoop/core-site.xml`, informs the Hadoop +daemon where `NameNode` is running. In this tutorial, our `NameNode` runs in our `localhost`. Follow these steps to set it up correctly: -#. Open the :file:`/etc/hadoop/core-site.xml` using the editor of your +#. Open the :file:`/etc/hadoop/core-site.xml` file using the editor of your choice and modify the file as follows: .. code-block:: xml @@ -78,7 +78,7 @@ to set it up correctly: -#. Edit :file:`/etc/hadoop/hdfs-site.xml`. This file configures the +#. Edit the :file:`/etc/hadoop/hdfs-site.xml` file. This file configures the :abbr:`HDFS (Hadoop Distributed File System)` daemons. This configuration includes the list of permitted and excluded data nodes and the size of said blocks. In this example, we are setting the number of block @@ -100,11 +100,11 @@ to set it up correctly: -#. Edit :file:`/etc/hadoop/mapred-site.xml`. This file configures all daemons - related to MapReduce: `JobTracker` and `TaskTrackers`. With MapReduce, - Hadoop can process big amounts of data in multiple systems. In our - example, we set :abbr:`YARN (Yet Another Resource Manager)` as our runtime - framework for executing MapReduce jobs as follows: +#. Edit the :file:`/etc/hadoop/mapred-site.xml` file. This file configures + all daemons related to `MapReduce`: `JobTracker` and `TaskTrackers`. With + `MapReduce`, Hadoop can process big amounts of data in multiple systems. In + our example, we set :abbr:`YARN (Yet Another Resource Manager)` as our + runtime framework for executing `MapReduce` jobs as follows: .. code-block:: xml :emphasize-lines: 5,6 @@ -118,9 +118,10 @@ to set it up correctly: -#. Edit :file:`/etc/hadoop/yarn-site.xml`. This file configures all daemons - related to YARN: `ResourceManager` and `NodeManager`. In our example, we - implement the `mapreduce_shuffle` service, which is the default as follows: +#. Edit the :file:`/etc/hadoop/yarn-site.xml` file. This file configures all + daemons related to `YARN`: `ResourceManager` and `NodeManager`. In our + example, we implement the `mapreduce_shuffle` service, which is the + default as follows: .. code-block:: xml :emphasize-lines: 4,5,8,9 @@ -140,7 +141,7 @@ to set it up correctly: Configure your SSH key ********************** -#. Create a SSH key. If you already have one, just skip this step. +#. Create a SSH key. If you already have one, skip this step. .. code-block:: bash @@ -166,23 +167,23 @@ Run the Hadoop daemons With all the configuration files properly edited, we are ready to start the daemons. -When we format `NameNode`, it formats the meta-data related to data-nodes. -Thus, all the information on the data nodes is lost and the nodes can be -reused for new data. +When we format the `NameNode` server, it formats the meta-data related to +data nodes. Thus, all the information on the data nodes is lost and the nodes +can be reused for new data. -#. Format `NameNode` with the following command: +#. Format the `NameNode` server with the following command: .. code-block:: bash sudo hdfs namenode -format -#. Start the DFS daemons `NameNode` and `DataNodes` with the following command: +#. Start the DFS in `NameNode` and `DataNodes` with the following command: .. code-block:: bash sudo start-dfs.sh -#. The console outputs: +#. The console output should be similar to: .. code-block:: console @@ -194,7 +195,7 @@ reused for new data. Enter `yes` to continue. -#. Start the YARN daemons `ResourceManager` and `NodeManager` with the +#. Start the `YARN` daemons `ResourceManager` and `NodeManager` with the following command: .. code-block:: bash @@ -233,7 +234,7 @@ Run the MapReduce wordcount example sudo hdfs dfs -copyFromLocal local-file /user/root/input -#. Run the wordcount example. +#. Run the `wordcount` example. .. code-block:: bash