diff --git a/source/clear-linux/tutorials/hadoop.rst b/source/clear-linux/tutorials/hadoop.rst new file mode 100644 index 00000000..a489c855 --- /dev/null +++ b/source/clear-linux/tutorials/hadoop.rst @@ -0,0 +1,256 @@ +.. _hadoop: + +Set up a single node cluster with Hadoop\* +########################################## + +This tutorial walks you through the process of installing, configuring, and +running Apache\* Hadoop on |CLOSIA|. The Apache Hadoop software library is a +framework for distributed processing of large data sets across clusters of +computers using simple programming models. It is designed to scale up from +single servers to thousands of machines, with each machine offering local +computation and storage. + +Prerequisites +************* + +Before following this tutorial, you should follow the +:ref:`bare-metal-install` to ensure you have installed |CLOSIA|. + +Before you install any new packages, update |CL| with the following command: + +.. code-block:: bash + + sudo swupd update + +For the purposes of this tutorial, we will install Hadoop in a single machine +running both the master and slave daemons. + +Install Apache Hadoop +********************* + +Apache Hadoop is included in the `big-data-basic` bundle. To install the +framework, enter the following command: + +.. code-block:: bash + + sudo swupd bundle-add big-data-basic + +Configure Apache Hadoop +*********************** + +#. To create the configuration directory, enter the following command: + + .. code-block:: bash + + sudo mkdir /etc/hadoop + +#. Copy the defaults from :file:`/usr/share/defaults/hadoop` to + :file:`/etc/hadoop` with the following command: + + .. code-block:: bash + + $ sudo cp /usr/share/defaults/hadoop/* /etc/hadoop + + .. note:: Since |CL| is a stateless system, you should never modify the + files under the :file:`/usr/share/defaults` directory. The software + updater will overwrite those files. + +Once all the configuration files are in :file:`/etc/hadoop`, we must edit +them to fit our needs. The `NameNode` server is the master server. It manages +the namespace of the files system and regulates the clients' access to files. +The first file we edit, :file:`/etc/hadoop/core-site.xml`, informs the Hadoop +daemon where `NameNode` is running. + +In this tutorial, our `NameNode` runs in our `localhost`. Follow these steps +to set it up correctly: + +#. Open the :file:`/etc/hadoop/core-site.xml` file using the editor of your + choice and modify the file as follows: + + .. code-block:: xml + + + + + + fs.default.name + hdfs://localhost:9000 + + + +#. Edit the :file:`/etc/hadoop/hdfs-site.xml` file. This file configures the + :abbr:`HDFS (Hadoop Distributed File System)` daemons. This configuration + includes the list of permitted and excluded data nodes and the size of + said blocks. In this example, we are setting the number of block + replication to 1 from the default of 3 as follows: + + .. code-block:: xml + :emphasize-lines: 6 + + + + + + dfs.replication + 1 + + + dfs.permission + false + + + +#. Edit the :file:`/etc/hadoop/mapred-site.xml` file. This file configures + all daemons related to `MapReduce`: `JobTracker` and `TaskTrackers`. With + `MapReduce`, Hadoop can process big amounts of data in multiple systems. In + our example, we set :abbr:`YARN (Yet Another Resource Manager)` as our + runtime framework for executing `MapReduce` jobs as follows: + + .. code-block:: xml + :emphasize-lines: 5,6 + + + + + + mapreduce.framework.name + yarn + + + +#. Edit the :file:`/etc/hadoop/yarn-site.xml` file. This file configures all + daemons related to `YARN`: `ResourceManager` and `NodeManager`. In our + example, we implement the `mapreduce_shuffle` service, which is the + default as follows: + + .. code-block:: xml + :emphasize-lines: 4,5,8,9 + + + + + yarn.nodemanager.aux-services + mapreduce_shuffle + + + yarn.nodemanager.auxservices.mapreduce.shuffle.class + org.apache.hadoop.mapred.ShuffleHandler + + + +Configure your SSH key +********************** + +#. Create a SSH key. If you already have one, skip this step. + + .. code-block:: bash + + sudo ssh-keygen -t rsa + + +#. Copy the key to your authorized keys. + + .. code-block:: bash + + sudo cat /root/.ssh/id_rsa.pub | sudo tee -a /root/.ssh/authorized_keys + +#. Log into the localhost. If no password prompt appears, you are ready to + run the Hadoop daemons. + + .. code-block:: bash + + sudo ssh localhost + +Run the Hadoop daemons +********************** + +With all the configuration files properly edited, we are ready to start the +daemons. + +When we format the `NameNode` server, it formats the meta-data related to +data nodes. Thus, all the information on the data nodes is lost and the nodes +can be reused for new data. + +#. Format the `NameNode` server with the following command: + + .. code-block:: bash + + sudo hdfs namenode -format + +#. Start the DFS in `NameNode` and `DataNodes` with the following command: + + .. code-block:: bash + + sudo start-dfs.sh + +#. The console output should be similar to: + + .. code-block:: console + + Starting namenodes on [localhost] + The authenticity of host 'localhost (::1)' can't be established. + ECDSA key fingerprint is + SHA256:97e+7TnomsS9W7GjFPjzY75HGBp+f1y6sA+ZFcOPIPU. + Are you sure you want to continue connecting (yes/no)? + + Enter `yes` to continue. + +#. Start the `YARN` daemons `ResourceManager` and `NodeManager` with the + following command: + + .. code-block:: bash + + sudo start-yarn.sh + +#. Ensure everything is running as expected with the following command: + + .. code-block:: bash + + sudo jps + +#. The console output should be similar to: + + .. code-block:: console + + 22674 DataNode + 26228 Jps + 22533 NameNode + 23046 ResourceManager + 22854 SecondaryNameNode + 23150 NodeManager + +Run the MapReduce wordcount example +*********************************** + +#. Create the input directory. + + .. code-block:: bash + + sudo hdfs dfs -mkdir -p /user/root/input + +#. Copy a file from the local file system to the HDFS. + + .. code-block:: bash + + sudo hdfs dfs -copyFromLocal local-file /user/root/input + +#. Run the `wordcount` example. + + .. code-block:: bash + + sudo hadoop jar /usr/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar wordcount input output + +#. Read output file "part-r-00000". This file contains the number of times + each word appears in the file. + + .. code-block:: bash + + sudo hdfs dfs -cat /user/root/output/part-r-00000 + +**Congratulations!** + +You successfully installed and setup a single node Hadoop cluster. +Additionally, you ran a simple wordcount example. + +Your single node Hadoop cluster is up and running! + diff --git a/source/clear-linux/tutorials/tutorials.rst b/source/clear-linux/tutorials/tutorials.rst index f0cef09f..e1a660fd 100644 --- a/source/clear-linux/tutorials/tutorials.rst +++ b/source/clear-linux/tutorials/tutorials.rst @@ -14,3 +14,4 @@ specific |CLOSIA| use cases. machine-learning/machine-learning azure multi-boot/multi-boot + hadoop