From 2a95b2552656b63fdcfa4f4a45244beb0eef7919 Mon Sep 17 00:00:00 2001 From: Beth Dean Date: Tue, 5 Mar 2019 07:40:47 -0800 Subject: [PATCH] dlrs tutorial update (#410) * adding pytorch section * updated doc Minor nits and corrections * scripts: Code Block rst parser Signed-off-by: John Andersen * limited to bash code blocks and added description to the header * added ability to create a list of code blocks to parse * adding pytorch section * updating PyTorch image link & name * updating docker image link and adding PyTorch benchmarking link * Sphinx syntax corrections; reorganizes contents/toctree; line-edits. Signed-off-by: Michael Vincerra * updating link to PyTorch benchmarks --- source/clear-linux/tutorials/dlrs.rst | 138 +++++++++++++++++++------- 1 file changed, 102 insertions(+), 36 deletions(-) diff --git a/source/clear-linux/tutorials/dlrs.rst b/source/clear-linux/tutorials/dlrs.rst index 113f3c8e..fdf894e3 100644 --- a/source/clear-linux/tutorials/dlrs.rst +++ b/source/clear-linux/tutorials/dlrs.rst @@ -4,38 +4,45 @@ Deep Learning Reference Stack ############################# This tutorial shows you how to run benchmarking workloads in |CL-ATTR| using -TensorFlow\* and Kubeflow with the Deep Learning Reference Stack. +TensorFlow\* or PyTorch\* with the Deep Learning Reference Stack. We also +cover using Kubeflow for multi-node benchmarking. -The Deep Learning Reference Stack is available in two versions. -The first is `Eigen`_, which includes `TensorFlow`_ optimized for Intel® -architecture. The second is `Intel MKL-DNN`_, which includes the TensorFlow -framework optimized using Intel® Math Kernel Library for Deep Neural -Networks (Intel® MKL-DNN) primitives. - -.. contents:: :local: +.. contents:: + :local: :depth: 1 +The Deep Learning Reference Stack is available in four versions: + +* `Eigen`_, which includes `TensorFlow`_ optimized for Intel® architecture. +* `Intel MKL-DNN`_, which includes the TensorFlow framework optimized using + Intel® Math Kernel Library for Deep Neural Networks (Intel® MKL-DNN) primitives. +* `PyTorch with OpenBLAS`_, which includes PyTorch with OpenBlas. +* `PyTorch with Intel MKL-DNN`_, which includes PyTorch optimized using + Intel® Math Kernel Library (Intel® MKL)and Intel MKL-DNN. + Release notes -============= +************* -View current `release notes`_ for the Deep Learning Reference Stack. - -View current `benchmark results`_ for the Deep Learning Reference Stack. +* View current `release notes`_ for the Deep Learning Reference Stack. +* View current `TensorFlow benchmark results`_ for the Deep Learning + Reference Stack with TensorFlow. +* View current `PyTorch benchmark results`_ for the Deep Learning Reference + Stack with PyTorch. .. note:: Performance test numbers in the Deep Learning Reference Stack were obtained using `runc` as the runtime. Prerequisites -============= +************* -* |CL| installed on host system. If not installed, :ref:`bare-metal-install` +* |CL| installed on host system. :ref:`Install ` * `containers-basic` bundle * `cloud-native-basic` bundle In |CL|, `containers-basic` provides Docker\*, which is required for -TensorFlow benchmarking. Use the :command:`swupd` utility to check if -`containers-basic` and `cloud-native-basic` are present: +TensorFlow and PyTorch benchmarking. Use the :command:`swupd` utility to +check if `containers-basic` and `cloud-native-basic` are present: .. code-block:: bash @@ -47,7 +54,7 @@ If you need to install the `containers-basic` or `cloud-native-basic`, enter: sudo swupd bundle-add containers-basic cloud-native-basic -To ensure that kubernetes is correctly installed and configured, +To ensure that Kubernetes is correctly installed and configured, follow :ref:`kubernetes`. We have validated these steps against the following software package @@ -59,7 +66,7 @@ versions: * Go 1.11.12 TensorFlow single and multi-node benchmarks -============================================ +******************************************* This section describes running the `TensorFlow benchmarks`_ in single node. For multi-node testing, replicate these steps for each node. These steps @@ -73,13 +80,14 @@ TensorFlow. .. code-block:: bash - docker run --name --rm -i -t bash + docker run --name --rm -i -t bash .. note:: - You will enter the following commands in the running container. - - Replace with the you specified above. + Launching the docker image with the :command:`-i` argument will put + you into interactive mode within the container. You will enter the + following commands in the running container. #. Clone the benchmark repository: @@ -98,28 +106,66 @@ TensorFlow. You can replace the model with one of your choice supported by the TensorFlow benchmarks. +PyTorch single and multi-node benchmarks +**************************************** + +This section describes running the `PyTorch benchmarks`_ for Caffe2 in +single node. We will be looking at validating the Caffe2 APIs with the +official benchmarks, but the same process applies for other cases. + +#. Download either the `PyTorch with OpenBLAS`_ or the `PyTorch with Intel + MKL-DNN`_ docker image + from `Docker Hub`_. + +#. Run the image with Docker: + + .. code-block:: bash + + docker run --name --rm -i -t bash + + .. note:: + + Launching the docker image with the :command:`-i` argument will put + you into interactive mode within the container. You will enter the + following commands in the running container. + +#. Clone the benchmark repository: + + .. code-block:: bash + + git clone https://github.com/pytorch/pytorch.git + +#. Next, execute the benchmark script to run the benchmark. + + .. code-block:: bash + + cd pytorch/caffe2/python + python convnet_benchmarks.py --batch_size 32 \ + --cpu \ + --model AlexNet + Kubeflow multi-node benchmarks -============================== +****************************** The benchmark workload will run in a Kubernetes cluster. We will use `Kubeflow`_ for the Machine Learning workload deployment on three nodes. Kubernetes setup -**************** +================ Follow the instructions in the :ref:`kubernetes` tutorial to get set up on |CL|. The kubernetes community also has `instructions for creating a cluster`_. Kubernetes networking -********************* +===================== We used `flannel`_ as the network provider for these tests. If you are comfortable with another network layer, refer to the Kubernetes `networking documentation`_ for setup. Images -****** +====== We need to add `launcher.py` to our docker image to include the Deep Learning Reference Stack and put the benchmarks repo in the correct @@ -139,12 +185,14 @@ kubeflow. We are working to create these images as part of our release cycle. ksonnet\* -********* +========= -Kubeflow uses ksonnet* to manage deployments, so we need to install that before setting up Kubeflow. +Kubeflow uses ksonnet\* to manage deployments, so we need to install that +before setting up Kubeflow. -Since Clear Linux version 27550, the ksonnet was added to the bundle cloud-native-basic. But if using -old versions (not recommended), please manually install the ksonnet as below. +Since Clear Linux version 27550, the ksonnet was added to the bundle +cloud-native-basic. But if using old versions (not recommended), please +manually install the ksonnet as below. On |CL|, follow these steps: @@ -161,7 +209,7 @@ After the ksonnet installation is complete, ensure that binary `ks` is accessible across the environment. Kubeflow -******** +======== Once you have Kubernetes running on your nodes, you can setup `Kubeflow`_ by following these instructions from their `quick start guide`_. @@ -194,7 +242,7 @@ Now you have all the required kubeflow packages, and you can deploy the primary This creates the CustomResourceDefinition(CRD) endpoint to launch a TFJob. Run a TFJob -=========== +*********** #. Select this link for the `ksonnet registries for deploying TFJobs`_. @@ -229,29 +277,47 @@ Run a TFJob This will replicate and deploy three test setups in your Kubernetes cluster. Results of Running this Tutorial -================================ +******************************** + You need to parse the logs of the Kubernetes pod to get the performance numbers. The pods will still be around post completion and will be in ‘Completed’ state. You can get the logs from any of the pods to inspect the -benchmark results. More information about `Kubernetes logging`_ is available from the Kubernetes community. +benchmark results. More information about `Kubernetes logging`_ is available +from the Kubernetes community. .. _TensorFlow: https://www.tensorflow.org/ + .. _Kubeflow: https://www.kubeflow.org/ + .. _Docker Hub: https://hub.docker.com/ + .. _TensorFlow benchmarks: https://www.tensorflow.org/guide/performance/benchmarks + +.. _PyTorch benchmarks: https://github.com/pytorch/pytorch/blob/master/caffe2/python/convnet_benchmarks.py + .. _instructions for creating a cluster: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/ + .. _flannel: https://github.com/coreos/flannel + .. _networking documentation: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network + .. _quick start guide: https://www.kubeflow.org/docs/started/getting-started/ .. _Eigen: https://hub.docker.com/r/clearlinux/stacks-dlrs-oss/ + .. _Intel MKL-DNN: https://hub.docker.com/r/clearlinux/stacks-dlrs-mkl/ +.. _PyTorch with OpenBLAS: https://hub.docker.com/r/clearlinux/stacks-pytorch-oss + +.. _PyTorch with Intel MKL-DNN: https://hub.docker.com/r/clearlinux/stacks-pytorch-mkl + .. _release notes: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dlrs .. _ksonnet registries for deploying TFJobs: https://github.com/clearlinux/dockerfiles/tree/master/stacks/dlrs/kubeflow/dlrs-tfjob - .. _Kubernetes logging: https://kubernetes.io/docs/concepts/cluster-administration/logging/ -.. _benchmark results: https://clearlinux.org/stacks/deep-learning-reference-stack +.. _TensorFlow benchmark results: https://clearlinux.org/stacks/deep-learning-reference-stack + +.. _PyTorch benchmark results: https://clearlinux.org/stacks/deep-learning-reference-stack-pytorch +