diff --git a/source/_figures/stacks/mers-fig-1.png b/source/_figures/stacks/mers-fig-1.png new file mode 100644 index 00000000..799c6fbf Binary files /dev/null and b/source/_figures/stacks/mers-fig-1.png differ diff --git a/source/_figures/stacks/mers-fig-2.png b/source/_figures/stacks/mers-fig-2.png new file mode 100644 index 00000000..cefd8b1d Binary files /dev/null and b/source/_figures/stacks/mers-fig-2.png differ diff --git a/source/guides/stacks/mers.rst b/source/guides/stacks/mers.rst new file mode 100644 index 00000000..d914b56e --- /dev/null +++ b/source/guides/stacks/mers.rst @@ -0,0 +1,407 @@ +.. _mers: + +Media Reference Stack +##################### + +The Media Reference Stack (MeRS) is a highly optimized software stack for +Intel® architecture to enable media prioritized workloads, such as transcoding and analytics. + +This guide explains how to use the pre-built |MERS| container image, build +your own |MERS| container image, and use the reference stack. + +.. contents:: + :local: + :depth: 1 + +Overview +******** + +Finding the balance between quality and performance, understanding all of the +complex standard-compliant encoders, and optimizing across the +hardware-software stack for efficiency are all engineering and time +investments for developers. + +The Media Reference Stack (MeRS) offers a highly optimized software stack for Intel Architecture to enable media prioritized workloads, such as +transcoding and analytics. |MERS| abstracts away the complexity of +integrating multiple software components and specifically tunes them for Intel platforms. |MERS| allows media and visual cloud developers to deliver experiences using a simple containerized solution. + +Prerequisites +============= + +|MERS| can run on any host system that supports Docker\*. + +The steps in this guide use |CL-ATTR| as the host system. + +- To install |CL| on a host system, see how to + :ref:`install Clear Linux* OS from the live desktop + `. + +- To install Docker* on a |CL| host system, see + the :ref:`instructions for installing Docker* `. + +.. important:: + + For optimal performance, a processor with Vector Neural Network + Instructions (VNNI) should be used. VNNI is an extension of Intel® + Advanced Vector Extensions 512 (Intel® AVX-512) and is available starting + with the 2nd generation of Intel® Xeon® Scalable Platform, providing AI + inference acceleration. + +Stack Features +============== + +The |MERS| provides a `pre-built Docker image available on DockerHub +`_, which includes +instructions on build the image from source. |MERS| is open-sourced to ensure +developers have easy access to the source code and are able to customize it. +|MERS| is built using the *clearlinux:latest* Docker image and aims to support the latest |CL| version. + +|MERS| provides the following libraries: + +.. list-table:: + :widths: auto + + * - SVT-HEVC + - Scalable Video Technology for HEVC encoding, also known as H.265 + * - SVT-AV1 + - Scalable Video Technology for AV1 encoding + * - x264 + - x264 for H.264/MPEG-4 AVC encoding + * - MKL-DNN + - `Intel® Math Kernel Library for Deep Neural Networks `_ + +Components of the |MERS| include: + +* |CL| as a base for performance and security + +* `Intel® Distribution of OpenVINO™ toolkit + `_ for inference. + +* `FFmpeg `_ with `Scalable Video Technology (SVT) + `_ plugins for encoding, decoding, and transcoding. + +* `GStreamer `_ with `Scalable Video + Technology (SVT) `_ and `OpenVINO™ toolkit + `_ plugins for + analytics. + +.. note:: + + The pre-built |MERS| container image configures :command:`FFmpeg` without + certain elements (specific encoder, decoder, muxer, etc.) that you may + require. If you require changes to :command:`FFmpeg` we suggest + `rebuilding the image <#building-mers-image>`_. + +.. note:: + + The Media Reference Stack is a collective work, and each piece of software + within the work has its own license. Please see the `MeRS Terms of Use + `_ for more details about licensing and usage of the Media Reference Stack. + + +Getting the pre-built |MERS| container image +******************************************** + +Pre-built |MERS| Docker images are available on DockerHub at +https://hub.docker.com/r/clearlinux/stacks-mers + + +To use the |MERS|: + +#. Pull the image directly from `Docker Hub + `_. + + .. code-block:: bash + + docker pull clearlinux/stacks-mers + + .. note :: + + The |MERS| docker image is large in size and will take some time to + download depending on your Internet connection. + + If you are on a network with outbound proxies, be sure to configure Docker allow access. See the `Docker service proxy + `_ and + `Docker client proxy + `_ + documentation for more details. + +#. Once you have downloaded the image, run it with: + + .. code-block:: bash + + docker run -it clearlinux/stacks-mers + + This will launch the image and drop you into a bash shell inside the + container. :command:`GStreamer` and :command:`FFmpeg` programs are + installed in the container image and accessible in the default $PATH. These programs can be used as you would normally outside of |MERS|. + + Paths to media files and video devices, such as cameras, can be shared from the host to the container with the :command:`--volume` switch + `using Docker volumes `_. + + +Building the |MERS| container image from source +*********************************************** + +If you choose to build your own MeRS container image, you can optionally add +customizations as needed. The :file:`Dockerfile` for the MeRS is available on +`GitHub `_ and +can be used for reference. + +#. The |MERS| image is part of the dockerfiles repository inside the |CL| + organization on GitHub. Clone the :file:`stacks` repository. + + .. code-block:: bash + + git clone https://github.com/intel/stacks.git + +#. Navigate to the :file:`stacks/mers/clearlinux` directory which contains + the Dockerfile for the |MERS|. + + .. code-block:: bash + + cd ./stacks/mers/clearlinux + +#. Use the :command:`docker build` command with the :file:`Dockerfile` to the + MeRS container image. + + .. code-block:: bash + + docker build --no-cache -t clearlinux/stacks-mers . + +Using the |MERS| container image +******************************** + +Below are some examples of how the |MERS| container image can be used to +process media files. + +The models and video source can be substituted from your use-case. Some +publicly licensed sample videos are available at `sample-videos repsoitory +`_ for testing. + + +Example 1: Transcoding +====================== + +This example shows how to perform transcoding with :command:`FFmpeg`. + +#. On the host system, setup a workspace for data and models: + + .. code:: bash + + mkdir ~/ffmpeg + mkdir ~/ffmpeg/input + mkdir ~/ffmpeg/output + +#. Copy a video file to :file:`~/ffmpeg/input`. + + .. code:: bash + + cp ~/ffmpeg/input + +#. Run the *clearlinux/stack-mers* docker image, allowing shared access to the + workspace on the host: + + + .. code:: bash + + docker run -it \ + -v ~/ffmpeg:/home/mers-user:ro \ + clearlinux/stacks-mers:latest + + After running the :command:`docker run` command, you enter a bash shell + inside the container. + +#. From the container shell, you can run :command:`FFmpeg` against the videos + in :file:`/home/mers-user/input` as you would normally outside of |MERS|. + + For example, to transcode raw yuv420 content to SVT-HEVC and mp4: + + .. code:: bash + + ffmpeg -f rawvideo -vcodec rawvideo -s 320x240 -r 30 -pix_fmt yuv420p -i -c:v libsvt_hevc -y + + Some more generic examples of :command:`FFmpeg` commands can be found in + the `OpenVisualCloud repository + `_ and used for reference with |MERS|. + + For more information on using :command:`FFmpeg`, refer to the `FFmpeg + documentation `_. + +Example 2: Analytics +==================== + +This example shows how to perform analytics and inferences with +:command:`GStreamer`. + +The steps here are referenced from the `gst-video-analytics Getting Started +Guide `_ except simply +substituting the *gst-video-analytics* docker image for the +*clearlinux/stacks-mers* image. + +The example below shows how to use the |MERS| container image to perform +video with object detection and attributes recognition of a video using GStreamer using pre-trained models and sample video files. + +#. On the host system, setup a workspace for data and models: + + .. code:: bash + + mkdir ~/gva + mkdir ~/gva/data + mkdir ~/gva/data/models + mkdir ~/gva/data/models/intel + mkdir ~/gva/data/models/common + mkdir ~/gva/data/video + +#. Clone the opencv/gst-video-analytics repository into the workspace: + + .. code:: bash + + git clone https://github.com/opencv/gst-video-analytics ~/gva/gst-video-analytics + cd ~/gva/gst-video-analytics + git submodule init + git submodule update + +#. Clone the Open Model Zoo repository into the workspace: + + .. code:: bash + + git clone https://github.com/opencv/open_model_zoo.git ~/gva/open_model_zoo + +#. Use the Model Downloader tool of Open Model Zoo to download ready to use + pre-trained models in IR format. + + .. note:: + + If you are on a network with outbound proxies, you will need to + configure set environment variables with the proxy server. + Refer to the documentation on :ref:`proxy` for detailed steps. + + On |CL| systems you will need the *python-extras* bundle. + Use :command:`sudo swupd bundle-add python-extras` for the downloader script to work. + + .. code:: bash + + cd ~/gva/open_model_zoo/tools/downloader + python3 downloader.py --list ~/gva/gst-video-analytics/samples/model_downloader_configs/intel_models_for_samples.LST -o ~/gva/data/models/intel + + +#. Copy a video file in h264 or mp4 format to :file:`~/gva/data/video`. Any + video with cars, pedestrians, human bodies, and/or human faces can be used. + + .. code:: bash + + git clone https://github.com/intel-iot-devkit/sample-videos.git ~/gva/data/video + + This example simply clones all the video files from the `sample-videos + repsoitory `_. + +#. From a desktop terminal, allow local access to the X host display. + + .. code:: bash + + xhost local:root + + export DATA_PATH=~/gva/data + export GVA_PATH=~/gva/gst-video-analytics + export MODELS_PATH=~/gva/data/models + export INTEL_MODELS_PATH=~/gva/data/models/intel + export VIDEO_EXAMPLES_PATH=~/gva/data/video + +#. Run the *clearlinux/stack-mers* docker image, allowing shared access to + the X server and workspace on the host: + + .. code:: bash + + docker run -it --runtime=runc --net=host \ + -v ~/.Xauthority:/root/.Xauthority \ + -v /tmp/.X11-unix:/tmp/.X11-unix \ + -e DISPLAY=$DISPLAY \ + -e HTTP_PROXY=$HTTP_PROXY \ + -e HTTPS_PROXY=$HTTPS_PROXY \ + -e http_proxy=$http_proxy \ + -e https_proxy=$https_proxy \ + -v $GVA_PATH:/home/mers-user/gst-video-analytics \ + -v $INTEL_MODELS_PATH:/home/mers-user/intel_models \ + -v $MODELS_PATH:/home/mers-user/models \ + -v $VIDEO_EXAMPLES_PATH:/home/mers-user/video-examples \ + -e MODELS_PATH=/home/mers-user/intel_models:/home/mers-user/models \ + -e VIDEO_EXAMPLES_DIR=/home/mers-user/video-examples \ + clearlinux/stacks-mers:latest + + .. note:: + + In the :command:`docker run` command above: + + - :command:`--runtime=runc` specifies the container runtime to be + *runc* for this container. It is needed for correct interaction with X + server. + + - :command:`--net=host` provides host network access to container. It is + needed for correct interaction with X server. + + - Files :file:`~/.Xauthority` and :file:`/tmp/.X11-unix` mapped to the + container are needed to ensure smooth authentication with X server. + + - :command:`-v` instances are needed to map host system directories + inside Docker container. + + - :command:`-e` instances set Docker container environment variables. + Samples need them some of them set correctly to operate. Proxy variables + are needed if host is behind firewall. + + + After running the :command:`docker run` command, it will drop you into a + bash shell inside the container. + +#. From the container shell, run a sample analytics program in + :file:`~/gva/gst-video-analytics/samples` against your video source. + + Below are sample analytics that can be run against the sample videos. + Choose one to run: + + - Samples with *face detection and classification*: + + .. code:: bash + + ./gst-video-analytics/samples/shell/face_detection_and_classification.sh $VIDEO_EXAMPLES_DIR/face-demographics-walking-and-pause.mp4 + ./gst-video-analytics/samples/shell/face_detection_and_classification.sh $VIDEO_EXAMPLES_DIR/face-demographics-walking.mp4 + ./gst-video-analytics/samples/shell/face_detection_and_classification.sh $VIDEO_EXAMPLES_DIR/head-pose-face-detection-female-and-male.mp4 + ./gst-video-analytics/samples/shell/face_detection_and_classification.sh $VIDEO_EXAMPLES_DIR/head-pose-face-detection-male.mp4 + ./gst-video-analytics/samples/shell/face_detection_and_classification.sh $VIDEO_EXAMPLES_DIR/head-pose-face-detection-female.mp4 + + When running, a video with object detection and attributes recognition + (bounding boxes around faces with recognized attributes) should be + played. + + .. figure:: /_figures/stacks/mers-fig-1.png + :scale: 60% + :align: center + :alt: Face detection with the Clear Linux* OS Media Reference Stack + + Figure 1: Screenshot of |MERS| running face detection with GSTreamer + and OpenVINO. + + - Sample with *vehicle detection*: + + .. code:: bash + + ./gst-video-analytics/samples/shell/vehicle_detection_2sources_cpu.sh $VIDEO_EXAMPLES_DIR/car-detection.mp4 + + When running, a video with object detection and attributes recognition + (bounding boxes around vehicles with recognized attributes) should be + played. + + .. figure:: /_figures/stacks/mers-fig-2.png + :scale: 60% + :align: center + :alt: Vehicle detection with the Clear Linux* OS Media Reference Stack + + Figure 2: Screenshot of |MERS| running vehicle detection with + GSTreamer and OpenVINO. + + - Sample with *FPS measurement*: + + .. code:: bash + + ./gst-video-analytics/samples/shell/console_measure_fps_cpu.sh $VIDEO_EXAMPLES_DIR/bolt-detection.mp4 diff --git a/source/substitutions.txt b/source/substitutions.txt index d780b561..c3630210 100644 --- a/source/substitutions.txt +++ b/source/substitutions.txt @@ -9,3 +9,5 @@ .. |VB| replace:: VirtualBox .. |VBM| replace:: VirtualBox Manager + +.. |MERS| replace:: :abbr:`MeRS (Media Reference Stack)` diff --git a/source/tutorials/proxy.rst b/source/tutorials/proxy.rst index 0665b63c..e1a52d24 100644 --- a/source/tutorials/proxy.rst +++ b/source/tutorials/proxy.rst @@ -1,4 +1,4 @@ -.. _tutorial-proxy: +.. _proxy: Proxy configuration ###################