mirror of
https://github.com/clearlinux/cloud-native-setup.git
synced 2026-08-26 18:36:03 +00:00
metrics: Imrove documentation
Improve and expand the documents across the metrics subsystem. Clarify and re-order some documents. Add some more details around each individual test. Note that only the 'rapid' test is currently actively used, and the other tests may need some nurturing if they are found to be useful. Signed-off-by: Graham Whaley <graham.whaley@intel.com>
This commit is contained in:
Graham Whaley
committed by
David Lyle
David Lyle
parent
e10260e99c
commit
7121418dd3
+41
-118
@@ -1,160 +1,83 @@
|
||||
* [Metric testing for scaling on Kubernetes.](#metric-testing-for-scaling-on-kubernetes)
|
||||
* [Results storage and analysis](#results-storage-and-analysis)
|
||||
* [Developers](#developers)
|
||||
* [Metrics gathering](#metrics-gathering)
|
||||
* [`collectd` statistics](#collectd-statistics)
|
||||
* [privileged statistics pods](#privileged-statistics-pods)
|
||||
* [Configuring constant 'loads'](#configuring-constant-loads)
|
||||
|
||||
# Metric testing for scaling on Kubernetes.
|
||||
|
||||
This folder contains tools to aid in measuring the scaling capabilities of
|
||||
Kubernetes clusters.
|
||||
|
||||
Primarily these tools were designed to measure scaling of large number of pods on a single node, but
|
||||
the code is structured to handle multiple nodes, and may also be useful in that scenario.
|
||||
|
||||
The tools tend to take one of two forms:
|
||||
|
||||
- Tools to take measurements
|
||||
- Tools to launch jobs and take measurements
|
||||
- Tools to analyse results
|
||||
|
||||
For more details, see individual sub-folders. A brief summary of available tools
|
||||
is below:
|
||||
|
||||
| Tool | Description |
|
||||
| Folder | Description |
|
||||
| ---- | ----------- |
|
||||
| collectd | `collectd` based statistics/metrics gathering daemonset code |
|
||||
| lib | General library helper functions for forming and launching workloads, and storing results in a uniform manner to aid later analysis |
|
||||
| lib/cpu-load* | Routines to enable CPU load generation on a cluster |
|
||||
| report | Rmarkdown based report generator, used to produce a PDF comparison report of 1 or more sets of results |
|
||||
| scaling | Tests to measure scaling, such as linear or parallel launching of pods |
|
||||
| lib/[cpu-load*](lib/cpu-load.md) | Helper functions to enable CPU load generation on a cluster whilst under test |
|
||||
| [report](report/README.md) | Rmarkdown based report generator, used to produce a PDF comparison report of one or more sets of results |
|
||||
| [scaling](scaling/README.md) | Tests to measure scaling, such as linear or parallel launching of pods |
|
||||
|
||||
## Results storage and analysis
|
||||
|
||||
The tools generate JSON formatted results files via the `lib/json.bash` functions. The `metrics_json_save()`
|
||||
function in that file has the ability to also `curl` or `socat` the JSON results to a database defined
|
||||
The tools generate JSON formatted results files via the [`lib/json.bash`](lib/json.bash) functions. The `metrics_json_save()`
|
||||
function has the ability to also `curl` or `socat` the JSON results to a database defined
|
||||
by environment variables (see the file source for details). This method has been used to store results in
|
||||
Elasticsearch and InfluxDB databases for instance, but should be adaptable to use with any REST API that accepts
|
||||
JSON input.
|
||||
|
||||
## Scaling execution
|
||||
This section describes a complete step-by-step scaling execution up to results reporting by using `scaling/k8s_scale.sh` tool which launches a series of workloads and take memory metric measurements after each launch.
|
||||
## Prerequisites
|
||||
|
||||
There are some basic pre-requisites required in order to run the test and process the results:
|
||||
|
||||
**Requirements**
|
||||
* A Kubernetes cluster up and running (tested on v1.15.3).
|
||||
* `bc` and `jq` packages.
|
||||
* Docker (only for report generation).
|
||||
|
||||
The steps to execute a run of the scaling framework are listed below, which need to be executed on the master node of a Kubernetes cluster to avoid network issues:
|
||||
1. Clone `cloud-native-setup` repository into a preferred directory and change directory up to `cloud-native-setup/metrics`:
|
||||
```sh
|
||||
$ git clone https://github.com/clearlinux/cloud-native-setup.git
|
||||
$ cd cloud-native-setup/metrics
|
||||
```
|
||||
2. Launch the execution by:
|
||||
```sh
|
||||
$ ./scaling/k8s_scale.sh
|
||||
INFO: Initialising
|
||||
command: bc: yes
|
||||
command: jq: yes
|
||||
INFO: Checking Kubernetes accessible
|
||||
INFO: 1 Kubernetes nodes in 'Ready' state found
|
||||
starting kubectl proxy
|
||||
Starting to serve on 127.0.0.1:8090
|
||||
daemonset.apps/stats created
|
||||
Waiting for daemon set "stats" rollout to finish: 0 of 1 updated pods are available...
|
||||
daemon set "stats" successfully rolled out
|
||||
INFO: Running test
|
||||
INFO: And grab some stats
|
||||
INFO: idle [98.49] free [29031100] launch [0] node [clr-30f01b5149ba4ab8b05a7ee03b6812a5] inodes_free [31103039]
|
||||
INFO: Testing replicas 1 of 20
|
||||
INFO: Content of runtime_command=:/@RUNTIMECLASS@/d
|
||||
...
|
||||
```
|
||||
The above execution might take about 4min because it launch up to 20 pods by default and takes measurements for CPU utilization, memory utilization and pod boot time, finally it will generate a `k8s-scaling.json` result file at `result` directory.
|
||||
|
||||
**Note**: to test the launch of pods concurrently, `k8s_parallel.sh` may be used. For quicker testing, `k8s_scale_rapid.sh` can be used in place of `k8s_scale.sh`. The rest of the launch instructions remain consistent other than script name.
|
||||
|
||||
**Note**: by default the scaling framework makes call to the Kubernetes API directly so, if facing connectivity issues verify that `kubelet` service's proxies and `no_proxy` environment variable are properly setup.
|
||||
|
||||
**Note**: by default the scaling framework uses default values for all its required variables, which can be checked through `scaling/k8s_scale.sh -h` and updated when launching the execution, i.e.:
|
||||
```
|
||||
$ ./scaling/k8s_scale.sh -h
|
||||
Usage: ./scaling/k8s_scale.sh [-h] [options]
|
||||
Description:
|
||||
Launch a series of workloads and take memory metric measurements after
|
||||
each launch.
|
||||
Options:
|
||||
-h, Help page.
|
||||
|
||||
Environment variables:
|
||||
Name (default)
|
||||
Description
|
||||
TEST_NAME (k8s scaling)
|
||||
Can be set to over-ride the default JSON results filename
|
||||
NUM_PODS (20)
|
||||
Number of pods to launch
|
||||
STEP (1)
|
||||
Number of pods to launch per cycle
|
||||
wait_time (30)
|
||||
Seconds to wait for pods to become ready
|
||||
delete_wait_time (600)
|
||||
Seconds to wait for all pods to be deleted
|
||||
settle_time (5)
|
||||
Seconds to wait after pods ready before taking measurements
|
||||
use_api (yes)
|
||||
specify yes or no to use the API to launch pods
|
||||
grace (30)
|
||||
specify the grace period in seconds for workload pod termination
|
||||
|
||||
$ use_api=no ./scaling/k8s_scale.sh
|
||||
```
|
||||
|
||||
The steps to generate the result report are listed below:
|
||||
|
||||
1. Having the `results/k8s-scaling.json` result file, create a subdirectory in the `results` directory with a preferred name and copy the `k8s-scaling.json` file into it, so the file distribution looks like:
|
||||
```sh
|
||||
$ tree result
|
||||
results/
|
||||
└── scaling
|
||||
└── k8s-scaling.json
|
||||
```
|
||||
|
||||
**Note**: if `k8s_scale_rapid.sh` was run instead of `k8s_scale.sh`, that the `<node_name>.tar.gz` files that appear in the results directory also need to be copied into the newly created subdirectory. And the results file is named `k8s-rapid.json` rather than `k8s-scaling.json`.
|
||||
If k8s_parallel.sh was run, the results file is named `k8s-parallel.json` rather than `k8s-scaling.json`.
|
||||
|
||||
2. Launch the report generation by:
|
||||
```sh
|
||||
./report/makereport.sh
|
||||
```
|
||||
**Note**: the first time you launch the report generation it will build a docker container to generate the reports and this process can take several minutes. Subsequent runs will be much faster.
|
||||
|
||||
The above execution will generate a `report/output` directory with the final reports, such as:
|
||||
```sh
|
||||
$ tree report/output/
|
||||
report/output/
|
||||
├── dut-1.png
|
||||
├── metrics_report.pdf
|
||||
├── scaling-1.png
|
||||
├── scaling-2.png
|
||||
├── scaling-3.png
|
||||
└── scaling-4.png
|
||||
```
|
||||
More details about result reporting can be reviewed at [`report`](./report) directory.
|
||||
|
||||
# Developers
|
||||
|
||||
This section provides some details of how the code is structured and configured. This may be of use whilst modifying
|
||||
existing or creating new tests.
|
||||
Below are some architecture and internal details of how the code is structured and configured. This will be
|
||||
helpful for improving, modifying or submitting fixes to the code base.
|
||||
|
||||
## Metrics gathering
|
||||
|
||||
Metrics can be gathered using either a daemonset deployment of privileged pods used to gather statistics directly from the nodes using a combination of `mpstat`, `free` and `df`, or a daemonset deployment based around `collectd`.
|
||||
Metrics can be gathered using either a daemonset deployment of privileged pods used to gather statistics
|
||||
directly from the nodes using a combination of `mpstat`, `free` and `df`, or a daemonset deployment based
|
||||
around `collectd`. The general recommendation is to use the `collectd` based collection if possible, as it
|
||||
is more efficient, as the system does not have to poll and wait for results, and thus executes the test
|
||||
cycle faster. The `collectd` results are collected asyncronously, and the report generator code later
|
||||
aligns the results with the pod execution in the timeline.
|
||||
|
||||
### `collectd` statistics
|
||||
|
||||
The `collected` based code can be found in the `collectd` subdirectory. It uses the `collected` configuration found in the `collectd.conf` file to gather statistics, and store the results on the nodes themselves whilst tests are running. At the end of the test, the results are copied from the nodes and stored in the results directory for later processing.
|
||||
The `collected` based code can be found in the `collectd` subdirectory. It uses the `collected` configuration
|
||||
found in the `collectd.conf` file to gather statistics, and store the results on the nodes themselves whilst
|
||||
tests are running. At the end of the test, the results are copied from the nodes and stored in the results
|
||||
directory for later processing.
|
||||
|
||||
The `collectd` statistics are only configured and gathered if the environment variable `SMF_USE_COLLECTD` is set to non-empty by the test code (that is, only enabled upon request).
|
||||
The `collectd` statistics are only configured and gathered if the environment variable `SMF_USE_COLLECTD`
|
||||
is set to non-empty by the test code (that is, it is only enabled upon request).
|
||||
|
||||
### privileged statistics pods
|
||||
|
||||
The privileged statistics pods `YAML` can be found in the `scaling/stats.yaml` file. An example of how to invoke and use this daemonset to extract statistics can be found in the `scaling/k8s_scale.sh` file.
|
||||
The privileged statistics pods `YAML` can be found in the [`scaling/stats.yaml`](scaling/stats.yaml) file.
|
||||
An example of how to invoke and use this daemonset to extract statistics can be found in the
|
||||
[`scaling/k8s_scale.sh`](scaling/k8s_scale.sh) file.
|
||||
|
||||
## Configuring constant 'loads'
|
||||
|
||||
The framework includes some tooling to assist in setting up constant pre-defined 'loads' across the cluster to aid evaluation of their impacts on the scaling metrics.
|
||||
|
||||
### CPU load generator
|
||||
|
||||
Details of how to configure a constant CPU load are detailed in the [cpu-load documentation](lib/cpu-load.md).
|
||||
The framework includes some tooling to assist in setting up constant pre-defined 'loads' across the cluster
|
||||
to aid evaluation of their impacts on the scaling metrics. See the [cpu-load documentation](lib/cpu-load.md)
|
||||
for more information.
|
||||
|
||||
@@ -0,0 +1,118 @@
|
||||
# Scaling metrics tests
|
||||
|
||||
This directory contains a number of scripts to perform a variety of system scaling tests.
|
||||
|
||||
The tests are described in their individual sections below.
|
||||
|
||||
Each test has a number of configurable options. Many of those options are common across all tests.
|
||||
Those options are detailed in their own section below.
|
||||
|
||||
> **Note:** `k8s_scale_rapid.sh` is the most complete and upto date test. It is the only test to
|
||||
> currently use the `collectd` data collection method. Other tests use a privileged container to
|
||||
> gather statistics.
|
||||
>
|
||||
> If you find one of the other tests useful, please consider updating it and the corresponding report
|
||||
> generation code to use the `collectd` method and send a Pull Request with your updates to this codebase.
|
||||
|
||||
## Global test configuration options
|
||||
|
||||
The following variables are settable for many of the tests. Check each individual tests help
|
||||
for specifics and their individual default values.
|
||||
|
||||
| Variable | Default Value | Description |
|
||||
| -------- | ------------- | ----------- |
|
||||
| TEST_NAME | test dependant | Can be set to over-ride the default JSON results filename |
|
||||
| NUM_PODS | 20 | Number of pods to launch |
|
||||
| STEP | 1 | Number of pods to launch per cycle |
|
||||
| wait_time | 30 | Seconds to wait for pods to become ready |
|
||||
| delete_wait_time | 600 | Seconds to wait for all pods to be deleted |
|
||||
| settle_time | 5 | Seconds to wait after pods ready before taking measurements |
|
||||
| use_api | yes | specify yes or no to use the JSON API to launch pods (otherwise, launch via YAML) |
|
||||
| grace | 30 | specify the grace period in seconds for workload pod termination |
|
||||
| RUNTIME | unset | specify the `RuntimeClass` to use to launch the pods |
|
||||
|
||||
## k8s_parallel.sh
|
||||
|
||||
Measures pod create and delete times whilst increasing the number of pods launched in parallel.
|
||||
|
||||
The test works by creating and destroying deployments with the required number of replicas being scaled.
|
||||
|
||||
## k8s_scale_nc.sh
|
||||
|
||||
Measures pod response time using `nc` to test network connection response. Stores results as percentile
|
||||
values. Is used to see if the response time latency and jitter is affected by scaling the number of pods.
|
||||
|
||||
## k8s_scale_net.sh
|
||||
|
||||
Measures pod response time to a `curl` HTTP get request from the K8S e2e `agnhost` image.
|
||||
Used to measure if the 'ready to respond' time scales with the number of service ports in use.
|
||||
|
||||
## k8s_scale_rapid.sh
|
||||
|
||||
Measures how pod launch and the k8s system scales whilst launching more and more pods.
|
||||
|
||||
Uses the `collectd` method to gather a number of statistics, including:
|
||||
|
||||
- cpu usage
|
||||
- memory usage
|
||||
- network connections
|
||||
- disk usage
|
||||
- ipc stats
|
||||
|
||||
## k8s_scale.sh
|
||||
|
||||
The fore-runner to `k8s_scale_rapid.sh`, using the privileged pod method to gather statistics. It is recommended
|
||||
to use `k8s_scale_rapid.sh` in preference if possible.
|
||||
|
||||
# Example
|
||||
|
||||
Below is a brief example of running the `k8s_scale_rapid.sh` test and generating a report from the results.
|
||||
|
||||
1. Run the test
|
||||
|
||||
The test will run against the default `kubectl` configured cluster.
|
||||
```sh
|
||||
$ ./scaling/k8s_scale.sh
|
||||
```
|
||||
|
||||
Results are stored in the `results` directory. The results will comprise of one `JSON` file for the test, and
|
||||
one `.tar.gz` file for each node found in the cluster.
|
||||
|
||||
> **Note:** Only the `collectd` based tests generate `.tar.gz` files. All other tests only generate a single
|
||||
> `JSON` file for each run.
|
||||
|
||||
1. Move the results files
|
||||
|
||||
In order to generate the report, the results files should be moved into an appropriately named sub-directory.
|
||||
The report generator can process and compare multiple sets of results. Each set of results should be placed
|
||||
into its own sub-directory. The below example uses the name `run1` as an example:
|
||||
|
||||
```sh
|
||||
$ cd results
|
||||
$ mkdir run1
|
||||
$ mv *.json run1
|
||||
$ mv *.tar.gz run1
|
||||
```
|
||||
|
||||
This sequence can be repeated to gather multiple test data sets. Place each data set in its own subdirectory.
|
||||
The report generator will process and compare all data set subdirectories found in the `results` directory.
|
||||
|
||||
1. Generate the report
|
||||
|
||||
The report generator in the `report` subdirectory processes the sub-directories of the `results` directory
|
||||
to produce a `PDF` report and individual `PNG` based graphs.. The report generator utilises `docker` to create
|
||||
a docker image containing all the tooling necessary.
|
||||
|
||||
```sh
|
||||
$ cd report
|
||||
$ ./makereport.sh
|
||||
...
|
||||
$ tree output
|
||||
output/
|
||||
├── dut-1.png
|
||||
├── metrics_report.pdf
|
||||
├── scaling-1.png
|
||||
├── scaling-2.png
|
||||
├── scaling-3.png
|
||||
└── scaling-4.png
|
||||
```
|
||||
Reference in New Issue
Block a user