Currently we loose collectd data from a node when scaling ends to a
system failure on the node - yet this data can be very helpful in root
causing the failure. This patch changes collectd configuration so that
the output will be continuously written to host filesystem instead of
the collectd container overlay that will be lost unless scaling
reaches graceful exit.
Signed-off-by: Antti Kervinen <antti.kervinen@intel.com>
Clean up the rest of the report R files to allow them to quit
cleanly when they find an error or missing data, so that the
final PDF report gives meaninful errors such as 'No data found',
rather than cryptic R errors.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
When there are no files to process, we tend to quit with a loud
and not helpful error. Improve that by spotting the obvious error
cases (such as no files to process for a specific test), and quit
with a nicer error/warning message that ends up in the rendered
report.
Start with the tidy_scaling test. The only clean way to quit a
fragment of Rmarkdown R looks to be to place it inside a function
so we can 'return'. Otherwise, all other forms of 'quit', quit the
whole Rmarkdown render pipeline, which is not what we want - we
want to carry on and try to process the rest of the fragments for
the rest of the tests.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Fix some indentation that had gone rogue.
Note, there are other whitespace fixes that can be done in this file,
it appears to have a mix of tabs and spaces.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Improve and expand the documents across the metrics subsystem.
Clarify and re-order some documents. Add some more details around
each individual test.
Note that only the 'rapid' test is currently actively used, and the
other tests may need some nurturing if they are found to be useful.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
This PR uses a specific version of rocker/tidyverse as the latest version
does not have the latex-xcolor package which makes impossible to create
the metrics report.
Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>
Some CNIs takes longer for its related deployments to become ready, that
is why `proc_wait_time` needs to be customized. Now `proc_wait_time` can
be set at execution time and has a default value too for time to pod
network test harness.
Signed-off-by: Morales Quispe, Marcela <marcela.morales.quispe@intel.com>
To measure the time to pod network, a deployment that uses agnhost
image is used, which get exposed as a net server and replies to curl
calls, the test measure this reply time and saves it for further reporting.
Then, only the exposed net service gets deleted.
Signed-off-by: Morales Quispe, Marcela <marcela.morales.quispe@intel.com>
Move the legends to the bottom (underneath) for the tidy scaling graphs
to make them wider on the page, and thus easier to read with more
resolution.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
The bootdata assignments were outside the 'valid file' check loop,
which meant in the case there was a data directory which did not
contain a valid scaling file, we would fail the assignment (as the
`local_bootdata` would be empty).
Fix by moving the assignments into the loop, thus only assigning when
we know we have valid data.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Most of the time we have 0 interface errors or drops, so we pin the y
scale to '1', so we don't hit 'infinity' errors. That left us with a
strange y-axis label anomoly - as the axis was automatically divided
into 5 labels, and we got for some reason the sequence '0,0,0,1,1'.
That just plain looked wrong and confusing.
Fix it by using `pretty_breaks()` for the error/drop y axis, whilst
maintaining the `comma` count for the pod count y axis.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
The pdf output by default has large page margins, which wastes a lot of
page space, and reduces our 'resolution'. Shrink the margins to a pretty
minimal 1cm to increase the graph resolution. The document itself then
does not look as 'pretty', but we can see more data visually.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Move the legends under the graphs to give more width, and thus
resolution, to the final pictures.
This works well for the collectd graphs as they are spread out
into sets of single column graphs per page.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
This is in order to avoid tty-related issues in our CI systems
which by default is not supporting tty. With this change we'll
avoid the following faling report's generation `docker run` command.
```
the input device is not a TTY
```
Signed-off-by: Obed N Munoz <obed.n.munoz@intel.com>
Under some circumstances, the pod_per_gb value would come out as <0,
and be generated without any leading 0's (such as `.14` rather than
`0.14`). This is not valid JSON, and would break the report generation
parsing in R.
Use `printf` to force a leading 0 prefix onto the value.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
If the cpu-load function is enabled, save its config settings into the
JSON results file.
This required a little bit of re-sequencing of the json library calls,
to ensure we did the init of the JSON early enough, but not more than
once.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Rejig the framework a little to unify the init/shutdown calls and code,
which allows us to add the cpu-load enable/disable ability to all the
existing metrics.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Add library code that can generate a variety of cpu loads across the
cluster. Configuration is via environment variables, documented in the
.md file.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Add the `collectd` subdir to the top level description of the code
layout, and add some developer details about how and where the stats
code lives and is configured/enabled.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
On some machines the value for hostname, that is used in naming
the csv directory, isn't always determined by collectd to be
localhost. The scaling code assumes it will always be localhost.
This patch specifies the hostname to be localhost.
Signed-off-by: David Lyle <dklyle0@gmail.com>
KIND uses the older network interface naming standard. Other
operating system images may as well. Adding support for 'eth'
network interface naming prefix.
Signed-off-by: David Lyle <dklyle0@gmail.com>
Since collectd is started before the pods are launched and
shutdown after the last pod is launched, we gather data outside
the pod launch window which can adversely influence the per pod
launch stats. This is especially true after the last pod launches
as all the pods are then deleted before collectd stops collecting
metrics.
This patch isolates the collectd data used to only
coincide with the pod launch window.
And additional change in this patch is to improve the secondary
y axis scaling. There was an ill-advised check in previously to
force the scale to be at least 1. This does not work well when
the pod number is significantly higher than say 100 (the max
possible cpu idle value).
This patch changes the scaling to be across all data to be graphed.
The special condition for interface drops and interface errors,
where the data is typically 0. We don't scale by 0.
Signed-off-by: David Lyle <dklyle0@gmail.com>
A recent change to debian apt repositories led to build errors
for the report container. The error was around stretch release
files. The container image we are based on published an update
which fixes this error. This patch updates to use :latest to
avoid errors when building.
Signed-off-by: David Lyle <dklyle0@gmail.com>
The collectd csv plugin starts a new file for each day that data is
being recorded. Currently, collectd_scaling.R only reads from the
first day's file. This leads to incomplete data being rendered in
the report charts. All the data files are collected and present,
they just need to be read.
This patch makes changes to read all the days of collectd data and add
them to the data set.
Signed-off-by: David Lyle <dklyle0@gmail.com>
Add Dockerfile for building an alpine image with collectd.
Published an image to dockerhub and reference it in collectd.yaml
Signed-off-by: David Lyle <dklyle0@gmail.com>
As part of the existing json file, there is a 'kubectl get nodes'
json dump. Instead of collecting this information each pod launch,
use the data already gathered. This data is immutable during the
test runs. This greatly improves the overall runtime of the test
script by preventing expensive kubectl execs per node per iteration.
Signed-off-by: David Lyle <dklyle0@gmail.com>
The secondary y axis have to be scaled by a factor of the primary
y axis, the default being 1. Some of the charts graph both a data
value and the number of pods. The scale for those are different
and dependent on the data collected. Logic has been added to
calculate the scale for each chart based on the data.
Signed-off-by: David Lyle <dklyle0@gmail.com>
Adding support for more than one network interface in a given node.
Also supporting more interface naming patterns.
Signed-off-by: David Lyle <dklyle0@gmail.com>
This patch adds a new script, scaling/k8s_scaling_rapid.sh, for
launching pods and collecting metrics. The goal is to two fold.
The first goal is improve the required runtime duration of scaling
to large numbers of pods. k8s_scaling.sh can take up to 29 hours
to scale to 2900 pods. The is largely due to the overhead of
collecting system utilization stats after each new pod is launched.
This new script will collect system utilization stats asynchronously.
The second goal is to make it easier to collect additional system
utilization stats by leveraging the plugins supported by collectd.
Instead of using the stats daemon set, a new daemon set
that runs collectd on each node is added. collectd configuration
is handled by collectd/collectd.conf A configmap is added to the
K8s cluster containing collectd.conf, so the user of the script
can test new configurations easily. The configmap is created and
deleted as part of the test run. The data from collectd is stored on
each node in the cluster via the csv plugin and the data is
collected to the master node at the end of the test run.
Several new pages have been added to the metrics_report.pdf
These pages cover the same metrics as k8s_scale.sh results, but are
populated with data from collectd. Additionally, network interface
results are added.
To run the report, in addition to the previous steps of creating
a new directory and copying the result json file into it, all the
new <node_name>.tar.gz files must be copied in as well.
Signed-off-by: David Lyle <dklyle0@gmail.com>
A recent change put global variable declarations in a function,
the variables no longer are globally visible. This change removes
the encapsulation in a function.
- Send n requests for each of the running pods.
- Measure response time to response.
- Report response time percentiles: 5-25-50-75-95 %.
- Refactor scaling/*.sh variables to scaling/common.bash.
Signed-off-by: Antti Kervinen <antti.kervinen@intel.com>
This script configures a node in the cluster for pod scaling
testing. This configuration is necessary for pushing beyond the
110 pod max default in Kubernetes.
In the rewrite of the report generation code, we regressed to
using the passed in requested number of pods rather than the
actual. Since we typically attempt to launch more pods than the
system is capable of, the math for the stats are done incorrectly
and misreported.
This changes the behavior back to using the correct value of the
total number of pods launched.
Use `tr` rather than `sed` for doing the line/whitespace conversions
on the JSON stream, if asked. Slightly more efficient, and we also drop
the wrapping ""'s, to allow the data to land in logstash without error.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
To aid in incorporating the results graphs and tables into other
documents, generate PNG files of the tables and plots alongside
the full PDF report, and place them in the output directory.
Achieve this by processing an html as well as a pdf targetted
output document, and grabbing the png's generated from that
operation.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Debug mode of the report generator now volume mounts the scripts
into the container (rather than just COPYing them in), to allow
interactive edit/run debug cycles. Document it.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Now the new tidy_scaling.R seems to be functional, drop the
deprecated scaling.R.
Yes, we could/should probably have just replaced scaling.R, but
that is just the way it panned out.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Do not account for 'noschedule' nodes in the stats calculations,
as they don't host the pods we are measuring.
Show the boot times in seconds, rather than milliseconds.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
We now use `bc` for some of the stats calculations. Ensure it
is installed, or we silently generate bad field data.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Antti Kervinen