Since collectd is started before the pods are launched and
shutdown after the last pod is launched, we gather data outside
the pod launch window which can adversely influence the per pod
launch stats. This is especially true after the last pod launches
as all the pods are then deleted before collectd stops collecting
metrics.
This patch isolates the collectd data used to only
coincide with the pod launch window.
And additional change in this patch is to improve the secondary
y axis scaling. There was an ill-advised check in previously to
force the scale to be at least 1. This does not work well when
the pod number is significantly higher than say 100 (the max
possible cpu idle value).
This patch changes the scaling to be across all data to be graphed.
The special condition for interface drops and interface errors,
where the data is typically 0. We don't scale by 0.
Signed-off-by: David Lyle <dklyle0@gmail.com>
A recent change to debian apt repositories led to build errors
for the report container. The error was around stretch release
files. The container image we are based on published an update
which fixes this error. This patch updates to use :latest to
avoid errors when building.
Signed-off-by: David Lyle <dklyle0@gmail.com>
The collectd csv plugin starts a new file for each day that data is
being recorded. Currently, collectd_scaling.R only reads from the
first day's file. This leads to incomplete data being rendered in
the report charts. All the data files are collected and present,
they just need to be read.
This patch makes changes to read all the days of collectd data and add
them to the data set.
Signed-off-by: David Lyle <dklyle0@gmail.com>
As part of the existing json file, there is a 'kubectl get nodes'
json dump. Instead of collecting this information each pod launch,
use the data already gathered. This data is immutable during the
test runs. This greatly improves the overall runtime of the test
script by preventing expensive kubectl execs per node per iteration.
Signed-off-by: David Lyle <dklyle0@gmail.com>
The secondary y axis have to be scaled by a factor of the primary
y axis, the default being 1. Some of the charts graph both a data
value and the number of pods. The scale for those are different
and dependent on the data collected. Logic has been added to
calculate the scale for each chart based on the data.
Signed-off-by: David Lyle <dklyle0@gmail.com>
Adding support for more than one network interface in a given node.
Also supporting more interface naming patterns.
Signed-off-by: David Lyle <dklyle0@gmail.com>
This patch adds a new script, scaling/k8s_scaling_rapid.sh, for
launching pods and collecting metrics. The goal is to two fold.
The first goal is improve the required runtime duration of scaling
to large numbers of pods. k8s_scaling.sh can take up to 29 hours
to scale to 2900 pods. The is largely due to the overhead of
collecting system utilization stats after each new pod is launched.
This new script will collect system utilization stats asynchronously.
The second goal is to make it easier to collect additional system
utilization stats by leveraging the plugins supported by collectd.
Instead of using the stats daemon set, a new daemon set
that runs collectd on each node is added. collectd configuration
is handled by collectd/collectd.conf A configmap is added to the
K8s cluster containing collectd.conf, so the user of the script
can test new configurations easily. The configmap is created and
deleted as part of the test run. The data from collectd is stored on
each node in the cluster via the csv plugin and the data is
collected to the master node at the end of the test run.
Several new pages have been added to the metrics_report.pdf
These pages cover the same metrics as k8s_scale.sh results, but are
populated with data from collectd. Additionally, network interface
results are added.
To run the report, in addition to the previous steps of creating
a new directory and copying the result json file into it, all the
new <node_name>.tar.gz files must be copied in as well.
Signed-off-by: David Lyle <dklyle0@gmail.com>
In the rewrite of the report generation code, we regressed to
using the passed in requested number of pods rather than the
actual. Since we typically attempt to launch more pods than the
system is capable of, the math for the stats are done incorrectly
and misreported.
This changes the behavior back to using the correct value of the
total number of pods launched.
To aid in incorporating the results graphs and tables into other
documents, generate PNG files of the tables and plots alongside
the full PDF report, and place them in the output directory.
Achieve this by processing an html as well as a pdf targetted
output document, and grabbing the png's generated from that
operation.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Debug mode of the report generator now volume mounts the scripts
into the container (rather than just COPYing them in), to allow
interactive edit/run debug cycles. Document it.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Now the new tidy_scaling.R seems to be functional, drop the
deprecated scaling.R.
Yes, we could/should probably have just replaced scaling.R, but
that is just the way it panned out.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Do not account for 'noschedule' nodes in the stats calculations,
as they don't host the pods we are measuring.
Show the boot times in seconds, rather than milliseconds.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Due to the nature of the shape of the JSON data import, the scaling
R had some hard wired constructs bound to the data shape. This limited
our extending the JSON data.
recode to be more 'R tidyverse' style.
Also add in a launch time stats table, and split the graphing
across 3 pages to de-cramp them (particularly when rendering >1
test run).
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
When in debug mode, mount the scripts dir into the container to
allow 'live' editing and reloading into R. This makes the dev cycle
much faster.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Add a table to the end of the report showing useful details about
the systems that were under test - version numbers, resources etc.
Fixes: #145
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
The time we measure is the time to launch and delete the whole
deployment, not the 'pod'. Change the graph titles to be more
accurate.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Rather than duplicate the same aes() over all the parts of the ggplot(),
add the aes to the base ggplot, and then default to using that for
all the other additions.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
geom_smooth can generate a mass of warnings when applied to a small
dataset. Use the same test (>20 samples) we do for 'do we draw dots',
but in the inverse - so, we either draw dots (small set) or a smoothed
overlayed line (bit sets).
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Now the data has the ability to 'step' (that is, increment
the pod count by >1 between samples), use that n_pod value
for the x-axis, rather than a +1 count, so the x-axis makes
a lot more sense.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
The boot 'points' code was accidentally updating the cpu plot
(copy and paste error). Correct it to update the correct chart.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Test how long it takes to launch, and delete, a deployment
running up 'n' pods. Used to ascertain if parallel launch times
are linear with number of pods or not.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
We were carrying a bunch of legacy docker based functions
that we do not use. Drop them. We can always add them back
in later if we find we do need some.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Add the first basic report generator code, utilising R, Rmarkdown
and pandoc to do the data processing and pdf report generation.
Fixes: #121
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
David Lyle