As part of the existing json file, there is a 'kubectl get nodes'
json dump. Instead of collecting this information each pod launch,
use the data already gathered. This data is immutable during the
test runs. This greatly improves the overall runtime of the test
script by preventing expensive kubectl execs per node per iteration.
Signed-off-by: David Lyle <dklyle0@gmail.com>
This patch adds a new script, scaling/k8s_scaling_rapid.sh, for
launching pods and collecting metrics. The goal is to two fold.
The first goal is improve the required runtime duration of scaling
to large numbers of pods. k8s_scaling.sh can take up to 29 hours
to scale to 2900 pods. The is largely due to the overhead of
collecting system utilization stats after each new pod is launched.
This new script will collect system utilization stats asynchronously.
The second goal is to make it easier to collect additional system
utilization stats by leveraging the plugins supported by collectd.
Instead of using the stats daemon set, a new daemon set
that runs collectd on each node is added. collectd configuration
is handled by collectd/collectd.conf A configmap is added to the
K8s cluster containing collectd.conf, so the user of the script
can test new configurations easily. The configmap is created and
deleted as part of the test run. The data from collectd is stored on
each node in the cluster via the csv plugin and the data is
collected to the master node at the end of the test run.
Several new pages have been added to the metrics_report.pdf
These pages cover the same metrics as k8s_scale.sh results, but are
populated with data from collectd. Additionally, network interface
results are added.
To run the report, in addition to the previous steps of creating
a new directory and copying the result json file into it, all the
new <node_name>.tar.gz files must be copied in as well.
Signed-off-by: David Lyle <dklyle0@gmail.com>
A recent change put global variable declarations in a function,
the variables no longer are globally visible. This change removes
the encapsulation in a function.
- Send n requests for each of the running pods.
- Measure response time to response.
- Report response time percentiles: 5-25-50-75-95 %.
- Refactor scaling/*.sh variables to scaling/common.bash.
Signed-off-by: Antti Kervinen <antti.kervinen@intel.com>
We now use `bc` for some of the stats calculations. Ensure it
is installed, or we silently generate bad field data.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Keep track of memory consumed on the nodes, log that, and
generate an overall cluster value for 'pods-per-Gb'.
Generate this data here, whilst we have the data to hand, as
it can get much more difficult to calculate this information
the further you get down the monitoring stream (such as the
farther you get into an ELK stack).
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Add invididual date entries to each statistics sample, to
suppliment the top level date entry. This helps order the
samples if we are flattening the data into say Elasticsearch.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
When 'grace' handling was added, it got missed from the parallel
script (which shares the bb.*.in files). Add it in to fix the test.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
We were storing some JSON data (such as NUM_PODS) as strings.
This would default their type into the Elastic DB as strings,
and not ints. May as well store ints as ints, so make that so.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Test how long it takes to launch, and delete, a deployment
running up 'n' pods. Used to ascertain if parallel launch times
are linear with number of pods or not.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
We are very likely to re-use the k8s api code in many/all of the
tests, so move common code out to a common libray file.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
If we have $RUNTIME set in the env, then we set up the pod
RuntimeClass, otherwise, just use the default node runtimes.
Clean up some of the un-used library code around docker/RUNTIME
as well whilst we are there.
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
We need to check when we are launching our first instance of a
deployment when using the k8s API directly, as we have to modify
the URL for the first instance. Rather than check for '1', let's
check for 'STEP', as technically you can run the loop starting at
a non-1 index (particularly useful to run with NUM_PODS==STEP, as
you then get a 'single shot' test of 'n' containers).
Signed-off-by: Graham Whaley <graham.whaley@intel.com>
Morales Quispe, Marcela