Commit Graph

121 Commits

Author SHA1 Message Date
Graham Whaley 615e7d6e6f metrics: report: parallel: change title to 'deployment time'
The time we measure is the time to launch and delete the whole
deployment, not the 'pod'. Change the graph titles to be more
accurate.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-08-14 10:58:06 -06:00
Graham Whaley 1c14965e6c metrics: parallel: dedup use of aes()
Rather than duplicate the same aes() over all the parts of the ggplot(),
add the aes to the base ggplot, and then default to using that for
all the other additions.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-08-14 10:58:06 -06:00
Graham Whaley 645110832b metrics: parallel: Only smooth with big data sets
geom_smooth can generate a mass of warnings when applied to a small
dataset. Use the same test (>20 samples) we do for 'do we draw dots',
but in the inverse - so, we either draw dots (small set) or a smoothed
overlayed line (bit sets).

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-08-14 10:58:06 -06:00
Graham Whaley b3f40569b2 metrics: parallel: take into account 'stepping'
Now the data has the ability to 'step' (that is, increment
the pod count by >1 between samples), use that n_pod value
for the x-axis, rather than a +1 count, so the x-axis makes
a lot more sense.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-08-14 10:58:06 -06:00
Graham Whaley aeb6ebbab7 metrics: json: default RUNTIME to 'default'
If a RUNTIME is not set, then save it as 'default' in the JSON
rather than an empty string, as the emptry string does not render
well later in any graphing.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-08-14 08:20:36 -06:00
Graham Whaley 40cca1a8d4 metrics: scaling: whitespace fix
four-space to tab.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-08-14 08:20:36 -06:00
Graham Whaley 7b98d82e11 metrics: scaling: add datestamps to stats samples
Add invididual date entries to each statistics sample, to
suppliment the top level date entry. This helps order the
samples if we are flattening the data into say Elasticsearch.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-08-14 08:20:36 -06:00
Graham Whaley e4ef431b0f metrics: parallel scaling: add 'grace' handling
When 'grace' handling was added, it got missed from the parallel
script (which shares the bb.*.in files). Add it in to fix the test.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-08-14 08:20:36 -06:00
David Lyle 15ab28deaf fixing indexing, start with 0 2019-08-14 10:04:24 +01:00
NitinAtIntel 85fe594c20 Formatting cleanup 2019-08-12 18:29:11 -07:00
NitinAtIntel 0d71305d65 Splitting out detailed vagrant instructions 2019-08-12 18:29:11 -07:00
NitinAtIntel 5c4de21fe7 Adding detailed steps to get vagrant working with libvirt 2019-08-12 18:29:11 -07:00
Justin Scott 10307fde45 Use directories instead of partitions for rook
Signed-off-by: Justin Scott <justin.a.scott@intel.com>
2019-08-12 14:19:33 -07:00
root a29f8b7b71 Fixed vagrant issue with proxy
proxy_ip_list variable had scope issues, used as global variable
initially then as a local variable. Changed all to global.

Signed-off-by: root <root@csterret-desk1.amr.corp.intel.com>
2019-08-08 18:19:20 -07:00
Justin Scott 9f4ffb2e39 Kustomize all the things
This commit replaces our static manifests with Kustomize
versions and adds scripting to apply them.

Signed-off-by: Justin Scott <justin.a.scott@intel.com>
2019-08-08 16:31:18 -07:00
David Lyle 67ee1cb2d4 tuning node dataset charting, removing skip_points 2019-08-08 17:02:19 +01:00
David Lyle c88d9a61c3 clean up 2019-08-07 09:53:36 +01:00
David Lyle d72621ae2e show which node the pod launched on, store pod name 2019-08-07 09:53:36 +01:00
David Lyle af7b595e4a excluding NoSchedule nodes from workload calculations 2019-08-07 09:53:36 +01:00
David Lyle ec9dbe1f0e changing node shapes in charts rather than color 2019-08-02 17:39:44 +01:00
David Lyle 45c9a7821a format fixes 2019-08-02 17:39:44 +01:00
David Lyle d8a8d77d1b Adding charting support for multinode 2019-08-02 17:39:44 +01:00
David Lyle 510dd83b09 adding multinode utilization tracking 2019-08-02 17:39:44 +01:00
Graham Whaley 42359e1f8e metrics: scaling: store json int results as non-strings
We were storing some JSON data (such as NUM_PODS) as strings.
This would default their type into the Elastic DB as strings,
and not ints. May as well store ints as ints, so make that so.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-07-30 13:13:35 -06:00
Graham Whaley 5c9a5df264 metrics: report: Fix scaling points for boot chart
The boot 'points' code was accidentally updating the cpu plot
(copy and paste error). Correct it to update the correct chart.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-07-30 13:13:35 -06:00
Graham Whaley e73beb18e1 metrics: parallel: Test deployment parallel pod scaling
Test how long it takes to launch, and delete, a deployment
running up 'n' pods. Used to ascertain if parallel launch times
are linear with number of pods or not.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-07-30 13:13:35 -06:00
Graham Whaley 59a1f36e37 metrics: common: remove un-used docker-esque code
We were carrying a bunch of legacy docker based functions
that we do not use. Drop them. We can always add them back
in later if we find we do need some.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-07-30 13:13:35 -06:00
Graham Whaley fa94c2ce90 metrics: api: move some api code out to a common file
We are very likely to re-use the k8s api code in many/all of the
tests, so move common code out to a common libray file.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-07-30 13:13:35 -06:00
Graham Whaley 293a2b32d2 metrics: json: add k8s info sections
Capture some of the k8s setup information in the JSON results
files. Add sections for:
 - kubectl version
 - kubectl get nodes

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-07-30 13:13:35 -06:00
Graham Whaley bead9c7dfc metrics: use RUNTIME to determine RuntimeClass
If we have $RUNTIME set in the env, then we set up the pod
RuntimeClass, otherwise, just use the default node runtimes.

Clean up some of the un-used library code around docker/RUNTIME
as well whilst we are there.

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-07-30 13:13:35 -06:00
Graham Whaley 6e238d7cd9 metrics: scaling: change check for 'first deploy' to non-1 indexed
We need to check when we are launching our first instance of a
deployment when using the k8s API directly, as we have to modify
the URL for the first instance. Rather than check for '1', let's
check for 'STEP', as technically you can run the loop starting at
a non-1 index (particularly useful to run with NUM_PODS==STEP, as
you then get a 'single shot' test of 'n' containers).

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-07-30 11:51:06 -07:00
David Lyle c4b6824f07 Fixing proxy process management 2019-07-26 16:14:55 +01:00
David Lyle d7d40598b4 Add option to use API to launch 2019-07-26 16:14:55 +01:00
Graham Whaley 351540049d metrics: Add the start of the local report generator
Add the first basic report generator code, utilising R, Rmarkdown
and pandoc to do the data processing and pdf report generation.

Fixes: #121

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
2019-07-24 10:52:13 -06:00
David Lyle 0f39518f8d Moving to milliseconds rather than seconds 2019-07-24 17:46:21 +01:00
David Lyle c67566e41e Initial commit of metrics for scaling 2019-07-24 17:46:21 +01:00
Ganesh Maharaj Mahalingam 39a3f46ec8 Add swupd update back
Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
v1.6
2019-07-22 14:00:48 -07:00
NitinAtIntel 0e27cf06a3 As scripts will be distributed with bundle in clear, user can control their clear version if using scripts from git. 2019-07-22 10:35:37 -07:00
NitinAtIntel 41e9b05cc0 Enabling exit on error at top of script. Switched method to obtain k8s source with check for existing repo. Switched version check to look at client version as we want to test using same client k8s version as came bundled 2019-07-22 10:35:37 -07:00
NitinAtIntel 7106afcd9f Adding a script to execute upstream k8s e2e tests on the cloud native stack 2019-07-22 10:35:37 -07:00
Ganesh Maharaj Mahalingam 859da50c94 Move to containerd as the default manager
This setup scripts have been using cri-o all this while as the pod
controller/manager system. Recent past cri-o has had some issues with
kata-deploy (A restart of the service will not be able to re-connect
with the existing pods and restart all of them including kata-deploy,
which will hit an endless loop). Moving to containerd as default for
now.

Firecracker cannot be used with a released version of containerd as
there is no default block based snapshotter available today. If you wish
to use cri-o make sure you set `RUNNER=crio` in your environment prior
to using the script.

Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
2019-07-19 17:32:01 -07:00
Manohar Castelino 0ffacbd904 Versioning: Update clearlinux version to 30270
Clearlinux release 30270 is now a known good version compatible
with kubernetes.

RuntimeName:  cri-o
RuntimeVersion:  1.14.4
RuntimeApiVersion:  v1alpha1

Kubernetes v1.15.0

runc version 1.0.0-rc5
spec: 1.0.0

systemd 242 (242)
+PAM +AUDIT -SELINUX +IMA -APPARMOR -SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 -IDN -PCRE2 default-hierarchy=legacy

Kernel: 4.19.57-60.lts2018

Signed-off-by: Manohar Castelino <manohar.r.castelino@intel.com>
2019-07-10 17:09:45 -07:00
Saikrishna Edupuganti bbeb447a8a Pinning to working box version and clear version
Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com>
2019-06-25 10:04:49 -07:00
Saikrishna Edupuganti 8ab98c53db Configure canal to ignore rpf instead of host
Modifying rp_filter setting on host seems to be causing a routing issue
when connecting to a pod from remote node. Until we figure out the best
way to solve the issue, we are configuring canal to ignore default
setting of 2

Fixes: #102

Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com>
2019-06-20 17:03:48 -07:00
Ganesh Maharaj Mahalingam 0e280f40fa cri-o update fixes
ClearLinux now has cri-o version 1.14.1 which allows multiple plugin
locations and also creates /opt/cni/bin by default. We no longer need
the hacks for them

Fixes: #82
Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
v1.5
2019-06-04 11:53:36 -07:00
Ganesh Maharaj Mahalingam d31d78c193 calico-node fails to come up cause of all.rp_filter
Currently net.ipv4.conf.all.rp_filter is set to 2 in Clear and
calico-node fails to come up unless that value is either 0 or 1.

Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
2019-05-30 14:17:44 -07:00
Jose Carlos Venegas Munoz 04487a2cfe reset_stack: stop containerd and crio
- If stop CRI service if is running
- Restart the CRI service only if enabled
- Do not enable crio on reset_stack.sh this should be only part of the
setup.

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
2019-05-29 18:20:33 -07:00
Jose Carlos Venegas Munoz e67630b0bd create_stack: quote variables to avoid code expands
General fixes make my vim mark less warnings from shellcheck

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
2019-05-24 11:40:49 -07:00
Jose Carlos Venegas Munoz e08c92f6d5 stack: do not provide CRI socket
kubeadm autodetects the socket path based on defaults from well known
CRI servers.

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
2019-05-24 11:37:08 -07:00
Jose Carlos Venegas Munoz d00a5ea9d8 README: update flavor setup information.
Add docuementation to use ./clr-k8s-examples/reset_stack.sh help

Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
2019-05-23 20:38:45 -07:00