fix canal download URL

Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Remove static cpu from kubeadm.yaml
2026-08-18 21:16:16 +00:00 · 2022-05-18 10:57:13 -06:00 · 2022-04-04 14:31:15 -07:00 · 2022-03-29 16:17:07 -07:00 · 2022-03-04 12:00:27 -07:00 · 2021-05-18 09:08:55 -06:00
40 changed files with 1971 additions and 1450 deletions
@@ -0,0 +1,2 @@
+resources:
+  - canal/canal.yaml
@@ -0,0 +1,4 @@
+resources:
+  - canal/canal.yaml
+
+
@@ -0,0 +1,2 @@
+resources:
+        cilium/cilium.yaml
@@ -0,0 +1,5 @@
+ipam:
+  mode: "cluster-pool"
+  operator:
+    clusterPoolIPv4PodCIDR: "10.244.0.0/16"
+    clusterPoolIPv4MaskSize: 24
@@ -0,0 +1,2 @@
+resources:
+        - flannel/Documentation/kube-flannel.yml
@@ -0,0 +1,2 @@
+resources:
+        - flannel/Documentation/kube-flannel.yml
@@ -0,0 +1,8 @@
+resources:
+  - components.yaml
+patchesJson6902:
+  - target:
+      version: v1
+      kind: Deployment
+      name: metrics-server
+    path: patch_metricstls.yaml
@@ -0,0 +1,3 @@
+- op: add
+  path: "/spec/template/spec/containers/0/args/-"
+  value: --kubelet-insecure-tls
@@ -0,0 +1,76 @@
+resources:
+  - kube-prometheus/manifests/kubeStateMetrics-serviceAccount.yaml
+  - kube-prometheus/manifests/blackboxExporter-clusterRole.yaml
+  - kube-prometheus/manifests/nodeExporter-serviceAccount.yaml
+  - kube-prometheus/manifests/prometheus-prometheusRule.yaml
+  - kube-prometheus/manifests/kubernetesControlPlane-serviceMonitorKubeScheduler.yaml
+  - kube-prometheus/manifests/prometheus-serviceMonitor.yaml
+  - kube-prometheus/manifests/grafana-dashboardSources.yaml
+  - kube-prometheus/manifests/kubeStateMetrics-prometheusRule.yaml
+  - kube-prometheus/manifests/kubePrometheus-prometheusRule.yaml
+  - kube-prometheus/manifests/prometheus-clusterRole.yaml
+  - kube-prometheus/manifests/blackboxExporter-serviceMonitor.yaml
+  - kube-prometheus/manifests/prometheus-roleSpecificNamespaces.yaml
+  - kube-prometheus/manifests/alertmanager-service.yaml
+  - kube-prometheus/manifests/prometheusAdapter-serviceMonitor.yaml
+  - kube-prometheus/manifests/nodeExporter-prometheusRule.yaml
+  - kube-prometheus/manifests/nodeExporter-service.yaml
+  - kube-prometheus/manifests/prometheus-roleConfig.yaml
+  - kube-prometheus/manifests/kubeStateMetrics-clusterRole.yaml
+  - kube-prometheus/manifests/prometheusOperator-deployment.yaml
+  - kube-prometheus/manifests/prometheusOperator-serviceMonitor.yaml
+  - kube-prometheus/manifests/prometheusAdapter-deployment.yaml
+  - kube-prometheus/manifests/kubernetesControlPlane-serviceMonitorApiserver.yaml
+  - kube-prometheus/manifests/prometheusAdapter-configMap.yaml
+  - kube-prometheus/manifests/kubernetesControlPlane-prometheusRule.yaml
+  - kube-prometheus/manifests/kubeStateMetrics-deployment.yaml
+  - kube-prometheus/manifests/blackboxExporter-configuration.yaml
+  - kube-prometheus/manifests/kubeStateMetrics-clusterRoleBinding.yaml
+  - kube-prometheus/manifests/blackboxExporter-serviceAccount.yaml
+  - kube-prometheus/manifests/grafana-dashboardDefinitions.yaml
+  - kube-prometheus/manifests/prometheusOperator-service.yaml
+  - kube-prometheus/manifests/grafana-service.yaml
+  - kube-prometheus/manifests/prometheus-prometheus.yaml
+  - kube-prometheus/manifests/kubernetesControlPlane-serviceMonitorKubeControllerManager.yaml
+  - kube-prometheus/manifests/alertmanager-alertmanager.yaml
+  - kube-prometheus/manifests/kubernetesControlPlane-serviceMonitorKubelet.yaml
+  - kube-prometheus/manifests/grafana-dashboardDatasources.yaml
+  - kube-prometheus/manifests/kubernetesControlPlane-serviceMonitorCoreDNS.yaml
+  - kube-prometheus/manifests/alertmanager-serviceMonitor.yaml
+  - kube-prometheus/manifests/grafana-deployment.yaml
+  - kube-prometheus/manifests/grafana-serviceAccount.yaml
+  - kube-prometheus/manifests/alertmanager-serviceAccount.yaml
+  - kube-prometheus/manifests/prometheusAdapter-clusterRoleAggregatedMetricsReader.yaml
+  - kube-prometheus/manifests/prometheusOperator-prometheusRule.yaml
+  - kube-prometheus/manifests/alertmanager-podDisruptionBudget.yaml
+  - kube-prometheus/manifests/prometheus-serviceAccount.yaml
+  - kube-prometheus/manifests/prometheus-service.yaml
+  - kube-prometheus/manifests/prometheusAdapter-clusterRoleServerResources.yaml
+  - kube-prometheus/manifests/prometheusAdapter-clusterRoleBinding.yaml
+  - kube-prometheus/manifests/prometheus-roleBindingConfig.yaml
+  - kube-prometheus/manifests/nodeExporter-daemonset.yaml
+  - kube-prometheus/manifests/prometheus-roleBindingSpecificNamespaces.yaml
+  - kube-prometheus/manifests/nodeExporter-clusterRoleBinding.yaml
+  - kube-prometheus/manifests/prometheusOperator-serviceAccount.yaml
+  - kube-prometheus/manifests/prometheusOperator-clusterRoleBinding.yaml
+  - kube-prometheus/manifests/kubeStateMetrics-serviceMonitor.yaml
+  - kube-prometheus/manifests/prometheusAdapter-roleBindingAuthReader.yaml
+  - kube-prometheus/manifests/prometheusAdapter-clusterRoleBindingDelegator.yaml
+  - kube-prometheus/manifests/prometheusAdapter-serviceAccount.yaml
+  - kube-prometheus/manifests/blackboxExporter-deployment.yaml
+  - kube-prometheus/manifests/alertmanager-prometheusRule.yaml
+  - kube-prometheus/manifests/prometheus-clusterRoleBinding.yaml
+  - kube-prometheus/manifests/prometheusAdapter-clusterRole.yaml
+  - kube-prometheus/manifests/grafana-serviceMonitor.yaml
+  - kube-prometheus/manifests/nodeExporter-clusterRole.yaml
+  - kube-prometheus/manifests/prometheusAdapter-service.yaml
+  - kube-prometheus/manifests/prometheus-podDisruptionBudget.yaml
+  - kube-prometheus/manifests/blackboxExporter-service.yaml
+  - kube-prometheus/manifests/nodeExporter-serviceMonitor.yaml
+  - kube-prometheus/manifests/blackboxExporter-clusterRoleBinding.yaml
+  - kube-prometheus/manifests/alertmanager-secret.yaml
+  - kube-prometheus/manifests/prometheusAdapter-apiService.yaml
+  - kube-prometheus/manifests/prometheusOperator-clusterRole.yaml
+  - kube-prometheus/manifests/grafana-config.yaml
+  - kube-prometheus/manifests/prometheusAdapter-podDisruptionBudget.yaml
+  - kube-prometheus/manifests/kubeStateMetrics-service.yaml
@@ -0,0 +1,3 @@
+resources:
+  - rook/deploy/examples/cluster.yaml
+  - rook/deploy/examples/csi/rbd/storageclass.yaml
@@ -0,0 +1,8 @@
+resources:
+  - rook/deploy/examples/cluster.yaml
+  - rook/deploy/examples/csi/rbd/storageclass.yaml
+
+patchesStrategicMerge:
+  # patches rook to use 'directories' instead of partitions.
+  # comment out to use partitions
+  - patch_cephcluster.yaml
@@ -0,0 +1,19 @@
+---
+apiVersion: ceph.rook.io/v1
+kind: CephCluster
+metadata:
+  name: rook-ceph
+  namespace: rook-ceph
+spec:
+  mon:
+    allowMultiplePerNode: true
+---
+apiVersion: ceph.rook.io/v1
+kind: CephBlockPool
+metadata:
+  name: replicapool
+  namespace: rook-ceph
+spec:
+  replicated:
+    requireSafeReplicaSize: false
+    size: 1
@@ -1,20 +1,28 @@
 # Build multus plugin
-FROM golang:1.10 AS multus
-RUN git clone -q --depth 1 https://github.com/intel/multus-cni.git /go/src/github.com/intel/multus-cni
-WORKDIR /go/src/github.com/intel/multus-cni
-RUN ./build
+FROM busybox AS multus
+ARG MULTUS_VER=3.4.2
+RUN wget -O multus.tgz https://github.com/intel/multus-cni/releases/download/v${MULTUS_VER}/multus-cni_${MULTUS_VER}_linux_amd64.tar.gz
+RUN tar xvzf multus.tgz --strip-components=1 -C /bin

 # Build sriov plugin
-FROM golang:1.10 AS sriov-cni
-RUN git clone -q -b dev/k8s-deviceid-model https://github.com/Intel-Corp/sriov-cni.git /go/src/github.com/intel-corp/sriov-cni
-WORKDIR /go/src/github.com/intel-corp/sriov-cni
-RUN ./build
+FROM golang AS sriov-cni
+ARG SRIOV_CNI_VER=2.3
+RUN wget -qO sriov-cni.tgz https://github.com/intel/sriov-cni/archive/v${SRIOV_CNI_VER}.tar.gz
+RUN mkdir -p sriov-cni && \
+    tar xzf sriov-cni.tgz --strip-components=1 -C sriov-cni && \
+    cd sriov-cni && \
+    make && \
+    cp build/sriov /bin

 # Build sriov device plugin
-FROM golang:1.10 AS sriov-dp
-RUN git clone -q https://github.com/intel/sriov-network-device-plugin.git /go/src/github.com/intel/sriov-network-device-plugin
-WORKDIR /go/src/github.com/intel/sriov-network-device-plugin
-RUN make
+FROM golang AS sriov-dp
+ARG SRIOV_DP_VER=3.2
+RUN wget -qO sriov-dp.tgz https://github.com/intel/sriov-network-device-plugin/archive/v${SRIOV_DP_VER}.tar.gz
+RUN mkdir -p sriov-dp && \
+    tar xzf sriov-dp.tgz --strip-components=1 -C sriov-dp && \
+    cd sriov-dp && \
+    make && \
+    cp build/sriovdp /bin

 # Build vfioveth plugin
 FROM busybox as vfioveth
@@ -25,9 +33,9 @@ RUN chmod +x /bin/vfioveth /bin/jq
 # Final image
 FROM centos/systemd
 WORKDIR /tmp/cni/bin
-COPY --from=multus /go/src/github.com/intel/multus-cni/bin/multus .
-COPY --from=sriov-cni /go/src/github.com/intel-corp/sriov-cni/bin/sriov .
+COPY --from=multus /bin/multus-cni .
+COPY --from=sriov-cni /bin/sriov .
 COPY --from=vfioveth /bin/vfioveth .
 COPY --from=vfioveth /bin/jq .
 WORKDIR /usr/bin
-COPY --from=sriov-dp /go/src/github.com/intel/sriov-network-device-plugin/build/sriovdp .
+COPY --from=sriov-dp /bin/sriovdp .
@@ -9,11 +9,10 @@ directories on the host with the necessary binaries and configuration files.

 ### Customization

-The device plugin will register the SR-IOV enabled devices on the host, specified as
-`rootDevices` in [sriov-conf.yaml](sriov-conf.yaml). Helper [systemd unit](systemd/sriov.service)
-file is provided, which enables SR-IOV for the above `rootDevices`
-
-> NOTE: This assumes homogenous nodes in the cluster
+The device plugin will register the SR-IOV enabled devices on the host, specified with
+`selectors` in [sriov-conf.yaml](sriov-conf.yaml). Helper [systemd unit](systemd/sriov.service)
+file is provided, which enables SR-IOV for the above devices. More config options
+are listed [here](https://github.com/intel/sriov-network-device-plugin#configurations).

 ### Pre-req (SR-IOV only)

@@ -50,9 +50,9 @@ delete_pair_ns() {
 case $CNI_COMMAND in
 ADD)
 	res=$(ipam)
-	ip=$(echo $res | jq -r '.ip4.ip')
+	ip=$(echo $res | jq -r '.ips[0].address')
 	add_pair_ns $ip
-	echo '{"cniVersion":"0.2.0"}' | jq -c --arg ip $ip '.ip4.ip = $ip' >&3
+	echo '{"cniVersion":"0.3.1"}' | jq -c --arg ip $ip '.ips[0].address = $ip' >&3
 	;;
 DEL)
 	set +o errexit
@@ -2,25 +2,34 @@
 apiVersion: apiextensions.k8s.io/v1
 kind: CustomResourceDefinition
 metadata:
-  # name must match the spec fields below, and be in the form: <plural>.<group>
  name: network-attachment-definitions.k8s.cni.cncf.io
 spec:
-  # group name to use for REST API: /apis/<group>/<version>
  group: k8s.cni.cncf.io
-  # version name to use for REST API: /apis/<group>/<version>
-  version: v1
-  # either Namespaced or Cluster
  scope: Namespaced
  names:
-    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
    plural: network-attachment-definitions
-    # singular name to be used as an alias on the CLI and for display
    singular: network-attachment-definition
-    # kind is normally the CamelCased singular type. Your resource manifests use this.
    kind: NetworkAttachmentDefinition
-    # shortNames allow shorter string to match your resource on the CLI
    shortNames:
    - net-attach-def
+  versions:
+    - name: v1
+      served: true
+      storage: true
+      schema:
+        openAPIV3Schema:
+          description: 'NetworkAttachmentDefinition is a CRD schema specified by the Network Plumbing
+            Working Group to express the intent for attaching pods to one or more logical or physical
+            networks. More information available at: https://github.com/k8snetworkplumbingwg/multi-net-spec'
+          type: object
+          properties:
+            spec:
+              description: 'NetworkAttachmentDefinition spec defines the desired state of a network attachment'
+              type: object
+              properties:
+                config:
+                  description: 'NetworkAttachmentDefinition config is a JSON-formatted CNI configuration'
+                  type: string
 ---
 apiVersion: v1
 kind: ServiceAccount
@@ -40,17 +49,43 @@ type: kubernetes.io/service-account-token
 kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
-  name: multus-pod-networks-lister
+  name: multus-pod-updater
 rules:
- apiGroups: [""]
-  resources: ["pods"]
-  verbs: ["get"]
- apiGroups: [""]
-  resources: ["pods/status"]
-  verbs: ["update"]
- apiGroups: ["k8s.cni.cncf.io"]
-  resources: ["*"]
-  verbs: ["get"]
+  - apiGroups: ["k8s.cni.cncf.io"]
+    resources:
+      - '*'
+    verbs:
+      - '*'
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - pods/status
+    verbs:
+      - get
+      - update
+  - apiGroups:
+      - ""
+      - events.k8s.io
+    resources:
+      - events
+    verbs:
+      - create
+      - patch
+      - update
+---
+kind: ClusterRoleBinding
+apiVersion: rbac.authorization.k8s.io/v1
+metadata:
+  name: multus-rb
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: multus-pod-updater
+subjects:
+- kind: ServiceAccount
+  name: multus-sa
+  namespace: kube-system
 ---
 kind: ConfigMap
 apiVersion: v1
@@ -94,8 +129,9 @@ data:
    MASTER_PLUGIN_JSON="$(cat /host/etc/cni/net.d/$MASTER_PLUGIN)"
    cat > /host/etc/cni/net.d/00-multus.conf <<EOF
    {
+      "cniVersion": "0.3.1",
      "name": "multus-cni-network",
-      "type": "multus",
+      "type": "multus-cni",
      "logFile": "/var/log/multus.log",
      "logLevel": "debug",
      "kubeconfig": "/etc/cni/net.d/multus-kubeconfig",
@@ -105,19 +141,6 @@ data:
    }
    EOF
 ---
-kind: ClusterRoleBinding
-apiVersion: rbac.authorization.k8s.io/v1
-metadata:
-  name: multus-rb
-roleRef:
-  apiGroup: rbac.authorization.k8s.io
-  kind: ClusterRole
-  name: multus-pod-networks-lister
-subjects:
- kind: ServiceAccount
-  name: multus-sa
-  namespace: kube-system
---
 apiVersion: apps/v1
 kind: DaemonSet
 metadata:
@@ -134,14 +157,14 @@ spec:
    spec:
      initContainers:
      - name: multus
-        image: krsna1729/multus-sriov:k8s-1.13
+        image: krsna1729/multus-sriov:3.4.2-2.3-3.2
        command: [ "bash", "-c" ]
        args:
-        - cp /tmp/cni/bin/{multus,sriov,vfioveth,jq} /host/opt/cni/bin/;
+        - cp /tmp/cni/bin/{multus-cni,sriov,vfioveth,jq} /host/opt/cni/bin/;
          /tmp/multus/install-multus-conf.sh;
          /tmp/multus/install-certs.sh;
          echo "Restarting crio kubelet";
-          systemctl restart crio; # Needed when crio manages ns lifecycle
+          systemctl restart crio;
          systemctl restart kubelet;
        volumeMounts:
        - name: usr-bin
@@ -160,7 +183,7 @@ spec:
          mountPath: /run/systemd
      containers:
      - name: sriovdp
-        image: krsna1729/multus-sriov:k8s-1.13
+        image: krsna1729/multus-sriov:3.4.2-2.3-3.2
        command: [ "sh", "-c" ]
        args:
        - /usr/bin/sriovdp --logtostderr -v 10;
@@ -209,4 +232,3 @@ spec:
          path: /var/lib/kubelet/device-plugins/
      hostNetwork: true
      hostPID: true
-
@@ -11,15 +11,15 @@ data:
      [
        {
          "resourceName": "sriov_netdevice",
-          "rootDevices": ["07:00.0"],
-          "sriovMode": true,
-          "deviceType": "netdevice"
+          "selectors": {
+                  "drivers": ["i40evf", "iavf"]
+          }
        },
        {
          "resourceName": "sriov_vfio",
-          "rootDevices": ["07:00.1"],
-          "sriovMode": true,
-          "deviceType": "vfio"
+          "selectors": {
+                  "drivers": ["vfio-pci"]
+          }
        }
      ]
    }
@@ -0,0 +1,100 @@
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: dpdk-1711
+  annotations:
+      k8s.v1.cni.cncf.io/networks: sriov-net-dpdk
+spec:
+  restartPolicy: Never
+  containers:
+  - name: dpdk
+    image: krsna1729/dpdk:17.11
+    stdin: true
+    tty: true
+    command: [ "/bin/bash", "-c"]
+    args:
+    - ls -l /dev/vfio;
+      testpmd --no-huge -m 2048 -- --stats-period=10 --nb-port=1 --port-topology=chained --auto-start --total-num-mbufs=2048 --forward-mode=macswap;
+    securityContext:
+      capabilities:
+        add:
+        - IPC_LOCK
+    resources:
+      limits:
+        intel.com/sriov_vfio: '1'
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: dpdk-1811
+  annotations:
+      k8s.v1.cni.cncf.io/networks: sriov-net-dpdk
+spec:
+  restartPolicy: Never
+  containers:
+  - name: dpdk
+    image: krsna1729/dpdk:18.11
+    stdin: true
+    tty: true
+    command: [ "/bin/bash", "-c"]
+    args:
+    - ls -l /dev/vfio;
+      testpmd --no-huge -m 2048 -- --stats-period=10 --nb-port=1 --port-topology=chained --auto-start --total-num-mbufs=2048 --forward-mode=macswap;
+    securityContext:
+      capabilities:
+        add:
+        - IPC_LOCK
+    resources:
+      limits:
+        intel.com/sriov_vfio: '1'
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: dpdk-1911
+  annotations:
+      k8s.v1.cni.cncf.io/networks: sriov-net-dpdk
+spec:
+  restartPolicy: Never
+  containers:
+  - name: dpdk
+    image: krsna1729/dpdk:19.11
+    stdin: true
+    tty: true
+    command: [ "/bin/bash", "-c"]
+    args:
+    - ls -l /dev/vfio;
+      testpmd --no-huge -m 2048 -- --stats-period=10 --nb-port=1 --port-topology=chained --auto-start --total-num-mbufs=2048 --forward-mode=macswap;
+    securityContext:
+      capabilities:
+        add:
+        - IPC_LOCK
+    resources:
+      limits:
+        intel.com/sriov_vfio: '1'
+---
+apiVersion: v1
+kind: Pod
+metadata:
+  name: dpdk-2002
+  annotations:
+      k8s.v1.cni.cncf.io/networks: sriov-net-dpdk
+spec:
+  restartPolicy: Never
+  containers:
+  - name: dpdk
+    image: krsna1729/dpdk:20.02
+    stdin: true
+    tty: true
+    command: [ "/bin/bash", "-c"]
+    args:
+    - ls -l /dev/vfio;
+      testpmd --no-huge -m 2048 -- --stats-period=10 --nb-port=1 --port-topology=chained --auto-start --total-num-mbufs=2048 --forward-mode=macswap;
+    securityContext:
+      capabilities:
+        add:
+        - IPC_LOCK
+    resources:
+      limits:
+        intel.com/sriov_vfio: '1'
@@ -0,0 +1,17 @@
+FROM ubuntu:bionic as ubuntu-build
+RUN apt-get update && \
+    apt-get -y install \
+        build-essential \
+        git \
+        libnuma-dev
+
+ARG DPDK_VER='master'
+ENV DPDK_DIR='/dpdk'
+ENV RTE_TARGET='x86_64-native-linuxapp-gcc'
+RUN git clone -b $DPDK_VER -q --depth 1 http://dpdk.org/git/dpdk-stable $DPDK_DIR 2>&1
+RUN cd ${DPDK_DIR} && \
+    sed -ri 's,(IGB_UIO=).*,\1n,' config/common_linux* && \
+    sed -ri 's,(KNI_KMOD=).*,\1n,' config/common_linux* && \
+    make config T=x86_64-native-linuxapp-gcc && \
+    make -j $CPUS
+ENV PATH="$PATH:$DPDK_DIR/build/app/"
@@ -3,6 +3,11 @@
 ## Prerequisite
 This setup currently will work with Kubernetes 1.14 & above. Any version of Kubernetes before that might work, but is not guaranteed.

+## QUICK NOTE
+The version of Kubernetes* was bumped from 1.17.7 to 1.19.4 in Clear Linux* OS release 34090. The [guide](https://docs.01.org/clearlinux/latest/guides/clear/k8s-migration.html) and the Clear Linux OS bundle k8s-migration were created to help facilitate migration of a cluster from 1.17.x to the latest 1.19.x .
+
+The new Clear Linux OS bundle k8s-migration was added in Clear Linux* OS release 34270. Please follow the guide for an upgrade.
+
 ## Sample multi-node vagrant setup

 To be able to test this tool, you can create a 3-node vagrant setup. In this tutorial, we will talk about using [libvirt](https://github.com/vagrant-libvirt/vagrant-libvirt), but you can use any hypervisor that you are familiar with.
@@ -63,6 +68,28 @@ devicemapper storage. This should not be used for production.

 > NOTE: This step is done automatically if using vagrant.

+### For HA, setup the load balancer node
+
+Ideally, the load balancer node will be a separate node. However, one of the
+master nodes can also serve as the load balancer for the cluster. [HAProxy](https://www.haproxy.org/)
+is used in these instructions.
+
+```bash
+sudo swupd bundle-add haproxy
+sudo systemctl enable haproxy
+```
+
+Edit the master IP addresses and load balancer address and ports in [`haproxy.cfg.example`](haproxy.cfg.example)
+to match the IPs for the new cluster. If using a master node for the load balancer
+make sure that the `frontend bind` port is different than the Kubernetes API port, 6443.
+If using a separate machine for load balancing, the port can be 6443 if desired.
+
+```bash
+sudo mkdir -p /etc/haproxy
+sudo cp haproxy.cfg.example /etc/haproxy/haproxy.cfg
+sudo systemctl start haproxy
+```
+
 ## Bring up the master

 Run [`create_stack.sh`](create_stack.sh) on the master node. This sets up the
@@ -75,8 +102,9 @@ more information.

 > NOTE: Before running [`create_stack.sh`](create_stack.sh) script, make sure to export
 the necessary environment variables if needed to be changed. By default it will use
-`CLRK8S_CNI` to be canal, and `CLRK8S_RUNNER` to be crio. Cilium is tested only in the 
-Vagrant.
+`CLRK8S_CNI` to be canal, and `CLRK8S_RUNNER` to be crio. Cilium is tested only in the
+Vagrant. If creating an HA cluster, make sure to specify `LOAD_BALANCER_IP` and
+`LOAD_BALANCER_PORT`.

 ```bash
 # default shows help
@@ -86,6 +114,13 @@ Vagrant.
 In order to enable running greater than 110 pods per node, set the environment
 variable `HIGH_POD_COUNT` to any non-empty value.

+If creating an HA cluster, join the other master nodes to the cluster.
+
+```bash
+kubeadm join <load-balancer-ip>:<load-balancer-port> --token <token> --discovery-token-ca-cert-hash <hash> \
+    --control-plane --certificate-key <certificate-key> --cri-socket=/run/crio/crio.sock
+```
+
 ## Join Workers to the cluster

 ```bash
@@ -94,6 +129,9 @@ kubeadm join <master-ip>:<master-port> --token <token> --discovery-token-ca-cert

 Note: Remember to append `--cri-socket=/run/crio/crio.sock` to the join command generated by the master.

+If creating an HA cluster, join the other worker nodes to the cluster. The same way,
+but replacing the `<master-ip>:<master-port>` with `<load-balancer-ip>:<load-balancer-port>`.
+
 On workers just use the join command that the master spits out. There nothing
 else you need to run on the worker. All the other Kubernetes customizations are pushed
 in from master via the values setup in the `kubeadm.yaml` file.
@@ -16,7 +16,7 @@ $box = "AntonioMeireles/ClearLinux"
 $box_ver = (ENV['CLEAR_VBOX_VER'])
 File.exists?("/usr/share/qemu/OVMF.fd") ? $loader = "/usr/share/qemu/OVMF.fd" : $loader = File.join(File.dirname(__FILE__), "OVMF.fd")
 $vm_name_prefix = "clr"
-$base_ip = IPAddr.new("192.52.100.10")
+$base_ip = IPAddr.new("10.10.100.10")
 $hosts = {}
 $proxy_ip_list = "192.168.121.0/24"
 $driveletters = ('a'..'z').to_a
@@ -76,6 +76,7 @@ Vagrant.configure("2") do |config|
        lv.loader = $loader
        lv.cpus = $cpus
        lv.memory = $memory
+        lv.machine_virtual_size = 40
        (1..$disks).each do |d|
          lv.storage :file, :device => "hd#{$driveletters[d]}", :path => "disk-#{$disk_prefix}-#{vm_name}-#{d}.disk", :size => $disk_size, :type => "raw"
        end
@@ -14,21 +14,24 @@ SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
 : ${MASTER_IP:=}
 : ${CERT_SANS:=}
 HIGH_POD_COUNT=${HIGH_POD_COUNT:-""}
+LOAD_BALANCER_IP=${LOAD_BALANCER_IP:-""}
+LOAD_BALANCER_PORT="${LOAD_BALANCER_PORT:-6444}"

 # versions
-CANAL_VER="${CLRK8S_CANAL_VER:-v3.10}"
-CILIUM_VER="${CLRK8S_CILIUM_VER:-v1.6.4}"
-FLANNEL_VER="${CLRK8S_FLANNEL_VER:-960b3243b9a7faccdfe7b3c09097105e68030ea7}"
+CANAL_VER="${CLRK8S_CANAL_VER:-v3.22}"
+CILIUM_VER="${CLRK8S_CILIUM_VER:-v1.9.13}"
+FLANNEL_VER="${CLRK8S_FLANNEL_VER:-v0.16.3}"
+CILIUM_VAL_OVERRIDE=""
 K8S_VER="${CLRK8S_K8S_VER:-}"
-KATA_VER="${CLRK8S_KATA_VER:-1.9.1-kernel-config}"
-ROOK_VER="${CLRK8S_ROOK_VER:-v1.2.6}"
-METRICS_VER="${CLRK8S_METRICS_VER:-v0.3.6}"
+KATA_VER="${CLRK8S_KATA_VER:-2.3.3}"
+ROOK_VER="${CLRK8S_ROOK_VER:-v1.8.6}"
+METRICS_VER="${CLRK8S_METRICS_VER:-v0.6.1}"
 DASHBOARD_VER="${CLRK8S_DASHBOARD_VER:-v2.0.0-beta2}"
 INGRES_VER="${CLRK8S_INGRES_VER:-nginx-0.26.1}"
 EFK_VER="${CLRK8S_EFK_VER:-v1.15.1}"
 METALLB_VER="${CLRK8S_METALLB_VER:-v0.8.3}"
 NPD_VER="${CLRK8S_NPD_VER:-v0.6.6}"
-PROMETHEUS_VER="${CLRK8S_PROMETHEUS_VER:-f458e85e5d7675f7bc253072e1b4c8892b51af0f}"
+PROMETHEUS_VER="${CLRK8S_PROMETHEUS_VER:-v0.10.0}"
 CNI=${CLRK8S_CNI:-"canal"}
 if [[ -z "${RUNNER+x}" ]]; then RUNNER="${CLRK8S_RUNNER:-"crio"}"; fi

@@ -90,7 +93,13 @@ function cluster_init() {
 		echo "/var/lib/etcd exists! skipping init."
 		return
 	fi
-	sudo -E kubeadm init --config=./kubeadm.yaml
+
+	if [[ -n "${LOAD_BALANCER_IP}" ]]; then
+		sed -i "s/ClusterConfiguration/ClusterConfiguration\ncontrolPlaneEndpoint: ${LOAD_BALANCER_IP}:${LOAD_BALANCER_PORT}/g" ./kubeadm.yaml
+	fi
+	# upload-certs will automatically upload certificates that should be shared
+	# across control-plane nodes in HA clusters. It is harmless in non-HA cases.
+	sudo -E kubeadm init --upload-certs --config=./kubeadm.yaml

 	rm -rf "${HOME}/.kube"
 	mkdir -p "${HOME}/.kube"
@@ -115,11 +124,13 @@ function cluster_init() {

 function kata() {
 	KATA_VER=${1:-$KATA_VER}
-	KATA_URL="https://github.com/kata-containers/packaging.git"
+	KATA_URL="https://github.com/kata-containers/kata-containers.git"
 	KATA_DIR="8-kata"
 	get_repo "${KATA_URL}" "${KATA_DIR}/overlays/${KATA_VER}"
-	set_repo_version "${KATA_VER}" "${KATA_DIR}/overlays/${KATA_VER}/packaging"
-	kubectl apply -k "${KATA_DIR}/overlays/${KATA_VER}"
+	set_repo_version "${KATA_VER}" "${KATA_DIR}/overlays/${KATA_VER}/kata-containers"
+	kubectl apply -f "${KATA_DIR}/overlays/${KATA_VER}/kata-containers/tools/packaging/kata-deploy/kata-rbac/base/kata-rbac.yaml"
+	kubectl apply -f "${KATA_DIR}/overlays/${KATA_VER}/kata-containers/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml"
+	kubectl apply -f "${KATA_DIR}/overlays/${KATA_VER}/kata-containers/tools/packaging/kata-deploy/runtimeclasses/kata-runtimeClasses.yaml"

 }

@@ -128,7 +139,7 @@ function cni() {
 	canal)
 		# note version is not semver
 		CANAL_VER=${1:-$CANAL_VER}
-		CANAL_URL="https://docs.projectcalico.org/${CANAL_VER}/manifests"
+		CANAL_URL="https://projectcalico.docs.tigera.io/archive/${CANAL_VER}/manifests"
 		if [[ "$CANAL_VER" == "v3.3" ]]; then
 			CANAL_URL="https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/canal"
 		fi
@@ -136,7 +147,7 @@ function cni() {

 		# canal manifests are not kept in repo but in docs site so use curl
 		mkdir -p "${CANAL_DIR}/overlays/${CANAL_VER}/canal"
-		curl -o "${CANAL_DIR}/overlays/${CANAL_VER}/canal/canal.yaml" "$CANAL_URL/canal.yaml"
+		curl -L -o "${CANAL_DIR}/overlays/${CANAL_VER}/canal/canal.yaml" "$CANAL_URL/canal.yaml"
 		if [[ "$CANAL_VER" == "v3.3" ]]; then
 			curl -o "${CANAL_DIR}/overlays/${CANAL_VER}/canal/rbac.yaml" "$CANAL_URL/rbac.yaml"
 		fi
@@ -145,7 +156,7 @@ function cni() {
 		;;
 	flannel)
 		FLANNEL_VER=${1:-$FLANNEL_VER}
-		FLANNEL_URL="https://github.com/coreos/flannel"
+		FLANNEL_URL="https://github.com/flannel-io/flannel"
 		FLANNEL_DIR="0-flannel"

 		get_repo "${FLANNEL_URL}" "${FLANNEL_DIR}/overlays/${FLANNEL_VER}"
@@ -153,13 +164,17 @@ function cni() {
 		kubectl apply -k "${FLANNEL_DIR}/overlays/${FLANNEL_VER}"
 		;;
 	cilium)
+		local podsubnet=$(grep -Po 'podSubnet:\ \K[^*]*' ${SCRIPT_DIR}/kubeadm.yaml)
 		CILIUM_VER=${1:-$CILIUM_VER}
 		CILIUM_URL="https://github.com/cilium/cilium.git"
 		CILIUM_DIR="0-cilium"

 		get_repo "${CILIUM_URL}" "${CILIUM_DIR}/overlays/${CILIUM_VER}"
 		set_repo_version "${CILIUM_VER}" "${CILIUM_DIR}/overlays/${CILIUM_VER}/cilium/"
-		helm template "${CILIUM_DIR}/overlays/${CILIUM_VER}/cilium/install/kubernetes/cilium" --namespace kube-system --set global.containerRuntime.integration="$RUNNER" | kubectl apply -f -
+		if [ -f "${CILIUM_DIR}/overlays/${CILIUM_VER}/values.yaml" ]; then
+			CILIUM_VAL_OVERRIDE="--values ${CILIUM_DIR}/overlays/${CILIUM_VER}/values.yaml"
+		fi
+		helm template "${CILIUM_DIR}/overlays/${CILIUM_VER}/cilium/install/kubernetes/cilium" --namespace kube-system --set containerRuntime.integration="$RUNNER" --set hubble.enabled=false --set ipam.operator.clusterPoolIPv4PodCIDR="${podsubnet}" | kubectl apply -f -
 		;;
 	*)
 		echo"Unknown cni $CNI"
@@ -170,21 +185,19 @@ function cni() {

 function metrics() {
 	METRICS_VER="${1:-$METRICS_VER}"
-	METRICS_URL="https://github.com/kubernetes-incubator/metrics-server.git"
+	METRICS_URL="https://github.com/kubernetes-sigs/metrics-server/releases/download/${METRICS_VER}/components.yaml"
 	METRICS_DIR="1-core-metrics"
-	get_repo "${METRICS_URL}" "${METRICS_DIR}/overlays/${METRICS_VER}"
-	set_repo_version "${METRICS_VER}" "${METRICS_DIR}/overlays/${METRICS_VER}/metrics-server"
-	kubectl apply -k "${METRICS_DIR}/overlays/${METRICS_VER}"
+
+	curl -L ${METRICS_URL} --output - >${METRICS_DIR}/overlays/${METRICS_VER}/components.yaml
+	kubectl apply -k "${METRICS_DIR}/overlays/${METRICS_VER}/"

 }
 function wait_on_pvc() {
 	# create and destroy pvc until successful
 	while [[ $(kubectl get pvc test-pv-claim --no-headers | grep Bound -c) -ne 1 ]]; do
 		sleep 30
-		kubectl delete pvc test-pv-claim
-		create_pvc
-		sleep 10
 	done
+	kubectl delete pvc test-pv-claim
 }
 function create_pvc() {
 	kubectl apply -f - <<HERE
@@ -209,9 +222,19 @@ function storage() {
 	ROOK_URL="https://github.com/rook/rook.git"
 	ROOK_DIR=7-rook

+  # This function might be called standalone, so good to check the mode we are in.
+	if [ "$(kubectl get nodes --no-headers | wc -l)" -eq 1 ]; then
+		mode="standalone"
+	fi
+
 	# get and apply rook
 	get_repo "${ROOK_URL}" "${ROOK_DIR}/overlays/${ROOK_VER}/${mode}"
 	set_repo_version "${ROOK_VER}" "${ROOK_DIR}/overlays/${ROOK_VER}/${mode}/rook"
+	kubectl apply -f ${ROOK_DIR}/overlays/${ROOK_VER}/${mode}/rook/deploy/examples/crds.yaml -f ${ROOK_DIR}/overlays/${ROOK_VER}/${mode}/rook/deploy/examples/common.yaml -f ${ROOK_DIR}/overlays/${ROOK_VER}/${mode}/rook/deploy/examples/operator.yaml
+	while [[ $(kubectl get po -n rook-ceph --field-selector=status.phase=Running | grep -e 'operator' -c) -lt 1 ]]; do
+		echo "Waiting on operator"
+		sleep 10
+	done
 	kubectl apply -k "${ROOK_DIR}/overlays/${ROOK_VER}/${mode}"
 	# wait for the rook OSDs to run which means rooks should be ready
 	while [[ $(kubectl get po --all-namespaces | grep -e 'osd.*Running.*' -c) -lt 1 ]]; do
@@ -233,12 +256,13 @@ function monitoring() {
 	PROMETHEUS_DIR="4-kube-prometheus"
 	get_repo "${PROMETHEUS_URL}" "${PROMETHEUS_DIR}/overlays/${PROMETHEUS_VER}"
 	set_repo_version "${PROMETHEUS_VER}" "${PROMETHEUS_DIR}/overlays/${PROMETHEUS_VER}/kube-prometheus"
-	kubectl apply -k "${PROMETHEUS_DIR}/overlays/${PROMETHEUS_VER}"
+	kubectl apply --server-side -f "${PROMETHEUS_DIR}/overlays/${PROMETHEUS_VER}/kube-prometheus/manifests/setup/"

-	while [[ $(kubectl get crd alertmanagers.monitoring.coreos.com prometheuses.monitoring.coreos.com prometheusrules.monitoring.coreos.com servicemonitors.monitoring.coreos.com >/dev/null 2>&1) || $? -ne 0 ]]; do
-		echo "Waiting for Prometheus CRDs"
-		sleep 2
+	while ! $(kubectl get servicemonitors --all-namespaces) ; do
+		echo "Waiting for prometheus crds"
+		sleep 10
 	done
+	kubectl apply -k "${PROMETHEUS_DIR}/overlays/${PROMETHEUS_VER}/"

 	#Expose the dashboards
 	#kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090 &
@@ -369,6 +393,8 @@ command_handlers[monitoring]=monitoring
 command_handlers[metallb]=metallb
 command_handlers[npd]=npd
 command_handlers[nfd]=nfd
+command_handlers[kata]=kata
+command_handlers[metrics]=metrics

 declare -A command_help
 command_help[init]="Only inits a cluster using kubeadm"
@@ -0,0 +1,34 @@
+global
+    log /dev/log    local0
+    chroot /var/lib/haproxy
+    stats socket /run/haproxy-master.sock mode 660 level admin
+    stats timeout 30s
+    user haproxy
+    group haproxy
+    daemon
+    # Default SSL material locations
+    ca-base /etc/ssl/certs
+    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
+    ssl-default-bind-options no-sslv3
+defaults
+    log     global
+    mode    http
+    option  httplog
+    option  dontlognull
+    timeout connect 5000
+    timeout client  50000
+    timeout server  50000
+    timeout tunnel  4h
+frontend kubernetes
+    bind 10.0.0.100:6444
+    option tcplog
+    mode tcp
+    default_backend kubernetes-master-nodes
+
+backend kubernetes-master-nodes
+    mode tcp
+    balance source
+    option tcp-check
+    server master-1 10.0.0.100:6443 check fall 3 rise 2
+    server master-2 10.0.0.101:6443 check fall 3 rise 2
+    server master-3 10.0.0.102:6443 check fall 3 rise 2
@@ -4,8 +4,6 @@ kind: InitConfiguration
 apiVersion: kubelet.config.k8s.io/v1beta1
 kind: KubeletConfiguration
 cgroupDriver: systemd
-# Allowing for CPU pinning and isolation in case of guaranteed QoS class
-cpuManagerPolicy: static
 systemReserved:
  cpu: 500m
  memory: 256M
@@ -29,13 +29,10 @@ function add_os_deps() {

 # permanently disable swap
 function disable_swap() {
-	swapcount=$(sudo grep '^/dev/\([0-9a-z]*\).*' /proc/swaps | wc -l)
-
-	if [ "$swapcount" != "0" ]; then
-		sudo systemctl mask "$(sed -n -e 's#^/dev/\([0-9a-z]*\).*#dev-\1.swap#p' /proc/swaps)" 2>/dev/null
-	else
-		echo "Swap not enabled"
-	fi
+	# disable current swap
+	sudo swapoff -a
+	# permanently disable swap
+	sudo systemctl mask swap.target
 }

 # enable ip forwarding
@@ -137,7 +134,6 @@ function enable_kubelet_runner() {

 # ensure that the system is ready without requiring a reboot
 function ensure_system_ready() {
-	sudo swapoff -a
 	sudo systemctl restart systemd-modules-load.service
 }

@@ -161,7 +157,8 @@ function setup_proxy() {
 			echo "Warning, failed to find /etc/profile.d/proxy.sh to edit no_proxy line"
 		fi

-		cat <<EOF | sudo bash -c "cat > /usr/lib/systemd/system.conf.d/proxy.conf"
+		sudo mkdir -p /etc/systemd/system.conf.d
+		cat <<EOF | sudo bash -c "cat > /etc/systemd/system.conf.d/proxy.conf"
 [Manager]
 DefaultEnvironment="HTTP_PROXY=${http_proxy}"
 DefaultEnvironment="HTTPS_PROXY=${https_proxy}"
@@ -191,12 +188,12 @@ fi

 echo "Init..."
 init
+echo "Disabling swap..."
+disable_swap
 echo "Setting OS Version..."
 upate_os_version
 echo "Adding OS Dependencies..."
 add_os_deps
-echo "Disabling swap..."
-disable_swap
 echo "Enabling IP Forwarding..."
 enable_ip_forwarding
 echo "Setting up modules to load..."
@@ -209,7 +206,7 @@ if [[ -n "${HIGH_POD_COUNT}" ]]; then
 fi
 echo "Reloading daemons..."
 daemon_reload
-echo "Enabling Kublet runner..."
+echo "Enabling kubelet runner..."
 enable_kubelet_runner
 echo "Ensuring system is ready..."
 ensure_system_ready
@@ -1,160 +1,83 @@
+* [Metric testing for scaling on Kubernetes.](#metric-testing-for-scaling-on-kubernetes)
+    * [Results storage and analysis](#results-storage-and-analysis)
+* [Developers](#developers)
+    * [Metrics gathering](#metrics-gathering)
+        * [`collectd` statistics](#collectd-statistics)
+        * [privileged statistics pods](#privileged-statistics-pods)
+    * [Configuring constant 'loads'](#configuring-constant-loads)
+
 # Metric testing for scaling on Kubernetes.

 This folder contains tools to aid in measuring the scaling capabilities of
 Kubernetes clusters.

+Primarily these tools were designed to measure scaling of large number of pods on a single node, but
+the code is structured to handle multiple nodes, and may also be useful in that scenario.
+
 The tools tend to take one of two forms:

- Tools to take measurements
+- Tools to launch jobs and take measurements
 - Tools to analyse results

 For more details, see individual sub-folders. A brief summary of available tools
 is below:

-| Tool | Description |
+| Folder | Description |
 | ---- | ----------- |
 | collectd | `collectd` based statistics/metrics gathering daemonset code |
 | lib | General library helper functions for forming and launching workloads, and storing results in a uniform manner to aid later analysis |
-| lib/cpu-load* | Routines to enable CPU load generation on a cluster |
-| report | Rmarkdown based report generator, used to produce a PDF comparison report of 1 or more sets of results |
-| scaling | Tests to measure scaling, such as linear or parallel launching of pods |
+| lib/[cpu-load*](lib/cpu-load.md) | Helper functions to enable CPU load generation on a cluster whilst under test |
+| [report](report/README.md) | Rmarkdown based report generator, used to produce a PDF comparison report of one or more sets of results |
+| [scaling](scaling/README.md) | Tests to measure scaling, such as linear or parallel launching of pods |

 ## Results storage and analysis

-The tools generate JSON formatted results files via the `lib/json.bash` functions. The `metrics_json_save()`
-function in that file has the ability to also `curl` or `socat` the JSON results to a database defined
+The tools generate JSON formatted results files via the [`lib/json.bash`](lib/json.bash) functions. The `metrics_json_save()`
+function has the ability to also `curl` or `socat` the JSON results to a database defined
 by environment variables (see the file source for details). This method has been used to store results in
 Elasticsearch and InfluxDB databases for instance, but should be adaptable to use with any REST API that accepts
 JSON input.

-## Scaling execution
-This section describes a complete step-by-step scaling execution up to results reporting by using `scaling/k8s_scale.sh` tool which launches a series of workloads and take memory metric measurements after each launch.
+## Prerequisites
+
+There are some basic pre-requisites required in order to run the test and process the results:

-**Requirements**
 * A Kubernetes cluster up and running (tested on v1.15.3).
 * `bc` and `jq` packages.
 * Docker (only for report generation).

-The steps to execute a run of the scaling framework are listed below, which need to be executed on the master node of a Kubernetes cluster to avoid network issues:
-1. Clone `cloud-native-setup` repository into a preferred directory and change directory up to `cloud-native-setup/metrics`:
-   ```sh
-   $ git clone https://github.com/clearlinux/cloud-native-setup.git
-   $ cd cloud-native-setup/metrics
-   ```
-2. Launch the execution by:
-   ```sh
-   $ ./scaling/k8s_scale.sh
-   INFO: Initialising
-   command: bc: yes
-   command: jq: yes
-   INFO: Checking Kubernetes accessible
-   INFO: 1 Kubernetes nodes in 'Ready' state found
-   starting kubectl proxy
-   Starting to serve on 127.0.0.1:8090
-   daemonset.apps/stats created
-   Waiting for daemon set "stats" rollout to finish: 0 of 1 updated pods are available...
-   daemon set "stats" successfully rolled out
-   INFO: Running test
-   INFO: And grab some stats
-   INFO: idle [98.49] free [29031100] launch [0] node [clr-30f01b5149ba4ab8b05a7ee03b6812a5] inodes_free [31103039]
-   INFO: Testing replicas 1 of 20
-   INFO: Content of runtime_command=:/@RUNTIMECLASS@/d
-   ...
-   ```
-The above execution might take about 4min because it launch up to 20 pods by default and takes measurements for CPU utilization, memory utilization and pod boot time, finally it will generate a `k8s-scaling.json` result file at `result` directory.
-
-**Note**: to test the launch of pods concurrently, `k8s_parallel.sh` may be used. For quicker testing, `k8s_scale_rapid.sh` can be used in place of `k8s_scale.sh`. The rest of the launch instructions remain consistent other than script name.
-
-**Note**: by default the scaling framework makes call to the Kubernetes API directly so, if facing connectivity issues verify that `kubelet` service's proxies and `no_proxy` environment variable are properly setup.
-
-**Note**: by default the scaling framework uses default values for all its required variables, which can be checked through `scaling/k8s_scale.sh -h` and updated when launching the execution, i.e.:
-```
-$ ./scaling/k8s_scale.sh -h
-Usage: ./scaling/k8s_scale.sh [-h] [options]
-   Description:
-	Launch a series of workloads and take memory metric measurements after
-	each launch.
-   Options:
-		-h,    Help page.
-
-Environment variables:
-	Name (default)
-		Description
-	TEST_NAME (k8s scaling)
-		Can be set to over-ride the default JSON results filename
-	NUM_PODS (20)
-		Number of pods to launch
-	STEP (1)
-		Number of pods to launch per cycle
-	wait_time (30)
-		Seconds to wait for pods to become ready
-	delete_wait_time (600)
-		Seconds to wait for all pods to be deleted
-	settle_time (5)
-		Seconds to wait after pods ready before taking measurements
-	use_api (yes)
-		specify yes or no to use the API to launch pods
-	grace (30)
-		specify the grace period in seconds for workload pod termination
-
-$ use_api=no ./scaling/k8s_scale.sh
-```
-
-The steps to generate the result report are listed below:
-
-1. Having the `results/k8s-scaling.json` result file, create a subdirectory in the `results` directory with a preferred name and copy the `k8s-scaling.json` file into it, so the file distribution looks like:
-   ```sh
-   $ tree result
-   results/
-   └── scaling
-       └── k8s-scaling.json
-   ```
-
-**Note**: if `k8s_scale_rapid.sh` was run instead of `k8s_scale.sh`, that the `<node_name>.tar.gz` files that appear in the results directory also need to be copied into the newly created subdirectory. And the results file is named `k8s-rapid.json` rather than `k8s-scaling.json`.
-If k8s_parallel.sh was run, the results file is named `k8s-parallel.json` rather than `k8s-scaling.json`.
-
-2. Launch the report generation by:
-   ```sh
-   ./report/makereport.sh
-   ```
-   **Note**: the first time you launch the report generation it will build a docker container to generate the reports and this process can take several minutes. Subsequent runs will be much faster.
-
-   The above execution will generate a `report/output` directory with the final reports, such as:
-   ```sh
-   $ tree report/output/
-   report/output/
-   ├── dut-1.png
-   ├── metrics_report.pdf
-   ├── scaling-1.png
-   ├── scaling-2.png
-   ├── scaling-3.png
-   └── scaling-4.png
-   ```
-More details about result reporting can be reviewed at [`report`](./report) directory.
-
 # Developers

-This section provides some details of how the code is structured and configured. This may be of use whilst modifying
-existing or creating new tests.
+Below are some architecture and internal details of how the code is structured and configured. This will be
+helpful for improving, modifying or submitting fixes to the code base.

 ## Metrics gathering

-Metrics can be gathered using either a daemonset deployment of privileged pods used to gather statistics directly from the nodes using a combination of `mpstat`, `free` and `df`, or a daemonset deployment based around `collectd`.
+Metrics can be gathered using either a daemonset deployment of privileged pods used to gather statistics
+directly from the nodes using a combination of `mpstat`, `free` and `df`, or a daemonset deployment based
+around `collectd`. The general recommendation is to use the `collectd` based collection if possible, as it
+is more efficient, as the system does not have to poll and wait for results, and thus executes the test
+cycle faster. The `collectd` results are collected asyncronously, and the report generator code later
+aligns the results with the pod execution in the timeline.

 ### `collectd` statistics

-The `collected` based code can be found in the `collectd` subdirectory. It uses the `collected` configuration found in the `collectd.conf` file to gather statistics, and store the results on the nodes themselves whilst tests are running. At the end of the test, the results are copied from the nodes and stored in the results directory for later processing.
+The `collected` based code can be found in the `collectd` subdirectory. It uses the `collected` configuration
+found in the `collectd.conf` file to gather statistics, and store the results on the nodes themselves whilst
+tests are running. At the end of the test, the results are copied from the nodes and stored in the results
+directory for later processing.

-The `collectd` statistics are only configured and gathered if the environment variable `SMF_USE_COLLECTD` is set to non-empty by the test code (that is, only enabled upon request).
+The `collectd` statistics are only configured and gathered if the environment variable `SMF_USE_COLLECTD`
+is set to non-empty by the test code (that is, it is only enabled upon request).

 ### privileged statistics pods

-The privileged statistics pods `YAML` can be found in the `scaling/stats.yaml` file. An example of how to invoke and use this daemonset to extract statistics can be found in the `scaling/k8s_scale.sh` file.
+The privileged statistics pods `YAML` can be found in the [`scaling/stats.yaml`](scaling/stats.yaml) file.
+An example of how to invoke and use this daemonset to extract statistics can be found in the
+[`scaling/k8s_scale.sh`](scaling/k8s_scale.sh) file.

 ## Configuring constant 'loads'

-The framework includes some tooling to assist in setting up constant pre-defined 'loads' across the cluster to aid evaluation of their impacts on the scaling metrics.
-
-### CPU load generator
-
-Details of how to configure a constant CPU load are detailed in the [cpu-load documentation](lib/cpu-load.md).
+The framework includes some tooling to assist in setting up constant pre-defined 'loads' across the cluster
+to aid evaluation of their impacts on the scaling metrics. See the [cpu-load documentation](lib/cpu-load.md)
+for more information.
@@ -12,15 +12,25 @@ collectd_pod="collectd"
 init_stats() {
 	local wait_time=$1

-	# create collectd-config configmap
+	# create collectd-config configmap, delete old if there is one
+	kubectl get configmap collectd-config >/dev/null 2>&1 && kubectl delete configmap collectd-config
 	kubectl create configmap collectd-config --from-file=${COLLECTD_DIR}/collectd.conf

+	# if there is collectd daemonset already running, delete it
+	# to make sure that the latest configmap will be used.
+	kubectl get daemonset collectd >/dev/null 2>&1 && kubectl delete daemonset --wait=true --timeout=${delete_wait_time}s "${collectd_pod}"
+
 	# Launch our stats gathering pod
 	kubectl apply -f ${COLLECTD_DIR}/${collectd_pod}.yaml
 	kubectl rollout status --timeout=${wait_time}s daemonset/${collectd_pod}

+	# clear existing collectd output
+	while read -u 3 name node; do
+		kubectl exec -ti $name -- sh -c "rm -rf /mnt/opt/collectd/run/localhost/*"
+	done 3< <(kubectl get pods --selector name=collectd-pods -o json | jq -r '.items[] | "\(.metadata.name) \(.spec.nodeName)"')
+
 	# attempting to provide buffer for collectd to be installed and running,
-    # and CPU collection to build adequate history
+	# and CPU collection to build adequate history
 	sleep 12
 }

@@ -30,11 +40,9 @@ cleanup_stats() {

 	# get logs before shutting down stats daemonset
 	while read -u 3 name node; do
-		kubectl exec -ti $name -- sh -c "cd /opt/collectd; tar -czvf localhost.tar.gz localhost"
-		# make a backup on the host in-case collection fail
-		kubectl exec -ti $name -- sh -c "mkdir -p /mnt/opt/collectd"
-		kubectl exec -ti $name -- sh -c "cp /opt/collectd/localhost.tar.gz /mnt/opt/collectd/localhost.tar.gz"
-		kubectl cp $name:/opt/collectd/localhost.tar.gz ${RESULT_DIR}/${node}.tar.gz
+		kubectl exec -ti $name -- sh -c "cd /mnt/opt/collectd/run; rm -f ../localhost.tar.gz; tar -czvf ../localhost.tar.gz localhost"
+		kubectl cp $name:/mnt/opt/collectd/localhost.tar.gz ${RESULT_DIR}/${node}.tar.gz
+		kubectl exec -ti $name -- sh -c "rm -rf /mnt/opt/collectd/run"
 	done 3< <(kubectl get pods --selector name=collectd-pods -o json | jq -r '.items[] | "\(.metadata.name) \(.spec.nodeName)"')

 	kubectl delete daemonset --wait=true --timeout=${delete_wait_time}s "${collectd_pod}" || true
@@ -17,7 +17,7 @@ Hostname localhost
    ValuesPercentage true
 </Plugin>
 <Plugin "csv">
-    DataDir "/opt/collectd"
+    DataDir "/mnt/opt/collectd/run"
    StoreRates true
 </Plugin>
 <Plugin "interface">
@@ -13,10 +13,10 @@
 # We would have used the 'verse' base, that already has some of the docs processing
 # installed, but I could not figure out how to add in the extra bits we needed to
 # the lite tex version is uses.
-FROM rocker/tidyverse:latest
+FROM rocker/tidyverse:3.6.0

 # Version of the Dockerfile
-LABEL DOCKERFILE_VERSION="1.0"
+LABEL DOCKERFILE_VERSION="1.1"

 # Without this some of the package installs stop to try and ask questions...
 ENV DEBIAN_FRONTEND=noninteractive
@@ -13,95 +13,105 @@ library(gridExtra)					# together.
 suppressMessages(suppressWarnings(library(ggpubr)))	# for ggtexttable.
 suppressMessages(library(jsonlite))			# to load the data.

-# A list of all the known results files we might find the information inside.
-resultsfiles=c(
-	"k8s-parallel.json",
-	"k8s-scaling.json",
-	"k8s-rapid.json"
-	)
+render_dut_details <- function()
+{
+	# A list of all the known results files we might find the information inside.
+	resultsfiles=c(
+		"k8s-parallel.json",
+		"k8s-scaling.json",
+		"k8s-rapid.json"
+		)

-data=c()
-stats=c()
-stats_names=c()
+	data=c()
+	stats=c()
+	stats_names=c()

-# For each set of results
-for (currentdir in resultdirs) {
-	count=1
-	dirstats=c()
-	for (resultsfile in resultsfiles) {
-		fname=paste(inputdir, currentdir, resultsfile, sep="/")
-		if ( !file.exists(fname)) {
-			#warning(paste("Skipping non-existent file: ", fname))
-			next
-		}
-
-		# Derive the name from the test result dirname
-		datasetname=basename(currentdir)
-
-		# Import the data
-		fdata=fromJSON(fname)
-
-		if (length(fdata$'kubectl-version') != 0 ) {
-			# We have kata-runtime data
-			dirstats=tibble("Client Ver"=as.character(fdata$'kubectl-version'$clientVersion$gitVersion))
-			dirstats=cbind(dirstats, "Server Ver"=as.character(fdata$'kubectl-version'$serverVersion$gitVersion))
-			numnodes= nrow(fdata$'kubectl-get-nodes'$items)
-			dirstats=cbind(dirstats, "No. nodes"=as.character(numnodes))
-
-			if (numnodes != 0) {
-				first_node=fdata$'kubectl-get-nodes'$items[1,]
-				dirstats=cbind(dirstats, "- Node0 name"=as.character(first_node$metadata$name))
-
-				havekata=first_node$metadata$labels$'katacontainers.io/kata-runtime'
-				if ( is.null(havekata) ) {
-					dirstats=cbind(dirstats, "  Have Kata"=as.character('false'))
-				} else {
-					dirstats=cbind(dirstats, "  Have Kata"=as.character(havekata))
-				}
-
-				dirstats=cbind(dirstats, "  CPUs"=as.character(first_node$status$capacity$cpu))
-				dirstats=cbind(dirstats, "  Memory"=as.character(first_node$status$capacity$memory))
-				dirstats=cbind(dirstats, "  MaxPods"=as.character(first_node$status$capacity$pods))
-				dirstats=cbind(dirstats, "  PodCIDR"=as.character(first_node$spec$podCIDR))
-
-				dirstats=cbind(dirstats, "  runtime"=as.character(first_node$status$nodeInfo$containerRuntimeVersion))
-				dirstats=cbind(dirstats, "  kernel"=as.character(first_node$status$nodeInfo$kernelVersion))
-				dirstats=cbind(dirstats, "  kubeProxy"=as.character(first_node$status$nodeInfo$kubeProxyVersion))
-				dirstats=cbind(dirstats, "  Kubelet"=as.character(first_node$status$nodeInfo$kubeletVersion))
-				dirstats=cbind(dirstats, "  OS"=as.character(first_node$status$nodeInfo$osImage))
+	# For each set of results
+	for (currentdir in resultdirs) {
+		count=1
+		dirstats=c()
+		datasetname=c()
+		for (resultsfile in resultsfiles) {
+			fname=paste(inputdir, currentdir, resultsfile, sep="/")
+			if ( !file.exists(fname)) {
+				#warning(paste("Skipping non-existent file: ", fname))
+				next
 			}

-			break
+			# Derive the name from the test result dirname
+			datasetname=basename(currentdir)
+
+			# Import the data
+			fdata=fromJSON(fname)
+
+			if (length(fdata$'kubectl-version') != 0 ) {
+				# We have kata-runtime data
+				dirstats=tibble("Client Ver"=as.character(fdata$'kubectl-version'$clientVersion$gitVersion))
+				dirstats=cbind(dirstats, "Server Ver"=as.character(fdata$'kubectl-version'$serverVersion$gitVersion))
+				numnodes= nrow(fdata$'kubectl-get-nodes'$items)
+				dirstats=cbind(dirstats, "No. nodes"=as.character(numnodes))
+
+				if (numnodes != 0) {
+					first_node=fdata$'kubectl-get-nodes'$items[1,]
+					dirstats=cbind(dirstats, "- Node0 name"=as.character(first_node$metadata$name))
+
+					havekata=first_node$metadata$labels$'katacontainers.io/kata-runtime'
+					if ( is.null(havekata) ) {
+						dirstats=cbind(dirstats, "  Have Kata"=as.character('false'))
+					} else {
+						dirstats=cbind(dirstats, "  Have Kata"=as.character(havekata))
+					}
+
+					dirstats=cbind(dirstats, "  CPUs"=as.character(first_node$status$capacity$cpu))
+					dirstats=cbind(dirstats, "  Memory"=as.character(first_node$status$capacity$memory))
+					dirstats=cbind(dirstats, "  MaxPods"=as.character(first_node$status$capacity$pods))
+					dirstats=cbind(dirstats, "  PodCIDR"=as.character(first_node$spec$podCIDR))
+
+					dirstats=cbind(dirstats, "  runtime"=as.character(first_node$status$nodeInfo$containerRuntimeVersion))
+					dirstats=cbind(dirstats, "  kernel"=as.character(first_node$status$nodeInfo$kernelVersion))
+					dirstats=cbind(dirstats, "  kubeProxy"=as.character(first_node$status$nodeInfo$kubeProxyVersion))
+					dirstats=cbind(dirstats, "  Kubelet"=as.character(first_node$status$nodeInfo$kubeletVersion))
+					dirstats=cbind(dirstats, "  OS"=as.character(first_node$status$nodeInfo$osImage))
+				}
+
+				break
+			}
 		}
+
+		if ( length(dirstats) == 0 ) {
+			cat(paste("No valid data found for directory ", currentdir, "\n\n"))
+		}
+
+		# use plyr rbind.fill so we can combine disparate version info frames
+		stats=rbind.fill(stats, dirstats)
+		stats_names=rbind(stats_names, datasetname)
 	}

-	if ( length(dirstats) == 0 ) {
-		warning(paste("No valid data found for directory ", currentdir))
+	if ( length(stats_names) == 0 ) {
+		cat("No system details found\n\n")
+		return()
 	}

-	# use plyr rbind.fill so we can combine disparate version info frames
-	stats=rbind.fill(stats, dirstats)
-	stats_names=rbind(stats_names, datasetname)
+	rownames(stats) = stats_names
+
+	# Rotate the tibble so we get data dirs as the columns
+	spun_stats = as_tibble(cbind(What=names(stats), t(stats)))
+
+	# Build us a text table of numerical results
+	# Set up as left hand justify, so the node data indent renders.
+	tablefontsize=8
+	tbody.style = tbody_style(hjust=0, x=0.1, size=tablefontsize)
+	stats_plot = suppressWarnings(ggtexttable(data.frame(spun_stats, check.names=FALSE),
+		theme=ttheme(base_size=tablefontsize, tbody.style=tbody.style),
+		rows=NULL
+		))
+
+	# It may seem odd doing a grid of 1x1, but it should ensure we get a uniform format and
+	# layout to match the other charts and tables in the report.
+	master_plot = grid.arrange(
+		stats_plot,
+		nrow=1,
+		ncol=1 )
 }

-rownames(stats) = stats_names
-
-# Rotate the tibble so we get data dirs as the columns
-spun_stats = as_tibble(cbind(What=names(stats), t(stats)))
-
-# Build us a text table of numerical results
-# Set up as left hand justify, so the node data indent renders.
-tablefontsize=8
-tbody.style = tbody_style(hjust=0, x=0.1, size=tablefontsize)
-stats_plot = suppressWarnings(ggtexttable(data.frame(spun_stats, check.names=FALSE),
-	theme=ttheme(base_size=tablefontsize, tbody.style=tbody.style),
-	rows=NULL
-	))
-
-# It may seem odd doing a grid of 1x1, but it should ensure we get a uniform format and
-# layout to match the other charts and tables in the report.
-master_plot = grid.arrange(
-	stats_plot,
-	nrow=1,
-	ncol=1 )
-
+render_dut_details()
@@ -0,0 +1,26 @@
+
+library('elasticsearchr')
+
+for_scaling <- query('{
+"bool": {
+	"must": [
+	{ "match":
+		{
+			"test.testname": "k8s scaling"
+		}
+	}
+	]
+}
+}')
+
+these_fields <- select_fields('{
+	"includes": [
+		"date.Date",
+		"k8s-scaling.BootResults.launch_time.Result",
+		"k8s-scaling.BootResults.n_pods.Result"
+	]
+}')
+
+sort_by_date <- sort_on('[{"date.Date": {"order": "asc"}}]')
+
+x=elastic("http://192.168.0.111:9200", "logtest") %search% (for_scaling + sort_by_date + these_fields)
@@ -32,7 +32,7 @@ This [test](https://github.com/clearlinux/cloud-native-setup/metrics/scaling/k8s
 measures the time taken to launch and delete pods in parallel using a deployment. The times
 are how long it takes for the whole deployment operation to complete.

-```{r parallel, echo=FALSE, fig.cap="K8S parallel pods"}
+```{r parallel, echo=FALSE, fig.cap="K8S parallel pods", results='asis'}
 source('parallel.R')
 ```

@@ -57,7 +57,7 @@ This table describes the test system details, as derived from the information co
 in the test results files.


-```{r dut, echo=FALSE, fig.cap="System configuration details"}
+```{r dut, echo=FALSE, fig.cap="System configuration details", results='asis'}
 source('dut-details.R')
 ```

@@ -67,6 +67,6 @@ source('dut-details.R')

 This table describes node details within the Kubernetes cluster that have been used for test.

-```{r node, echo=FALSE, fig.cap="Node information within Kubernetes cluster"}
+```{r node, echo=FALSE, fig.cap="Node information within Kubernetes cluster", results='asis'}
 source('node-info.R')
 ```
@@ -13,83 +13,96 @@ library(gridExtra)                                      # together.
 suppressMessages(suppressWarnings(library(ggpubr)))     # for ggtexttable.
 suppressMessages(library(jsonlite))                     # to load the data.

-# A list of all the known results files we might find the information inside.
-resultsfiles=c(
-	       "k8s-scaling.json"
-	       )
+render_node_info <- function()
+{
+	# A list of all the known results files we might find the information inside.
+	resultsfiles=c(
+		       "k8s-scaling.json"
+		       )

-stats=c()
-stats_names=c()
-max_char_name_node=18
+	stats=c()
+	stats_names=c()
+	datasetname=c()
+	complete_data=c()
+	max_char_name_node=18

-# list for each dirstats
-dirstats_list=list()
-j=1
+	# list for each dirstats
+	dirstats_list=list()
+	j=1

-# For each set of results
-for (currentdir in resultdirs) {
-	dirstats=c()
-	for (resultsfile in resultsfiles) {
-		fname=paste(inputdir, currentdir, resultsfile, sep="/")
-		if ( !file.exists(fname)) {
-			next
-		}
-
-		# Derive the name from the test result dirname
-		datasetname=basename(currentdir)
-
-		# Import the data
-		fdata=fromJSON(fname)
-
-		if (length(fdata$'kubectl-version') != 0 ) {
-			numnodes= nrow(fdata$'kubectl-get-nodes'$items)
-			for (i in 1:numnodes) {
-				node_i=fdata$'kubectl-get-nodes'$items[i,]
-				node_info=fdata$'socketsPerNode'[i,]
-
-				# Substring node name so it fits properly into final table
-				node_name=node_i$metadata$name
-				if ( nchar(node_name) >= max_char_name_node) {
-					dirstats=tibble("Node \nname"=as.character(substring(node_name, 1, max_char_name_node)))
-				} else {
-					dirstats=tibble("Node \nname"=as.character(node_name))
-				}
-
-				dirstats=cbind(dirstats, "CPUs"=as.character(node_i$status$capacity$cpu))
-				dirstats=cbind(dirstats, "Memory"=as.character(node_i$status$capacity$memory))
-				dirstats=cbind(dirstats, "Max \nPods"=as.character(node_i$status$capacity$pods))
-				dirstats=cbind(dirstats, "Count \nsockets"=as.character(node_info$num_sockets))
-				dirstats=cbind(dirstats, "Have \nhypervisor"=as.character(node_info$hypervisor))
-
-				dirstats=cbind(dirstats, "kernel"=as.character(node_i$status$nodeInfo$kernelVersion))
-				dirstats=cbind(dirstats, "OS"=as.character(node_i$status$nodeInfo$osImage))
-				dirstats=cbind(dirstats, "Test"=as.character(datasetname))
-
-				dirstats_list[[j]]=dirstats
-				j=j+1
+	# For each set of results
+	for (currentdir in resultdirs) {
+		dirstats=c()
+		for (resultsfile in resultsfiles) {
+			fname=paste(inputdir, currentdir, resultsfile, sep="/")
+			if ( !file.exists(fname)) {
+				next
+			}
+
+			# Derive the name from the test result dirname
+			datasetname=basename(currentdir)
+
+			# Import the data
+			fdata=fromJSON(fname)
+
+			if (length(fdata$'kubectl-version') != 0 ) {
+				numnodes= nrow(fdata$'kubectl-get-nodes'$items)
+				for (i in 1:numnodes) {
+					node_i=fdata$'kubectl-get-nodes'$items[i,]
+					node_info=fdata$'socketsPerNode'[i,]
+
+					# Substring node name so it fits properly into final table
+					node_name=node_i$metadata$name
+					if ( nchar(node_name) >= max_char_name_node) {
+						dirstats=tibble("Node \nname"=as.character(substring(node_name, 1, max_char_name_node)))
+					} else {
+						dirstats=tibble("Node \nname"=as.character(node_name))
+					}
+
+					dirstats=cbind(dirstats, "CPUs"=as.character(node_i$status$capacity$cpu))
+					dirstats=cbind(dirstats, "Memory"=as.character(node_i$status$capacity$memory))
+					dirstats=cbind(dirstats, "Max \nPods"=as.character(node_i$status$capacity$pods))
+					dirstats=cbind(dirstats, "Count \nsockets"=as.character(node_info$num_sockets))
+					dirstats=cbind(dirstats, "Have \nhypervisor"=as.character(node_info$hypervisor))
+
+					dirstats=cbind(dirstats, "kernel"=as.character(node_i$status$nodeInfo$kernelVersion))
+					dirstats=cbind(dirstats, "OS"=as.character(node_i$status$nodeInfo$osImage))
+					dirstats=cbind(dirstats, "Test"=as.character(datasetname))
+
+					dirstats_list[[j]]=dirstats
+					j=j+1
+				}
+				complete_data = do.call(rbind, dirstats_list)
 			}
-			complete_data = do.call(rbind, dirstats_list)
 		}
+
+		if ( length(complete_data) == 0 ) {
+			cat(paste("No valid data found for directory ", currentdir, "\n\n"))
+		}
+
+		# use plyr rbind.fill so we can combine disparate version info frames
+		stats=rbind.fill(stats, complete_data)
+		stats_names=rbind(stats_names, datasetname)
 	}

-	if ( length(complete_data) == 0 ) {
-		warning(paste("No valid data found for directory ", currentdir))
+	if ( length(stats_names) == 0 ) {
+		cat("No node stats found\n\n");
+		return()
 	}

-	# use plyr rbind.fill so we can combine disparate version info frames
-	stats=rbind.fill(stats, complete_data)
-	stats_names=rbind(stats_names, datasetname)
+	# Build us a text table of numerical results
+	# Set up as left hand justify, so the node data indent renders.
+	tablefontsize=8
+	tbody.style = tbody_style(hjust=0, x=0.1, size=tablefontsize)
+	stats_plot = suppressWarnings(ggtexttable(data.frame(complete_data, check.names=FALSE),
+						  theme=ttheme(base_size=tablefontsize, tbody.style=tbody.style),
+						  rows=NULL))
+
+	# It may seem odd doing a grid of 1x1, but it should ensure we get a uniform format and
+	# layout to match the other charts and tables in the report.
+	master_plot = grid.arrange(stats_plot,
+				   nrow=1,
+				   ncol=1 )
 }
-# Build us a text table of numerical results
-# Set up as left hand justify, so the node data indent renders.
-tablefontsize=8
-tbody.style = tbody_style(hjust=0, x=0.1, size=tablefontsize)
-stats_plot = suppressWarnings(ggtexttable(data.frame(complete_data, check.names=FALSE),
-					  theme=ttheme(base_size=tablefontsize, tbody.style=tbody.style),
-					  rows=NULL))

-# It may seem odd doing a grid of 1x1, but it should ensure we get a uniform format and
-# layout to match the other charts and tables in the report.
-master_plot = grid.arrange(stats_plot,
-			   nrow=1,
-			   ncol=1 )
+render_node_info()
@@ -13,113 +13,123 @@ suppressMessages(suppressWarnings(library(ggpubr)))	# for ggtexttable.
 suppressMessages(library(jsonlite))			# to load the data.
 suppressMessages(library(scales))			# For de-science notation of axis

-testnames=c(
-	"k8s-parallel*"
-)
+render_parallel <- function()
+{
+	testnames=c(
+		"k8s-parallel*"
+	)

-data=c()
-stats=c()
-rstats=c()
-rstats_names=c()
-cstats=c()
-cstats_names=c()
+	data=c()
+	stats=c()
+	rstats=c()
+	rstats_names=c()
+	cstats=c()
+	cstats_names=c()

-skip_points_enable_smooth=0	# Should we draw the points as well as lines on the graphs.
+	skip_points_enable_smooth=0	# Should we draw the points as well as lines on the graphs.

-for (currentdir in resultdirs) {
-	dirstats=c()
-	for (testname in testnames) {
-		matchdir=paste(inputdir, currentdir, sep="")
-		matchfile=paste(testname, '\\.json', sep="")
-		files=list.files(matchdir, pattern=matchfile)
-		if ( length(files) == 0 ) {
-			#warning(paste("Pattern [", matchdir, "/", matchfile, "] matched nothing"))
-		}
-		for (ffound in files) {
-			fname=paste(inputdir, currentdir, ffound, sep="")
-			if ( !file.exists(fname)) {
-				warning(paste("Skipping non-existent file: ", fname))
-				next
+	for (currentdir in resultdirs) {
+		dirstats=c()
+		for (testname in testnames) {
+			matchdir=paste(inputdir, currentdir, sep="")
+			matchfile=paste(testname, '\\.json', sep="")
+			files=list.files(matchdir, pattern=matchfile)
+			if ( length(files) == 0 ) {
+				#warning(paste("Pattern [", matchdir, "/", matchfile, "] matched nothing"))
 			}
+			for (ffound in files) {
+				fname=paste(inputdir, currentdir, ffound, sep="")
+				if ( !file.exists(fname)) {
+					warning(paste("Skipping non-existent file: ", fname))
+					next
+				}

-			# Derive the name from the test result dirname
-			datasetname=basename(currentdir)
+				# Derive the name from the test result dirname
+				datasetname=basename(currentdir)

-			# Import the data
-			fdata=fromJSON(fname)
-			# De-nest the test name specific data
-			shortname=substr(ffound, 1, nchar(ffound)-nchar(".json"))
-			fdata=fdata[[shortname]]
+				# Import the data
+				fdata=fromJSON(fname)
+				# De-nest the test name specific data
+				shortname=substr(ffound, 1, nchar(ffound)-nchar(".json"))
+				fdata=fdata[[shortname]]

-			testname=datasetname
+				testname=datasetname

-			# convert ms to seconds
-			cdata=data.frame(boot_time=as.numeric(fdata$BootResults$launch_time$Result)/1000)
-			cdata=cbind(cdata, delete_time=as.numeric(fdata$BootResults$delete_time$Result)/1000)
-			cdata=cbind(cdata, npod=as.numeric(fdata$BootResults$n_pods$Result))
+				# convert ms to seconds
+				cdata=data.frame(boot_time=as.numeric(fdata$BootResults$launch_time$Result)/1000)
+				cdata=cbind(cdata, delete_time=as.numeric(fdata$BootResults$delete_time$Result)/1000)
+				cdata=cbind(cdata, npod=as.numeric(fdata$BootResults$n_pods$Result))

-			# If we have more than 20 items to draw, then do not draw the points on
-			# the graphs, as they are then too noisy to read.
-			# But, do draw the smoothed lines to help read the now dense and potentially
-			# noisy graphs.
-			if (length(cdata[, "boot_time"]) > 20) {
-				skip_points_enable_smooth=1
+				# If we have more than 20 items to draw, then do not draw the points on
+				# the graphs, as they are then too noisy to read.
+				# But, do draw the smoothed lines to help read the now dense and potentially
+				# noisy graphs.
+				if (length(cdata[, "boot_time"]) > 20) {
+					skip_points_enable_smooth=1
+				}
+
+				cdata=cbind(cdata, testname=rep(testname, length(cdata[, "boot_time"]) ))
+				cdata=cbind(cdata, dataset=rep(datasetname, length(cdata[, "boot_time"]) ))
+
+				# Store away as a single set
+				data=rbind(data, cdata)
 			}
-
-			cdata=cbind(cdata, testname=rep(testname, length(cdata[, "boot_time"]) ))
-			cdata=cbind(cdata, dataset=rep(datasetname, length(cdata[, "boot_time"]) ))
-
-			# Store away as a single set
-			data=rbind(data, cdata)
 		}
 	}
+
+	# If we found nothing to process, quit early and nicely
+	if ( length(data) == 0 ) {
+		cat("No results files found for parallel tests\n\n")
+		return()
+	}
+
+	# Show how boot time changed
+	boot_line_plot <- ggplot( data=data, aes(npod, boot_time, colour=testname, group=dataset)) +
+		geom_line( alpha=0.2) +
+		xlab("parallel pods") +
+		ylab("Boot time (s)") +
+		ggtitle("Deployment boot time (detail)") +
+		#ylim(0, NA) + # For big machines, better to not 0-index
+		theme(axis.text.x=element_text(angle=90))
+
+		if ( skip_points_enable_smooth == 0 ) {
+			boot_line_plot = boot_line_plot + geom_point(alpha=0.3)
+		} else {
+			boot_line_plot = bool_line_plot + geom_smooth(se=FALSE, method="loess", size=0.3)
+		}
+
+		# And get a zero Y index plot.
+		boot_line_plot_zero = boot_line_plot + ylim(0, NA) +
+			ggtitle("Deployment boot time (0 index)")
+
+	# Show how boot time changed
+	delete_line_plot <- ggplot( data=data, aes(npod, delete_time, colour=testname, group=dataset)) +
+		geom_line(alpha=0.2) +
+		xlab("parallel pods") +
+		ylab("Delete time (s)") +
+		ggtitle("Deployment deletion time (detail)") +
+		#ylim(0, NA) + # For big machines, better to not 0-index
+		theme(axis.text.x=element_text(angle=90))
+
+		if ( skip_points_enable_smooth == 0 ) {
+			delete_line_plot = delete_line_plot + geom_point(alpha=0.3)
+		} else {
+			delete_line_plot = delete_line_plot + geom_smooth(se=FALSE, method="loess", size=0.3)
+		}
+
+		# And get a 0 indexed Y axis plot
+		delete_line_plot_zero = delete_line_plot + ylim(0, NA) +
+			ggtitle("Deployment deletion time (0 index)")
+
+	# See https://www.r-bloggers.com/ggplot2-easy-way-to-mix-multiple-graphs-on-the-same-page/ for
+	# excellent examples
+	master_plot = grid.arrange(
+		boot_line_plot_zero,
+		delete_line_plot_zero,
+		boot_line_plot,
+		delete_line_plot,
+		nrow=2,
+		ncol=2 )
 }

-# Show how boot time changed
-boot_line_plot <- ggplot( data=data, aes(npod, boot_time, colour=testname, group=dataset)) +
-	geom_line( alpha=0.2) +
-	xlab("parallel pods") +
-	ylab("Boot time (s)") +
-	ggtitle("Deployment boot time (detail)") +
-	#ylim(0, NA) + # For big machines, better to not 0-index
-	theme(axis.text.x=element_text(angle=90))
-
-	if ( skip_points_enable_smooth == 0 ) {
-		boot_line_plot = boot_line_plot + geom_point(alpha=0.3)
-	} else {
-		boot_line_plot = bool_line_plot + geom_smooth(se=FALSE, method="loess", size=0.3)
-	}
-
-	# And get a zero Y index plot.
-	boot_line_plot_zero = boot_line_plot + ylim(0, NA) +
-		ggtitle("Deployment boot time (0 index)")
-
-# Show how boot time changed
-delete_line_plot <- ggplot( data=data, aes(npod, delete_time, colour=testname, group=dataset)) +
-	geom_line(alpha=0.2) +
-	xlab("parallel pods") +
-	ylab("Delete time (s)") +
-	ggtitle("Deployment deletion time (detail)") +
-	#ylim(0, NA) + # For big machines, better to not 0-index
-	theme(axis.text.x=element_text(angle=90))
-
-	if ( skip_points_enable_smooth == 0 ) {
-		delete_line_plot = delete_line_plot + geom_point(alpha=0.3)
-	} else {
-		delete_line_plot = delete_line_plot + geom_smooth(se=FALSE, method="loess", size=0.3)
-	}
-
-	# And get a 0 indexed Y axis plot
-	delete_line_plot_zero = delete_line_plot + ylim(0, NA) +
-		ggtitle("Deployment deletion time (0 index)")
-
-# See https://www.r-bloggers.com/ggplot2-easy-way-to-mix-multiple-graphs-on-the-same-page/ for
-# excellent examples
-master_plot = grid.arrange(
-	boot_line_plot_zero,
-	delete_line_plot_zero,
-	boot_line_plot,
-	delete_line_plot,
-	nrow=2,
-	ncol=2 )
-
+render_parallel()
@@ -0,0 +1,9 @@
+
+suppressMessages(library(jsonlite))			# to load the data.
+
+options(digits=22)
+
+x=fromJSON('{"ns": 1567002188374607769}')
+
+print(x)
+print(fromJSON('{"ns": 1567002188374607769}'), digits=22)
@@ -13,308 +13,320 @@ suppressMessages(library(jsonlite))			# to load the data.
 suppressMessages(library(scales))			# For de-science notation of axis
 library(tibble)						# tibbles for tidy data

-testnames=c(
-	"k8s-scaling.*"
-)
+render_tidy_scaling <- function()
+{
+	testnames=c(
+		"k8s-scaling.*"
+	)

-bootdata=c()	# Track per-launch data
-nodedata=c()	# Track node status data
-memstats=c()	# Statistics for memory usage
-cpustats=c()	# Statistics for cpu usage
-bootstats=c()	# Statistics for boot (launch) times
-inodestats=c()	# Statistics for inode usage
+	bootdata=c()	# Track per-launch data
+	nodedata=c()	# Track node status data
+	memstats=c()	# Statistics for memory usage
+	cpustats=c()	# Statistics for cpu usage
+	bootstats=c()	# Statistics for boot (launch) times
+	inodestats=c()	# Statistics for inode usage

-# iterate over every set of results (test run)
-for (currentdir in resultdirs) {
-	# For every results file we are interested in evaluating
-	for (testname in testnames) {
-		matchdir=paste(inputdir, currentdir, sep="")
-		matchfile=paste(testname, '\\.json', sep="")
-		files=list.files(matchdir, pattern=matchfile)
-		if ( length(files) == 0 ) {
-			#warning(paste("Pattern [", matchdir, "/", matchfile, "] matched nothing"))
-		}
-
-		# For every matching results file
-		for (ffound in files) {
-			fname=paste(inputdir, currentdir, ffound, sep="")
-			if ( !file.exists(fname)) {
-				warning(paste("Skipping non-existent file: ", fname))
-				next
+	# iterate over every set of results (test run)
+	for (currentdir in resultdirs) {
+		# For every results file we are interested in evaluating
+		for (testname in testnames) {
+			matchdir=paste(inputdir, currentdir, sep="")
+			matchfile=paste(testname, '\\.json', sep="")
+			files=list.files(matchdir, pattern=matchfile)
+			if ( length(files) == 0 ) {
+				#warning(paste("Pattern [", matchdir, "/", matchfile, "] matched nothing"))
 			}

-			# Derive the name from the test result dirname
-			datasetname=basename(currentdir)
-
-			# Import the data
-			fdata=fromJSON(fname)
-			# De-nest the test name specific data
-			shortname=substr(ffound, 1, nchar(ffound)-nchar(".json"))
-			fdata=fdata[[shortname]]
-			testname=datasetname
-
-			# Most of the data we are looking for comes in BootResults, so pick it out to make
-			# referencing easier
-			br=fdata$BootResults
-
-			# The launched pods is a list of data frames when imported. It is much nicer
-			# for us to work with it as a single data frame, so convert it...
-			lp=do.call("rbind", br$launched_pods)
-
-			########################################################
-			#### Now extract all the pod launch boot data items ####
-			########################################################
-			local_bootdata=tibble(launch_time=br$launch_time$Result)
-			local_bootdata=cbind(local_bootdata, n_pods=br$n_pods$Result)
-			local_bootdata=cbind(local_bootdata, node=lp$node)
-			local_bootdata=cbind(local_bootdata, testname=rep(testname, length(local_bootdata$node)))
-
-
-			########################################################
-			#### Now extract all node performance information ######
-			########################################################
-			nu=br$node_util
-
-			# We need to associate a pod count with each result, but you
-			# get one result per-node, and the JSON does not carry the pod
-			# count in that table. Walk the node util structure, assigning the
-			# n_pods value from the boot results over to the list of node util
-			# entries associated with it - creating a new 'n_pods' field in the
-			# node util dataframe.
-			for (n in seq(length(br$n_pods$Result))) {
-				nu[[n]]$n_pods = br$n_pods$Result[[n]]
-			}
-
-			# node_util is a list of nested data frames. I'm sure there is some better R'ish
-			# way of extracting this data maybe with dplyr, map, select or melt, but I can't
-			# work it out right now, and at least this is semi-readable...
-			#
-			# Basically, we are de-listing and flattening the lists of dataframes into a
-			# singly 'tidy' dataframe...
-			nodes=do.call("rbind", lapply(nu, "[", "node"))
-			noschedule=do.call("rbind", lapply(nu, "[", "noschedule"))
-			n_pods=do.call("rbind", lapply(nu, "[", "n_pods"))
-			idle=lapply(nu, "[", "cpu_idle")
-			idle_df=do.call("rbind", lapply(idle, "[[", "cpu_idle"))
-			free=lapply(nu, "[", "mem_free")
-			free_df=do.call("rbind", lapply(free, "[[", "mem_free"))
-			used=lapply(nu, "[", "mem_used")
-			used_df=do.call("rbind", lapply(used, "[[", "mem_used"))
-			ifree=lapply(nu, "[", "inode_free")
-			ifree_df=do.call("rbind", lapply(ifree, "[[", "inode_free"))
-			iused=lapply(nu, "[", "inode_used")
-			iused_df=do.call("rbind", lapply(iused, "[[", "inode_used"))
-
-
-			# and build our rows
-			local_nodedata=tibble(node=nodes$node)
-			local_nodedata=cbind(local_nodedata, n_pods=n_pods)
-			local_nodedata=cbind(local_nodedata, noschedule=noschedule)
-			local_nodedata=cbind(local_nodedata, idle=idle_df$Result)
-			local_nodedata=cbind(local_nodedata, mem_free=free_df$Result)
-			local_nodedata=cbind(local_nodedata, mem_used=used_df$Result)
-			local_nodedata=cbind(local_nodedata, inode_free=ifree_df$Result)
-			local_nodedata=cbind(local_nodedata, inode_used=iused_df$Result)
-			local_nodedata=cbind(local_nodedata, testname=rep(testname, length(local_nodedata$node)))
-
-			# Now Calculate some stats. This gets more complicated as we may have n-nodes,
-			# and we want to show a 'pod average', so we try to assess for all nodes. If
-			# we have different 'size' nodes in a cluster, that could throw out the result,
-			# but the only other option would be to try and show every node separately in the
-			# table.
-
-			# Get a list of all the nodes
-			nodes=unique(local_nodedata$node)
-
-			memtotal=0
-			cputotal=0
-			inodetotal=0
-			# Calculate per-node totals, and tot them up to a global total.
-			for (n in nodes) {
-				# Make a frame with just that nodes data in
-				thisnode=subset(local_nodedata, node %in% c(n))
-
-				# Do not use the master (non-schedulable) nodes to calculate
-				# launched pod metrics
-				if(thisnode[1,]$noschedule == "true") {
+			# For every matching results file
+			for (ffound in files) {
+				fname=paste(inputdir, currentdir, ffound, sep="")
+				if ( !file.exists(fname)) {
+					warning(paste("Skipping non-existent file: ", fname))
 					next
 				}
-				memtotal = memtotal + thisnode[nrow(thisnode),]$mem_used
-				cpuused = thisnode[1,]$idle - thisnode[nrow(thisnode),]$idle
-				cputotal = cputotal + cpuused
-				inodetotal = inodetotal + thisnode[nrow(thisnode),]$inode_used
+
+				# Derive the name from the test result dirname
+				datasetname=basename(currentdir)
+
+				# Import the data
+				fdata=fromJSON(fname)
+				# De-nest the test name specific data
+				shortname=substr(ffound, 1, nchar(ffound)-nchar(".json"))
+				fdata=fdata[[shortname]]
+				testname=datasetname
+
+				# Most of the data we are looking for comes in BootResults, so pick it out to make
+				# referencing easier
+				br=fdata$BootResults
+
+				# The launched pods is a list of data frames when imported. It is much nicer
+				# for us to work with it as a single data frame, so convert it...
+				lp=do.call("rbind", br$launched_pods)
+
+				########################################################
+				#### Now extract all the pod launch boot data items ####
+				########################################################
+				local_bootdata=tibble(launch_time=br$launch_time$Result)
+				local_bootdata=cbind(local_bootdata, n_pods=br$n_pods$Result)
+				local_bootdata=cbind(local_bootdata, node=lp$node)
+				local_bootdata=cbind(local_bootdata, testname=rep(testname, length(local_bootdata$node)))
+
+
+				########################################################
+				#### Now extract all node performance information ######
+				########################################################
+				nu=br$node_util
+
+				# We need to associate a pod count with each result, but you
+				# get one result per-node, and the JSON does not carry the pod
+				# count in that table. Walk the node util structure, assigning the
+				# n_pods value from the boot results over to the list of node util
+				# entries associated with it - creating a new 'n_pods' field in the
+				# node util dataframe.
+				for (n in seq(length(br$n_pods$Result))) {
+					nu[[n]]$n_pods = br$n_pods$Result[[n]]
+				}
+
+				# node_util is a list of nested data frames. I'm sure there is some better R'ish
+				# way of extracting this data maybe with dplyr, map, select or melt, but I can't
+				# work it out right now, and at least this is semi-readable...
+				#
+				# Basically, we are de-listing and flattening the lists of dataframes into a
+				# singly 'tidy' dataframe...
+				nodes=do.call("rbind", lapply(nu, "[", "node"))
+				noschedule=do.call("rbind", lapply(nu, "[", "noschedule"))
+				n_pods=do.call("rbind", lapply(nu, "[", "n_pods"))
+				idle=lapply(nu, "[", "cpu_idle")
+				idle_df=do.call("rbind", lapply(idle, "[[", "cpu_idle"))
+				free=lapply(nu, "[", "mem_free")
+				free_df=do.call("rbind", lapply(free, "[[", "mem_free"))
+				used=lapply(nu, "[", "mem_used")
+				used_df=do.call("rbind", lapply(used, "[[", "mem_used"))
+				ifree=lapply(nu, "[", "inode_free")
+				ifree_df=do.call("rbind", lapply(ifree, "[[", "inode_free"))
+				iused=lapply(nu, "[", "inode_used")
+				iused_df=do.call("rbind", lapply(iused, "[[", "inode_used"))
+
+
+				# and build our rows
+				local_nodedata=tibble(node=nodes$node)
+				local_nodedata=cbind(local_nodedata, n_pods=n_pods)
+				local_nodedata=cbind(local_nodedata, noschedule=noschedule)
+				local_nodedata=cbind(local_nodedata, idle=idle_df$Result)
+				local_nodedata=cbind(local_nodedata, mem_free=free_df$Result)
+				local_nodedata=cbind(local_nodedata, mem_used=used_df$Result)
+				local_nodedata=cbind(local_nodedata, inode_free=ifree_df$Result)
+				local_nodedata=cbind(local_nodedata, inode_used=iused_df$Result)
+				local_nodedata=cbind(local_nodedata, testname=rep(testname, length(local_nodedata$node)))
+
+				# Now Calculate some stats. This gets more complicated as we may have n-nodes,
+				# and we want to show a 'pod average', so we try to assess for all nodes. If
+				# we have different 'size' nodes in a cluster, that could throw out the result,
+				# but the only other option would be to try and show every node separately in the
+				# table.
+
+				# Get a list of all the nodes
+				nodes=unique(local_nodedata$node)
+
+				memtotal=0
+				cputotal=0
+				inodetotal=0
+				# Calculate per-node totals, and tot them up to a global total.
+				for (n in nodes) {
+					# Make a frame with just that nodes data in
+					thisnode=subset(local_nodedata, node %in% c(n))
+
+					# Do not use the master (non-schedulable) nodes to calculate
+					# launched pod metrics
+					if(thisnode[1,]$noschedule == "true") {
+						next
+					}
+					memtotal = memtotal + thisnode[nrow(thisnode),]$mem_used
+					cpuused = thisnode[1,]$idle - thisnode[nrow(thisnode),]$idle
+					cputotal = cputotal + cpuused
+					inodetotal = inodetotal + thisnode[nrow(thisnode),]$inode_used
+				}
+
+				num_pods = local_bootdata$n_pods[length(local_bootdata$n_pods)]
+				# We get data in Kb, but want the graphs in Gb.
+				memtotal = memtotal / (1024*1024)
+				gb_per_pod = memtotal/num_pods
+				pod_per_gb = 1/gb_per_pod
+
+				# Memory usage stats.
+				local_mems = c(
+					"Test"=testname,
+					"n"=num_pods,
+					"Tot_Gb"=round(memtotal, 3),
+					"avg_Gb"=round(gb_per_pod, 4),
+					"n_per_Gb"=round(pod_per_gb, 2) 
+				)
+				memstats=rbind(memstats, local_mems)
+
+				# cpu usage stats
+				local_cpus = c(
+					"Test"=testname,
+					"n"=num_pods,
+					"Tot_CPU"=round(cputotal, 3),
+					"avg_CPU"=round(cputotal/num_pods, 4)
+				)
+				cpustats=rbind(cpustats, local_cpus)
+
+				# launch (boot) stats
+				local_boots = c(
+					"Test"=testname,
+					"n"=num_pods,
+					"median"=median(na.omit(local_bootdata)$launch_time)/1000,
+					"min"=min(na.omit(local_bootdata)$launch_time)/1000,
+					"max"=max(na.omit(local_bootdata)$launch_time)/1000,
+					"sd"=round(sd(na.omit(local_bootdata)$launch_time)/1000, 4)
+					)
+
+				bootstats=rbind(bootstats, local_boots)
+
+				# inode stats
+				local_inodes = c(
+					"Test"=testname,
+					"n"=num_pods,
+					"Tot_inode"=round(inodetotal, 3),
+					"avg_inode"=round(inodetotal/num_pods, 4)
+					)
+				inodestats=rbind(inodestats, local_inodes)
+
+				# And collect up our rows into our global table of all results
+				# These two tables *should* be the source of all the data we need to
+				# process and plot (apart from the stats....)
+				bootdata=rbind(bootdata, local_bootdata, make.row.names=FALSE)
+				nodedata=rbind(nodedata, local_nodedata, make.row.names=FALSE)
 			}
-
-			num_pods = local_bootdata$n_pods[length(local_bootdata$n_pods)]
-			# We get data in Kb, but want the graphs in Gb.
-			memtotal = memtotal / (1024*1024)
-			gb_per_pod = memtotal/num_pods
-			pod_per_gb = 1/gb_per_pod
-
-			# Memory usage stats.
-			local_mems = c(
-				"Test"=testname,
-				"n"=num_pods,
-				"Tot_Gb"=round(memtotal, 3),
-				"avg_Gb"=round(gb_per_pod, 4),
-				"n_per_Gb"=round(pod_per_gb, 2) 
-			)
-			memstats=rbind(memstats, local_mems)
-
-			# cpu usage stats
-			local_cpus = c(
-				"Test"=testname,
-				"n"=num_pods,
-				"Tot_CPU"=round(cputotal, 3),
-				"avg_CPU"=round(cputotal/num_pods, 4)
-			)
-			cpustats=rbind(cpustats, local_cpus)
-
-			# launch (boot) stats
-			local_boots = c(
-				"Test"=testname,
-				"n"=num_pods,
-				"median"=median(na.omit(local_bootdata)$launch_time)/1000,
-				"min"=min(na.omit(local_bootdata)$launch_time)/1000,
-				"max"=max(na.omit(local_bootdata)$launch_time)/1000,
-				"sd"=round(sd(na.omit(local_bootdata)$launch_time)/1000, 4)
-				)
-
-			bootstats=rbind(bootstats, local_boots)
-
-			# inode stats
-			local_inodes = c(
-				"Test"=testname,
-				"n"=num_pods,
-				"Tot_inode"=round(inodetotal, 3),
-				"avg_inode"=round(inodetotal/num_pods, 4)
-				)
-			inodestats=rbind(inodestats, local_inodes)
-
-			# And collect up our rows into our global table of all results
-			# These two tables *should* be the source of all the data we need to
-			# process and plot (apart from the stats....)
-			bootdata=rbind(bootdata, local_bootdata, make.row.names=FALSE)
-			nodedata=rbind(nodedata, local_nodedata, make.row.names=FALSE)
 		}
 	}
+
+	# Check if we got any stats at all by checking the memstats data. If we found no data,
+	# abort early and nicely
+	if ( length(memstats) == 0 ) {
+		cat("No results files found for scaling tests\n\n")
+		return()
+	}
+
+	# It's nice to show the graphs in Gb, at least for any decent sized test
+	# run, so make a new column with that pre-divided data in it for us to use.
+	nodedata$mem_free_gb = nodedata$mem_free/(1024*1024)
+	nodedata$mem_used_gb = nodedata$mem_used/(1024*1024)
+	# And show the boot times in seconds, not mS
+	bootdata$launch_time_s = bootdata$launch_time/1000
+
+	# The labels get messed up by us using an 'if' in the aes() - correct it by
+	# using the same 'if' to assign what we really want to use for the labels.
+	colour_label=(if(length(resultdirs)> 1) "testname" else "node")
+
+
+	########## Output memory page ##############
+	mem_stats_plot = suppressWarnings(ggtexttable(data.frame(memstats),
+		theme=ttheme(base_size=10),
+		rows=NULL
+		))
+
+	mem_line_plot <- ggplot(data=nodedata, aes(n_pods,
+			mem_free_gb,
+			colour=(if (length(resultdirs) > 1) testname else node),
+			group=interaction(testname, node))) +
+		labs(colour=colour_label) +
+		geom_line(alpha=0.2) +
+		geom_point(aes(shape=node), alpha=0.3, size=0.5) +
+		xlab("pods") +
+		ylab("System Avail (Gb)") +
+		scale_y_continuous(labels=comma) +
+		ggtitle("System Memory free") +
+		theme(legend.position="bottom") +
+		theme(axis.text.x=element_text(angle=90))
+
+	page1 = grid.arrange(
+		mem_line_plot,
+		mem_stats_plot,
+		ncol=1
+		)
+
+	# pagebreak, as the graphs overflow the page otherwise
+	cat("\n\n\\pagebreak\n")
+
+	########## Output cpu page ##############
+	cpu_stats_plot = suppressWarnings(ggtexttable(data.frame(cpustats),
+		theme=ttheme(base_size=10),
+		rows=NULL
+		))
+
+	cpu_line_plot <- ggplot(data=nodedata, aes(n_pods,
+			idle,
+			colour=(if (length(resultdirs) > 1) testname else node),
+			group=interaction(testname, node))) +
+		labs(colour=colour_label) +
+		geom_line(alpha=0.2) +
+		geom_point(aes(shape=node), alpha=0.3, size=0.5) +
+		xlab("pods") +
+		ylab("System CPU Idle (%)") +
+		ggtitle("System CPU usage") +
+		theme(legend.position="bottom") +
+		theme(axis.text.x=element_text(angle=90))
+
+	page2 = grid.arrange(
+		cpu_line_plot,
+		cpu_stats_plot,
+		ncol=1
+		)
+
+	# pagebreak, as the graphs overflow the page otherwise
+	cat("\n\n\\pagebreak\n")
+
+	########## Output boot page ##############
+	boot_stats_plot = suppressWarnings(ggtexttable(data.frame(bootstats),
+		theme=ttheme(base_size=10),
+		rows=NULL
+		))
+
+	boot_line_plot <- ggplot() +
+		geom_line( data=bootdata, aes(n_pods, launch_time_s, colour=testname, group=testname), alpha=0.2) +
+		geom_point( data=bootdata, aes(n_pods, launch_time_s, colour=interaction(testname, node), group=testname), alpha=0.6, size=0.6, stroke=0, shape=16) +
+		xlab("pods") +
+		ylab("Boot time (s)") +
+		ggtitle("Pod boot time") +
+		theme(legend.position="bottom") +
+		theme(axis.text.x=element_text(angle=90))
+
+	page3 = grid.arrange(
+		boot_line_plot,
+		boot_stats_plot,
+		ncol=1
+		)
+
+	# pagebreak, as the graphs overflow the page otherwise
+	cat("\n\n\\pagebreak\n")
+
+	########## Output inode page ##############
+	inode_stats_plot = suppressWarnings(ggtexttable(data.frame(inodestats),
+		theme=ttheme(base_size=10),
+		rows=NULL
+		))
+
+	inode_line_plot <- ggplot(data=nodedata, aes(n_pods,
+			inode_free,
+			colour=(if (length(resultdirs) > 1) testname else node),
+			group=interaction(testname, node))) +
+		labs(colour=colour_label) +
+		geom_line(alpha=0.2) +
+		geom_point(aes(shape=node), alpha=0.3, size=0.5) +
+		xlab("pods") +
+		ylab("inodes free") +
+		scale_y_continuous(labels=comma) +
+		ggtitle("inodes free") +
+		theme(legend.position="bottom") +
+		theme(axis.text.x=element_text(angle=90))
+
+	page4 = grid.arrange(
+		inode_line_plot,
+		inode_stats_plot,
+		ncol=1
+		)
 }

-# It's nice to show the graphs in Gb, at least for any decent sized test
-# run, so make a new column with that pre-divided data in it for us to use.
-nodedata$mem_free_gb = nodedata$mem_free/(1024*1024)
-nodedata$mem_used_gb = nodedata$mem_used/(1024*1024)
-# And show the boot times in seconds, not mS
-bootdata$launch_time_s = bootdata$launch_time/1000
-
-# The labels get messed up by us using an 'if' in the aes() - correct it by
-# using the same 'if' to assign what we really want to use for the labels.
-colour_label=(if(length(resultdirs)> 1) "testname" else "node")
-
-
-########## Output memory page ##############
-mem_stats_plot = suppressWarnings(ggtexttable(data.frame(memstats),
-	theme=ttheme(base_size=10),
-	rows=NULL
-	))
-
-mem_line_plot <- ggplot(data=nodedata, aes(n_pods,
-		mem_free_gb,
-		colour=(if (length(resultdirs) > 1) testname else node),
-		group=interaction(testname, node))) +
-	labs(colour=colour_label) +
-	geom_line(alpha=0.2) +
-	geom_point(aes(shape=node), alpha=0.3, size=0.5) +
-	xlab("pods") +
-	ylab("System Avail (Gb)") +
-	scale_y_continuous(labels=comma) +
-	ggtitle("System Memory free") +
-	theme(legend.position="bottom") +
-	theme(axis.text.x=element_text(angle=90))
-
-page1 = grid.arrange(
-	mem_line_plot,
-	mem_stats_plot,
-	ncol=1
-	)
-
-# pagebreak, as the graphs overflow the page otherwise
-cat("\n\n\\pagebreak\n")
-
-########## Output cpu page ##############
-cpu_stats_plot = suppressWarnings(ggtexttable(data.frame(cpustats),
-	theme=ttheme(base_size=10),
-	rows=NULL
-	))
-
-cpu_line_plot <- ggplot(data=nodedata, aes(n_pods,
-		idle,
-		colour=(if (length(resultdirs) > 1) testname else node),
-		group=interaction(testname, node))) +
-	labs(colour=colour_label) +
-	geom_line(alpha=0.2) +
-	geom_point(aes(shape=node), alpha=0.3, size=0.5) +
-	xlab("pods") +
-	ylab("System CPU Idle (%)") +
-	ggtitle("System CPU usage") +
-	theme(legend.position="bottom") +
-	theme(axis.text.x=element_text(angle=90))
-
-page2 = grid.arrange(
-	cpu_line_plot,
-	cpu_stats_plot,
-	ncol=1
-	)
-
-# pagebreak, as the graphs overflow the page otherwise
-cat("\n\n\\pagebreak\n")
-
-########## Output boot page ##############
-boot_stats_plot = suppressWarnings(ggtexttable(data.frame(bootstats),
-	theme=ttheme(base_size=10),
-	rows=NULL
-	))
-
-boot_line_plot <- ggplot() +
-	geom_line( data=bootdata, aes(n_pods, launch_time_s, colour=testname, group=testname), alpha=0.2) +
-	geom_point( data=bootdata, aes(n_pods, launch_time_s, colour=interaction(testname, node), group=testname), alpha=0.6, size=0.6, stroke=0, shape=16) +
-	xlab("pods") +
-	ylab("Boot time (s)") +
-	ggtitle("Pod boot time") +
-	theme(legend.position="bottom") +
-	theme(axis.text.x=element_text(angle=90))
-
-page3 = grid.arrange(
-	boot_line_plot,
-	boot_stats_plot,
-	ncol=1
-	)
-
-# pagebreak, as the graphs overflow the page otherwise
-cat("\n\n\\pagebreak\n")
-
-########## Output inode page ##############
-inode_stats_plot = suppressWarnings(ggtexttable(data.frame(inodestats),
-	theme=ttheme(base_size=10),
-	rows=NULL
-	))
-
-inode_line_plot <- ggplot(data=nodedata, aes(n_pods,
-		inode_free,
-		colour=(if (length(resultdirs) > 1) testname else node),
-		group=interaction(testname, node))) +
-	labs(colour=colour_label) +
-	geom_line(alpha=0.2) +
-	geom_point(aes(shape=node), alpha=0.3, size=0.5) +
-	xlab("pods") +
-	ylab("inodes free") +
-	scale_y_continuous(labels=comma) +
-	ggtitle("inodes free") +
-	theme(legend.position="bottom") +
-	theme(axis.text.x=element_text(angle=90))
-
-page4 = grid.arrange(
-	inode_line_plot,
-	inode_stats_plot,
-	ncol=1
-	)
+render_tidy_scaling()
@@ -0,0 +1,118 @@
+# Scaling metrics tests
+
+This directory contains a number of scripts to perform a variety of system scaling tests.
+
+The tests are described in their individual sections below.
+
+Each test has a number of configurable options. Many of those options are common across all tests.
+Those options are detailed in their own section below.
+
+> **Note:** `k8s_scale_rapid.sh` is the most complete and upto date test. It is the only test to
+> currently use the `collectd` data collection method. Other tests use a privileged container to
+> gather statistics.
+>
+> If you find one of the other tests useful, please consider updating it and the corresponding report
+> generation code to use the `collectd` method and send a Pull Request with your updates to this codebase.
+
+## Global test configuration options
+
+The following variables are settable for many of the tests. Check each individual tests help
+for specifics and their individual default values.
+
+| Variable | Default Value | Description |
+| -------- | ------------- | ----------- |
+| TEST_NAME | test dependant | Can be set to over-ride the default JSON results filename |
+| NUM_PODS | 20 | Number of pods to launch |
+| STEP | 1 | Number of pods to launch per cycle |
+| wait_time | 30 | Seconds to wait for pods to become ready |
+| delete_wait_time | 600 | Seconds to wait for all pods to be deleted |
+| settle_time | 5 | Seconds to wait after pods ready before taking measurements |
+| use_api | yes | specify yes or no to use the JSON API to launch pods (otherwise, launch via YAML) |
+| grace | 30 | specify the grace period in seconds for workload pod termination |
+| RUNTIME | unset | specify the `RuntimeClass` to use to launch the pods |
+
+## k8s_parallel.sh
+
+Measures pod create and delete times whilst increasing the number of pods launched in parallel.
+
+The test works by creating and destroying deployments with the required number of replicas being scaled.
+
+## k8s_scale_nc.sh
+
+Measures pod response time using `nc` to test network connection response. Stores results as percentile
+values. Is used to see if the response time latency and jitter is affected by scaling the number of pods.
+
+## k8s_scale_net.sh
+
+Measures pod response time to a `curl` HTTP get request from the K8S e2e `agnhost` image.
+Used to measure if the 'ready to respond' time scales with the number of service ports in use.
+
+## k8s_scale_rapid.sh
+
+Measures how pod launch and the k8s system scales whilst launching more and more pods.
+
+Uses the `collectd` method to gather a number of statistics, including:
+
+- cpu usage
+- memory usage
+- network connections
+- disk usage
+- ipc stats
+
+## k8s_scale.sh
+
+The fore-runner to `k8s_scale_rapid.sh`, using the privileged pod method to gather statistics. It is recommended
+to use `k8s_scale_rapid.sh` in preference if possible.
+
+# Example
+
+Below is a brief example of running the `k8s_scale_rapid.sh` test and generating a report from the results.
+
+1. Run the test
+
+    The test will run against the default `kubectl` configured cluster.
+    ```sh
+    $ ./scaling/k8s_scale.sh
+    ```
+
+    Results are stored in the `results` directory. The results will comprise of one `JSON` file for the test, and
+    one `.tar.gz` file for each node found in the cluster.
+
+    > **Note:** Only the `collectd` based tests generate `.tar.gz` files. All other tests only generate a single
+    > `JSON` file for each run.
+
+1. Move the results files
+
+    In order to generate the report, the results files should be moved into an appropriately named sub-directory.
+    The report generator can process and compare multiple sets of results. Each set of results should be placed
+    into its own sub-directory. The below example uses the name `run1` as an example:
+
+    ```sh
+    $ cd results
+    $ mkdir run1
+    $ mv *.json run1
+    $ mv *.tar.gz run1
+    ```
+
+    This sequence can be repeated to gather multiple test data sets. Place each data set in its own subdirectory.
+    The report generator will process and compare all data set subdirectories found in the `results` directory.
+
+1. Generate the report
+
+    The report generator in the `report` subdirectory processes the sub-directories of the `results` directory
+    to produce a `PDF` report and individual `PNG` based graphs.. The report generator utilises `docker` to create
+    a docker image containing all the tooling necessary.
+
+    ```sh
+    $ cd report
+    $ ./makereport.sh
+    ...
+    $ tree output
+    output/
+    ├── dut-1.png
+    ├── metrics_report.pdf
+    ├── scaling-1.png
+    ├── scaling-2.png
+    ├── scaling-3.png
+    └── scaling-4.png
+    ```
@@ -156,17 +156,17 @@ run() {
 		kubectl rollout status --timeout=${wait_time}s deployment/${deployment}
 		kubectl expose --port=8080 deployment $deployment

-        # Check service exposed
-        cmd="kubectl get services $deployment -n default --no-headers=true"
-        waitForProcess "$proc_wait_time" "$proc_sleep_time" "$cmd" "Waiting for service"
+		# Check service exposed
+		cmd="kubectl get services $deployment -n default --no-headers=true"
+		waitForProcess "$proc_wait_time" "$proc_sleep_time" "$cmd" "Waiting for service"

 		IP=$(kubectl get services $deployment -n default --no-headers=true | awk '{printf $3}')
 		end_net=$(date +%s%N)
 		info "IP: $IP"

 		# service health check
-        cmd="curl --noproxy \"*\" http://$IP:8080/healthz"
-        waitForProcess "$proc_wait_time" "$proc_sleep_time" "$cmd" "http server is not ready yet!!"
+		cmd="curl --noproxy \"*\" http://$IP:8080/healthz"
+		waitForProcess "$proc_wait_time" "$proc_sleep_time" "$cmd" "http server is not ready yet!!"

 		RESP=$(curl -s --noproxy "*" http://$IP:8080/echo?msg=curl%20request%20to%20$deployment)
 		local end_time=$(date +%s%N)
Author	SHA1	Message	Date
Ganesh Maharaj Mahalingam	32f421a2f9	fix canal download URL Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>	2022-05-18 10:57:13 -06:00
Justin Scott	6aff0ac601	Remove static cpu from kubeadm.yaml * Removing static cpu pinning due to confusion and complications it causes with disabling/enabling cores. * Static CPU policy by defaults pins the process to CPUs if a kubernetes pod limits are set to be full cpus. While this a neat thing, this also causes issues when setaffinity is tried in a particular CPU. While we would like to set limits, pinning a process to a core is something we will try to avoid in the near future.	2022-04-04 14:31:15 -07:00
Ganesh Maharaj Mahalingam	0d1d72e7a1	Update rook, kata, metrics-server and kube-prometheus (#340 ) Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>	2022-03-29 16:17:07 -07:00
Ganesh Maharaj Mahalingam	dc39af186f	Update CNIs to versions that work well with 1.20.X of Kubernetes Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>	2022-03-04 12:00:27 -07:00
Ganesh Maharaj Mahalingam	6b56a905a6	Update CNIs Canal: v3.18 Cilium: v1.9 Flannel: 0.14.0-rc1 Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>	2021-05-18 09:08:55 -06:00
Ganesh Maharaj Mahalingam	f9b260088d	Update README to take not of k8s-migration package (#334 ) Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>	2021-03-17 13:26:12 -07:00
Mark Horn	5ca64545b1	Disable swap before OS Update (#333 ) Mask all swap targets. Signed-off-by: Mark D Horn <mark.d.horn@intel.com>	2020-12-17 10:38:43 -08:00
Hyunsun Moon	e74b3ca892	Add support of CNI version 0.3.1 to vfioveth Tested with K8S v1.18.6 and the latest multus stable image.	2020-09-10 15:14:19 -07:00
António Meireles	55b2aa2d19	accommodate upstream libvirt boxes changes in a forward and backward way (#330 ) per https://github.com/AntonioMeireles/ClearLinux-packer/issues/24 the ClearLinux libvirt boxes default size was lowered from the previous 40G to a more manageable 5G, having the user the ability to, at box instantiation, to increase that value to whatever is adequate. In order to avoid surprises in this side the Vagrantfile was modded to have hardcoded the previous default value. This change is forward and backward compatible as it will just be ignored when using older boxes ( since the 'new' root volume size just the same as original one) and will resize the box to the expected 40G in newer ones. Signed-off-by: António Meireles <antonio.meireles@reformi.st>	2020-08-19 14:23:51 -07:00
Ganesh Maharaj Mahalingam	aad050f944	Switch private network for Vagrant VMs (#324 ) Previously used IP range seems to be allocated to NASA and probably not a good idea to use it here. Switching the private network to be part of the IPs allocated to private networks. Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>	2020-06-15 09:40:22 -07:00
David Lyle	a872ccca6c	adding support for HA clusters, using HAProxy (#317 ) * adding support for HA clusters, using HAProxy Signed-off-by: David Lyle <dklyle0@gmail.com> * fixing typo * fixing load balancer port default value	2020-06-15 09:38:34 -07:00
Saikrishna Edupuganti	1525407bd4	Update multi-net components to latest releases (#329 ) * Update multi-net components to latest releases Multus CNI 3.4.2 SR-IOV CNI 2.3 SR-IOV DP 3.2 Tested as per the README. Works fine. Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> * Provide dpdk-stable 3 LTS and 1 recent release Testpmd manifests of the last 3 LTS and 1 latest release from stable repo to help test actual DPDK app instead of sleep. Currently 17.11 and 19.11 are the only functioning ones without privileged. ``` NAME READY STATUS RESTARTS AGE dpdk-1711 1/1 Running 0 3m5s dpdk-1811 0/1 Error 0 3m5s dpdk-1911 1/1 Running 0 3m5s dpdk-2002 0/1 Error 0 3m5s ``` ``` EAL: PCI device 0000:07:06.4 on NUMA socket 0 EAL: probe driver: 8086:154c net_i40e_vf EAL: Getting a vfio_dev_fd for 0000:07:06.4 failed EAL: Requested device 0000:07:06.4 cannot be used … testpmd: No probed ethernet devices EAL: Error - exiting with code: 1 Cause: Invalid port 1 ``` Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com>	2020-06-05 09:06:08 -07:00
Miguel Bernal Marin	9b4f0a8582	setup_system: use local admin path for proxy.conf (#328 ) Currently the system.conf.d/proxy.conf file is saved at /usr/lib which is the vendor path, and can be dropped by "swupd repair --picky --force". This commit creates the local administrator /etc/systemd/system.conf.d directory and store the proxy.conf inside.	2020-05-26 10:45:25 -07:00
Antti Kervinen	696861ce66	metrics: change collectd output to host /opt/collectd/run Currently we loose collectd data from a node when scaling ends to a system failure on the node - yet this data can be very helpful in root causing the failure. This patch changes collectd configuration so that the output will be continuously written to host filesystem instead of the collectd container overlay that will be lost unless scaling reaches graceful exit. Signed-off-by: Antti Kervinen <antti.kervinen@intel.com>	2020-05-19 19:56:54 +01:00
Graham Whaley	07fd8412da	metrics: report: Error more cleanly Clean up the rest of the report R files to allow them to quit cleanly when they find an error or missing data, so that the final PDF report gives meaninful errors such as 'No data found', rather than cryptic R errors. Signed-off-by: Graham Whaley <graham.whaley@intel.com>	2020-05-19 19:53:23 +01:00
Graham Whaley	058e1753ae	metrics: report: quit cleanly on tidy_scaling failure When there are no files to process, we tend to quit with a loud and not helpful error. Improve that by spotting the obvious error cases (such as no files to process for a specific test), and quit with a nicer error/warning message that ends up in the rendered report. Start with the tidy_scaling test. The only clean way to quit a fragment of Rmarkdown R looks to be to place it inside a function so we can 'return'. Otherwise, all other forms of 'quit', quit the whole Rmarkdown render pipeline, which is not what we want - we want to carry on and try to process the rest of the fragments for the rest of the tests. Signed-off-by: Graham Whaley <graham.whaley@intel.com>	2020-05-19 19:53:23 +01:00
Graham Whaley	091e76c3d8	metrics: k8s_scale_net: whitespace fixes Fix some indentation that had gone rogue. Note, there are other whitespace fixes that can be done in this file, it appears to have a mix of tabs and spaces. Signed-off-by: Graham Whaley <graham.whaley@intel.com>	2020-05-15 09:11:37 -06:00
Graham Whaley	7121418dd3	metrics: Imrove documentation Improve and expand the documents across the metrics subsystem. Clarify and re-order some documents. Add some more details around each individual test. Note that only the 'rapid' test is currently actively used, and the other tests may need some nurturing if they are found to be useful. Signed-off-by: Graham Whaley <graham.whaley@intel.com>	2020-05-15 09:11:37 -06:00
Gabriela Cervantes	e10260e99c	metrics: Use a specific version of rocker/tidyverse This PR uses a specific version of rocker/tidyverse as the latest version does not have the latex-xcolor package which makes impossible to create the metrics report. Signed-off-by: Gabriela Cervantes <gabriela.cervantes.tellez@intel.com>	2020-05-13 15:18:05 +01:00
CraigSterrett	e732cc693b	Updated metrics github location (#323 ) The metrics-server package has moved out of the kubernetes incubator github location and is now in the sigs location. Signed-off-by: Craig Sterrett <craig.Sterrett@intel.com>	2020-05-06 13:00:07 -07:00