fix canal download URL

Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>
Remove static cpu from kubeadm.yaml
2026-08-18 21:16:16 +00:00 · 2022-05-18 10:57:13 -06:00 · 2022-04-04 14:31:15 -07:00 · 2022-03-29 16:17:07 -07:00 · 2022-03-04 12:00:27 -07:00 · 2021-05-18 09:08:55 -06:00
20 changed files with 271 additions and 44 deletions
@@ -0,0 +1,2 @@
+resources:
+  - canal/canal.yaml
@@ -0,0 +1,4 @@
+resources:
+  - canal/canal.yaml
+
+
@@ -0,0 +1,2 @@
+resources:
+        cilium/cilium.yaml
@@ -0,0 +1,5 @@
+ipam:
+  mode: "cluster-pool"
+  operator:
+    clusterPoolIPv4PodCIDR: "10.244.0.0/16"
+    clusterPoolIPv4MaskSize: 24
@@ -0,0 +1,2 @@
+resources:
+        - flannel/Documentation/kube-flannel.yml
@@ -0,0 +1,2 @@
+resources:
+        - flannel/Documentation/kube-flannel.yml
@@ -0,0 +1,8 @@
+resources:
+  - components.yaml
+patchesJson6902:
+  - target:
+      version: v1
+      kind: Deployment
+      name: metrics-server
+    path: patch_metricstls.yaml
@@ -0,0 +1,3 @@
+- op: add
+  path: "/spec/template/spec/containers/0/args/-"
+  value: --kubelet-insecure-tls
@@ -0,0 +1,76 @@
+resources:
+  - kube-prometheus/manifests/kubeStateMetrics-serviceAccount.yaml
+  - kube-prometheus/manifests/blackboxExporter-clusterRole.yaml
+  - kube-prometheus/manifests/nodeExporter-serviceAccount.yaml
+  - kube-prometheus/manifests/prometheus-prometheusRule.yaml
+  - kube-prometheus/manifests/kubernetesControlPlane-serviceMonitorKubeScheduler.yaml
+  - kube-prometheus/manifests/prometheus-serviceMonitor.yaml
+  - kube-prometheus/manifests/grafana-dashboardSources.yaml
+  - kube-prometheus/manifests/kubeStateMetrics-prometheusRule.yaml
+  - kube-prometheus/manifests/kubePrometheus-prometheusRule.yaml
+  - kube-prometheus/manifests/prometheus-clusterRole.yaml
+  - kube-prometheus/manifests/blackboxExporter-serviceMonitor.yaml
+  - kube-prometheus/manifests/prometheus-roleSpecificNamespaces.yaml
+  - kube-prometheus/manifests/alertmanager-service.yaml
+  - kube-prometheus/manifests/prometheusAdapter-serviceMonitor.yaml
+  - kube-prometheus/manifests/nodeExporter-prometheusRule.yaml
+  - kube-prometheus/manifests/nodeExporter-service.yaml
+  - kube-prometheus/manifests/prometheus-roleConfig.yaml
+  - kube-prometheus/manifests/kubeStateMetrics-clusterRole.yaml
+  - kube-prometheus/manifests/prometheusOperator-deployment.yaml
+  - kube-prometheus/manifests/prometheusOperator-serviceMonitor.yaml
+  - kube-prometheus/manifests/prometheusAdapter-deployment.yaml
+  - kube-prometheus/manifests/kubernetesControlPlane-serviceMonitorApiserver.yaml
+  - kube-prometheus/manifests/prometheusAdapter-configMap.yaml
+  - kube-prometheus/manifests/kubernetesControlPlane-prometheusRule.yaml
+  - kube-prometheus/manifests/kubeStateMetrics-deployment.yaml
+  - kube-prometheus/manifests/blackboxExporter-configuration.yaml
+  - kube-prometheus/manifests/kubeStateMetrics-clusterRoleBinding.yaml
+  - kube-prometheus/manifests/blackboxExporter-serviceAccount.yaml
+  - kube-prometheus/manifests/grafana-dashboardDefinitions.yaml
+  - kube-prometheus/manifests/prometheusOperator-service.yaml
+  - kube-prometheus/manifests/grafana-service.yaml
+  - kube-prometheus/manifests/prometheus-prometheus.yaml
+  - kube-prometheus/manifests/kubernetesControlPlane-serviceMonitorKubeControllerManager.yaml
+  - kube-prometheus/manifests/alertmanager-alertmanager.yaml
+  - kube-prometheus/manifests/kubernetesControlPlane-serviceMonitorKubelet.yaml
+  - kube-prometheus/manifests/grafana-dashboardDatasources.yaml
+  - kube-prometheus/manifests/kubernetesControlPlane-serviceMonitorCoreDNS.yaml
+  - kube-prometheus/manifests/alertmanager-serviceMonitor.yaml
+  - kube-prometheus/manifests/grafana-deployment.yaml
+  - kube-prometheus/manifests/grafana-serviceAccount.yaml
+  - kube-prometheus/manifests/alertmanager-serviceAccount.yaml
+  - kube-prometheus/manifests/prometheusAdapter-clusterRoleAggregatedMetricsReader.yaml
+  - kube-prometheus/manifests/prometheusOperator-prometheusRule.yaml
+  - kube-prometheus/manifests/alertmanager-podDisruptionBudget.yaml
+  - kube-prometheus/manifests/prometheus-serviceAccount.yaml
+  - kube-prometheus/manifests/prometheus-service.yaml
+  - kube-prometheus/manifests/prometheusAdapter-clusterRoleServerResources.yaml
+  - kube-prometheus/manifests/prometheusAdapter-clusterRoleBinding.yaml
+  - kube-prometheus/manifests/prometheus-roleBindingConfig.yaml
+  - kube-prometheus/manifests/nodeExporter-daemonset.yaml
+  - kube-prometheus/manifests/prometheus-roleBindingSpecificNamespaces.yaml
+  - kube-prometheus/manifests/nodeExporter-clusterRoleBinding.yaml
+  - kube-prometheus/manifests/prometheusOperator-serviceAccount.yaml
+  - kube-prometheus/manifests/prometheusOperator-clusterRoleBinding.yaml
+  - kube-prometheus/manifests/kubeStateMetrics-serviceMonitor.yaml
+  - kube-prometheus/manifests/prometheusAdapter-roleBindingAuthReader.yaml
+  - kube-prometheus/manifests/prometheusAdapter-clusterRoleBindingDelegator.yaml
+  - kube-prometheus/manifests/prometheusAdapter-serviceAccount.yaml
+  - kube-prometheus/manifests/blackboxExporter-deployment.yaml
+  - kube-prometheus/manifests/alertmanager-prometheusRule.yaml
+  - kube-prometheus/manifests/prometheus-clusterRoleBinding.yaml
+  - kube-prometheus/manifests/prometheusAdapter-clusterRole.yaml
+  - kube-prometheus/manifests/grafana-serviceMonitor.yaml
+  - kube-prometheus/manifests/nodeExporter-clusterRole.yaml
+  - kube-prometheus/manifests/prometheusAdapter-service.yaml
+  - kube-prometheus/manifests/prometheus-podDisruptionBudget.yaml
+  - kube-prometheus/manifests/blackboxExporter-service.yaml
+  - kube-prometheus/manifests/nodeExporter-serviceMonitor.yaml
+  - kube-prometheus/manifests/blackboxExporter-clusterRoleBinding.yaml
+  - kube-prometheus/manifests/alertmanager-secret.yaml
+  - kube-prometheus/manifests/prometheusAdapter-apiService.yaml
+  - kube-prometheus/manifests/prometheusOperator-clusterRole.yaml
+  - kube-prometheus/manifests/grafana-config.yaml
+  - kube-prometheus/manifests/prometheusAdapter-podDisruptionBudget.yaml
+  - kube-prometheus/manifests/kubeStateMetrics-service.yaml
@@ -0,0 +1,3 @@
+resources:
+  - rook/deploy/examples/cluster.yaml
+  - rook/deploy/examples/csi/rbd/storageclass.yaml
@@ -0,0 +1,8 @@
+resources:
+  - rook/deploy/examples/cluster.yaml
+  - rook/deploy/examples/csi/rbd/storageclass.yaml
+
+patchesStrategicMerge:
+  # patches rook to use 'directories' instead of partitions.
+  # comment out to use partitions
+  - patch_cephcluster.yaml
@@ -0,0 +1,19 @@
+---
+apiVersion: ceph.rook.io/v1
+kind: CephCluster
+metadata:
+  name: rook-ceph
+  namespace: rook-ceph
+spec:
+  mon:
+    allowMultiplePerNode: true
+---
+apiVersion: ceph.rook.io/v1
+kind: CephBlockPool
+metadata:
+  name: replicapool
+  namespace: rook-ceph
+spec:
+  replicated:
+    requireSafeReplicaSize: false
+    size: 1
@@ -50,9 +50,9 @@ delete_pair_ns() {
 case $CNI_COMMAND in
 ADD)
 	res=$(ipam)
-	ip=$(echo $res | jq -r '.ip4.ip')
+	ip=$(echo $res | jq -r '.ips[0].address')
 	add_pair_ns $ip
-	echo '{"cniVersion":"0.2.0"}' | jq -c --arg ip $ip '.ip4.ip = $ip' >&3
+	echo '{"cniVersion":"0.3.1"}' | jq -c --arg ip $ip '.ips[0].address = $ip' >&3
 	;;
 DEL)
 	set +o errexit
@@ -3,6 +3,11 @@
 ## Prerequisite
 This setup currently will work with Kubernetes 1.14 & above. Any version of Kubernetes before that might work, but is not guaranteed.

+## QUICK NOTE
+The version of Kubernetes* was bumped from 1.17.7 to 1.19.4 in Clear Linux* OS release 34090. The [guide](https://docs.01.org/clearlinux/latest/guides/clear/k8s-migration.html) and the Clear Linux OS bundle k8s-migration were created to help facilitate migration of a cluster from 1.17.x to the latest 1.19.x .
+
+The new Clear Linux OS bundle k8s-migration was added in Clear Linux* OS release 34270. Please follow the guide for an upgrade.
+
 ## Sample multi-node vagrant setup

 To be able to test this tool, you can create a 3-node vagrant setup. In this tutorial, we will talk about using [libvirt](https://github.com/vagrant-libvirt/vagrant-libvirt), but you can use any hypervisor that you are familiar with.
@@ -63,6 +68,28 @@ devicemapper storage. This should not be used for production.

 > NOTE: This step is done automatically if using vagrant.

+### For HA, setup the load balancer node
+
+Ideally, the load balancer node will be a separate node. However, one of the
+master nodes can also serve as the load balancer for the cluster. [HAProxy](https://www.haproxy.org/)
+is used in these instructions.
+
+```bash
+sudo swupd bundle-add haproxy
+sudo systemctl enable haproxy
+```
+
+Edit the master IP addresses and load balancer address and ports in [`haproxy.cfg.example`](haproxy.cfg.example)
+to match the IPs for the new cluster. If using a master node for the load balancer
+make sure that the `frontend bind` port is different than the Kubernetes API port, 6443.
+If using a separate machine for load balancing, the port can be 6443 if desired.
+
+```bash
+sudo mkdir -p /etc/haproxy
+sudo cp haproxy.cfg.example /etc/haproxy/haproxy.cfg
+sudo systemctl start haproxy
+```
+
 ## Bring up the master

 Run [`create_stack.sh`](create_stack.sh) on the master node. This sets up the
@@ -75,8 +102,9 @@ more information.

 > NOTE: Before running [`create_stack.sh`](create_stack.sh) script, make sure to export
 the necessary environment variables if needed to be changed. By default it will use
-`CLRK8S_CNI` to be canal, and `CLRK8S_RUNNER` to be crio. Cilium is tested only in the 
-Vagrant.
+`CLRK8S_CNI` to be canal, and `CLRK8S_RUNNER` to be crio. Cilium is tested only in the
+Vagrant. If creating an HA cluster, make sure to specify `LOAD_BALANCER_IP` and
+`LOAD_BALANCER_PORT`.

 ```bash
 # default shows help
@@ -86,6 +114,13 @@ Vagrant.
 In order to enable running greater than 110 pods per node, set the environment
 variable `HIGH_POD_COUNT` to any non-empty value.

+If creating an HA cluster, join the other master nodes to the cluster.
+
+```bash
+kubeadm join <load-balancer-ip>:<load-balancer-port> --token <token> --discovery-token-ca-cert-hash <hash> \
+    --control-plane --certificate-key <certificate-key> --cri-socket=/run/crio/crio.sock
+```
+
 ## Join Workers to the cluster

 ```bash
@@ -94,6 +129,9 @@ kubeadm join <master-ip>:<master-port> --token <token> --discovery-token-ca-cert

 Note: Remember to append `--cri-socket=/run/crio/crio.sock` to the join command generated by the master.

+If creating an HA cluster, join the other worker nodes to the cluster. The same way,
+but replacing the `<master-ip>:<master-port>` with `<load-balancer-ip>:<load-balancer-port>`.
+
 On workers just use the join command that the master spits out. There nothing
 else you need to run on the worker. All the other Kubernetes customizations are pushed
 in from master via the values setup in the `kubeadm.yaml` file.
@@ -16,7 +16,7 @@ $box = "AntonioMeireles/ClearLinux"
 $box_ver = (ENV['CLEAR_VBOX_VER'])
 File.exists?("/usr/share/qemu/OVMF.fd") ? $loader = "/usr/share/qemu/OVMF.fd" : $loader = File.join(File.dirname(__FILE__), "OVMF.fd")
 $vm_name_prefix = "clr"
-$base_ip = IPAddr.new("192.52.100.10")
+$base_ip = IPAddr.new("10.10.100.10")
 $hosts = {}
 $proxy_ip_list = "192.168.121.0/24"
 $driveletters = ('a'..'z').to_a
@@ -76,6 +76,7 @@ Vagrant.configure("2") do |config|
        lv.loader = $loader
        lv.cpus = $cpus
        lv.memory = $memory
+        lv.machine_virtual_size = 40
        (1..$disks).each do |d|
          lv.storage :file, :device => "hd#{$driveletters[d]}", :path => "disk-#{$disk_prefix}-#{vm_name}-#{d}.disk", :size => $disk_size, :type => "raw"
        end
@@ -14,21 +14,24 @@ SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
 : ${MASTER_IP:=}
 : ${CERT_SANS:=}
 HIGH_POD_COUNT=${HIGH_POD_COUNT:-""}
+LOAD_BALANCER_IP=${LOAD_BALANCER_IP:-""}
+LOAD_BALANCER_PORT="${LOAD_BALANCER_PORT:-6444}"

 # versions
-CANAL_VER="${CLRK8S_CANAL_VER:-v3.10}"
-CILIUM_VER="${CLRK8S_CILIUM_VER:-v1.6.4}"
-FLANNEL_VER="${CLRK8S_FLANNEL_VER:-960b3243b9a7faccdfe7b3c09097105e68030ea7}"
+CANAL_VER="${CLRK8S_CANAL_VER:-v3.22}"
+CILIUM_VER="${CLRK8S_CILIUM_VER:-v1.9.13}"
+FLANNEL_VER="${CLRK8S_FLANNEL_VER:-v0.16.3}"
+CILIUM_VAL_OVERRIDE=""
 K8S_VER="${CLRK8S_K8S_VER:-}"
-KATA_VER="${CLRK8S_KATA_VER:-1.9.1-kernel-config}"
-ROOK_VER="${CLRK8S_ROOK_VER:-v1.2.6}"
-METRICS_VER="${CLRK8S_METRICS_VER:-v0.3.6}"
+KATA_VER="${CLRK8S_KATA_VER:-2.3.3}"
+ROOK_VER="${CLRK8S_ROOK_VER:-v1.8.6}"
+METRICS_VER="${CLRK8S_METRICS_VER:-v0.6.1}"
 DASHBOARD_VER="${CLRK8S_DASHBOARD_VER:-v2.0.0-beta2}"
 INGRES_VER="${CLRK8S_INGRES_VER:-nginx-0.26.1}"
 EFK_VER="${CLRK8S_EFK_VER:-v1.15.1}"
 METALLB_VER="${CLRK8S_METALLB_VER:-v0.8.3}"
 NPD_VER="${CLRK8S_NPD_VER:-v0.6.6}"
-PROMETHEUS_VER="${CLRK8S_PROMETHEUS_VER:-f458e85e5d7675f7bc253072e1b4c8892b51af0f}"
+PROMETHEUS_VER="${CLRK8S_PROMETHEUS_VER:-v0.10.0}"
 CNI=${CLRK8S_CNI:-"canal"}
 if [[ -z "${RUNNER+x}" ]]; then RUNNER="${CLRK8S_RUNNER:-"crio"}"; fi

@@ -90,7 +93,13 @@ function cluster_init() {
 		echo "/var/lib/etcd exists! skipping init."
 		return
 	fi
-	sudo -E kubeadm init --config=./kubeadm.yaml
+
+	if [[ -n "${LOAD_BALANCER_IP}" ]]; then
+		sed -i "s/ClusterConfiguration/ClusterConfiguration\ncontrolPlaneEndpoint: ${LOAD_BALANCER_IP}:${LOAD_BALANCER_PORT}/g" ./kubeadm.yaml
+	fi
+	# upload-certs will automatically upload certificates that should be shared
+	# across control-plane nodes in HA clusters. It is harmless in non-HA cases.
+	sudo -E kubeadm init --upload-certs --config=./kubeadm.yaml

 	rm -rf "${HOME}/.kube"
 	mkdir -p "${HOME}/.kube"
@@ -115,11 +124,13 @@ function cluster_init() {

 function kata() {
 	KATA_VER=${1:-$KATA_VER}
-	KATA_URL="https://github.com/kata-containers/packaging.git"
+	KATA_URL="https://github.com/kata-containers/kata-containers.git"
 	KATA_DIR="8-kata"
 	get_repo "${KATA_URL}" "${KATA_DIR}/overlays/${KATA_VER}"
-	set_repo_version "${KATA_VER}" "${KATA_DIR}/overlays/${KATA_VER}/packaging"
-	kubectl apply -k "${KATA_DIR}/overlays/${KATA_VER}"
+	set_repo_version "${KATA_VER}" "${KATA_DIR}/overlays/${KATA_VER}/kata-containers"
+	kubectl apply -f "${KATA_DIR}/overlays/${KATA_VER}/kata-containers/tools/packaging/kata-deploy/kata-rbac/base/kata-rbac.yaml"
+	kubectl apply -f "${KATA_DIR}/overlays/${KATA_VER}/kata-containers/tools/packaging/kata-deploy/kata-deploy/base/kata-deploy.yaml"
+	kubectl apply -f "${KATA_DIR}/overlays/${KATA_VER}/kata-containers/tools/packaging/kata-deploy/runtimeclasses/kata-runtimeClasses.yaml"

 }

@@ -128,7 +139,7 @@ function cni() {
 	canal)
 		# note version is not semver
 		CANAL_VER=${1:-$CANAL_VER}
-		CANAL_URL="https://docs.projectcalico.org/${CANAL_VER}/manifests"
+		CANAL_URL="https://projectcalico.docs.tigera.io/archive/${CANAL_VER}/manifests"
 		if [[ "$CANAL_VER" == "v3.3" ]]; then
 			CANAL_URL="https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/canal"
 		fi
@@ -136,7 +147,7 @@ function cni() {

 		# canal manifests are not kept in repo but in docs site so use curl
 		mkdir -p "${CANAL_DIR}/overlays/${CANAL_VER}/canal"
-		curl -o "${CANAL_DIR}/overlays/${CANAL_VER}/canal/canal.yaml" "$CANAL_URL/canal.yaml"
+		curl -L -o "${CANAL_DIR}/overlays/${CANAL_VER}/canal/canal.yaml" "$CANAL_URL/canal.yaml"
 		if [[ "$CANAL_VER" == "v3.3" ]]; then
 			curl -o "${CANAL_DIR}/overlays/${CANAL_VER}/canal/rbac.yaml" "$CANAL_URL/rbac.yaml"
 		fi
@@ -145,7 +156,7 @@ function cni() {
 		;;
 	flannel)
 		FLANNEL_VER=${1:-$FLANNEL_VER}
-		FLANNEL_URL="https://github.com/coreos/flannel"
+		FLANNEL_URL="https://github.com/flannel-io/flannel"
 		FLANNEL_DIR="0-flannel"

 		get_repo "${FLANNEL_URL}" "${FLANNEL_DIR}/overlays/${FLANNEL_VER}"
@@ -153,13 +164,17 @@ function cni() {
 		kubectl apply -k "${FLANNEL_DIR}/overlays/${FLANNEL_VER}"
 		;;
 	cilium)
+		local podsubnet=$(grep -Po 'podSubnet:\ \K[^*]*' ${SCRIPT_DIR}/kubeadm.yaml)
 		CILIUM_VER=${1:-$CILIUM_VER}
 		CILIUM_URL="https://github.com/cilium/cilium.git"
 		CILIUM_DIR="0-cilium"

 		get_repo "${CILIUM_URL}" "${CILIUM_DIR}/overlays/${CILIUM_VER}"
 		set_repo_version "${CILIUM_VER}" "${CILIUM_DIR}/overlays/${CILIUM_VER}/cilium/"
-		helm template "${CILIUM_DIR}/overlays/${CILIUM_VER}/cilium/install/kubernetes/cilium" --namespace kube-system --set global.containerRuntime.integration="$RUNNER" | kubectl apply -f -
+		if [ -f "${CILIUM_DIR}/overlays/${CILIUM_VER}/values.yaml" ]; then
+			CILIUM_VAL_OVERRIDE="--values ${CILIUM_DIR}/overlays/${CILIUM_VER}/values.yaml"
+		fi
+		helm template "${CILIUM_DIR}/overlays/${CILIUM_VER}/cilium/install/kubernetes/cilium" --namespace kube-system --set containerRuntime.integration="$RUNNER" --set hubble.enabled=false --set ipam.operator.clusterPoolIPv4PodCIDR="${podsubnet}" | kubectl apply -f -
 		;;
 	*)
 		echo"Unknown cni $CNI"
@@ -170,21 +185,19 @@ function cni() {

 function metrics() {
 	METRICS_VER="${1:-$METRICS_VER}"
-	METRICS_URL="https://github.com/kubernetes-sigs/metrics-server.git"
+	METRICS_URL="https://github.com/kubernetes-sigs/metrics-server/releases/download/${METRICS_VER}/components.yaml"
 	METRICS_DIR="1-core-metrics"
-	get_repo "${METRICS_URL}" "${METRICS_DIR}/overlays/${METRICS_VER}"
-	set_repo_version "${METRICS_VER}" "${METRICS_DIR}/overlays/${METRICS_VER}/metrics-server"
-	kubectl apply -k "${METRICS_DIR}/overlays/${METRICS_VER}"
+
+	curl -L ${METRICS_URL} --output - >${METRICS_DIR}/overlays/${METRICS_VER}/components.yaml
+	kubectl apply -k "${METRICS_DIR}/overlays/${METRICS_VER}/"

 }
 function wait_on_pvc() {
 	# create and destroy pvc until successful
 	while [[ $(kubectl get pvc test-pv-claim --no-headers | grep Bound -c) -ne 1 ]]; do
 		sleep 30
-		kubectl delete pvc test-pv-claim
-		create_pvc
-		sleep 10
 	done
+	kubectl delete pvc test-pv-claim
 }
 function create_pvc() {
 	kubectl apply -f - <<HERE
@@ -209,9 +222,19 @@ function storage() {
 	ROOK_URL="https://github.com/rook/rook.git"
 	ROOK_DIR=7-rook

+  # This function might be called standalone, so good to check the mode we are in.
+	if [ "$(kubectl get nodes --no-headers | wc -l)" -eq 1 ]; then
+		mode="standalone"
+	fi
+
 	# get and apply rook
 	get_repo "${ROOK_URL}" "${ROOK_DIR}/overlays/${ROOK_VER}/${mode}"
 	set_repo_version "${ROOK_VER}" "${ROOK_DIR}/overlays/${ROOK_VER}/${mode}/rook"
+	kubectl apply -f ${ROOK_DIR}/overlays/${ROOK_VER}/${mode}/rook/deploy/examples/crds.yaml -f ${ROOK_DIR}/overlays/${ROOK_VER}/${mode}/rook/deploy/examples/common.yaml -f ${ROOK_DIR}/overlays/${ROOK_VER}/${mode}/rook/deploy/examples/operator.yaml
+	while [[ $(kubectl get po -n rook-ceph --field-selector=status.phase=Running | grep -e 'operator' -c) -lt 1 ]]; do
+		echo "Waiting on operator"
+		sleep 10
+	done
 	kubectl apply -k "${ROOK_DIR}/overlays/${ROOK_VER}/${mode}"
 	# wait for the rook OSDs to run which means rooks should be ready
 	while [[ $(kubectl get po --all-namespaces | grep -e 'osd.*Running.*' -c) -lt 1 ]]; do
@@ -233,12 +256,13 @@ function monitoring() {
 	PROMETHEUS_DIR="4-kube-prometheus"
 	get_repo "${PROMETHEUS_URL}" "${PROMETHEUS_DIR}/overlays/${PROMETHEUS_VER}"
 	set_repo_version "${PROMETHEUS_VER}" "${PROMETHEUS_DIR}/overlays/${PROMETHEUS_VER}/kube-prometheus"
-	kubectl apply -k "${PROMETHEUS_DIR}/overlays/${PROMETHEUS_VER}"
+	kubectl apply --server-side -f "${PROMETHEUS_DIR}/overlays/${PROMETHEUS_VER}/kube-prometheus/manifests/setup/"

-	while [[ $(kubectl get crd alertmanagers.monitoring.coreos.com prometheuses.monitoring.coreos.com prometheusrules.monitoring.coreos.com servicemonitors.monitoring.coreos.com >/dev/null 2>&1) || $? -ne 0 ]]; do
-		echo "Waiting for Prometheus CRDs"
-		sleep 2
+	while ! $(kubectl get servicemonitors --all-namespaces) ; do
+		echo "Waiting for prometheus crds"
+		sleep 10
 	done
+	kubectl apply -k "${PROMETHEUS_DIR}/overlays/${PROMETHEUS_VER}/"

 	#Expose the dashboards
 	#kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090 &
@@ -369,6 +393,8 @@ command_handlers[monitoring]=monitoring
 command_handlers[metallb]=metallb
 command_handlers[npd]=npd
 command_handlers[nfd]=nfd
+command_handlers[kata]=kata
+command_handlers[metrics]=metrics

 declare -A command_help
 command_help[init]="Only inits a cluster using kubeadm"
@@ -0,0 +1,34 @@
+global
+    log /dev/log    local0
+    chroot /var/lib/haproxy
+    stats socket /run/haproxy-master.sock mode 660 level admin
+    stats timeout 30s
+    user haproxy
+    group haproxy
+    daemon
+    # Default SSL material locations
+    ca-base /etc/ssl/certs
+    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
+    ssl-default-bind-options no-sslv3
+defaults
+    log     global
+    mode    http
+    option  httplog
+    option  dontlognull
+    timeout connect 5000
+    timeout client  50000
+    timeout server  50000
+    timeout tunnel  4h
+frontend kubernetes
+    bind 10.0.0.100:6444
+    option tcplog
+    mode tcp
+    default_backend kubernetes-master-nodes
+
+backend kubernetes-master-nodes
+    mode tcp
+    balance source
+    option tcp-check
+    server master-1 10.0.0.100:6443 check fall 3 rise 2
+    server master-2 10.0.0.101:6443 check fall 3 rise 2
+    server master-3 10.0.0.102:6443 check fall 3 rise 2
@@ -4,8 +4,6 @@ kind: InitConfiguration
 apiVersion: kubelet.config.k8s.io/v1beta1
 kind: KubeletConfiguration
 cgroupDriver: systemd
-# Allowing for CPU pinning and isolation in case of guaranteed QoS class
-cpuManagerPolicy: static
 systemReserved:
  cpu: 500m
  memory: 256M
@@ -29,13 +29,10 @@ function add_os_deps() {

 # permanently disable swap
 function disable_swap() {
-	swapcount=$(sudo grep '^/dev/\([0-9a-z]*\).*' /proc/swaps | wc -l)
-
-	if [ "$swapcount" != "0" ]; then
-		sudo systemctl mask "$(sed -n -e 's#^/dev/\([0-9a-z]*\).*#dev-\1.swap#p' /proc/swaps)" 2>/dev/null
-	else
-		echo "Swap not enabled"
-	fi
+	# disable current swap
+	sudo swapoff -a
+	# permanently disable swap
+	sudo systemctl mask swap.target
 }

 # enable ip forwarding
@@ -137,7 +134,6 @@ function enable_kubelet_runner() {

 # ensure that the system is ready without requiring a reboot
 function ensure_system_ready() {
-	sudo swapoff -a
 	sudo systemctl restart systemd-modules-load.service
 }

@@ -192,12 +188,12 @@ fi

 echo "Init..."
 init
+echo "Disabling swap..."
+disable_swap
 echo "Setting OS Version..."
 upate_os_version
 echo "Adding OS Dependencies..."
 add_os_deps
-echo "Disabling swap..."
-disable_swap
 echo "Enabling IP Forwarding..."
 enable_ip_forwarding
 echo "Setting up modules to load..."
@@ -210,7 +206,7 @@ if [[ -n "${HIGH_POD_COUNT}" ]]; then
 fi
 echo "Reloading daemons..."
 daemon_reload
-echo "Enabling Kublet runner..."
+echo "Enabling kubelet runner..."
 enable_kubelet_runner
 echo "Ensuring system is ready..."
 ensure_system_ready
Author	SHA1	Message	Date
Ganesh Maharaj Mahalingam	32f421a2f9	fix canal download URL Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>	2022-05-18 10:57:13 -06:00
Justin Scott	6aff0ac601	Remove static cpu from kubeadm.yaml * Removing static cpu pinning due to confusion and complications it causes with disabling/enabling cores. * Static CPU policy by defaults pins the process to CPUs if a kubernetes pod limits are set to be full cpus. While this a neat thing, this also causes issues when setaffinity is tried in a particular CPU. While we would like to set limits, pinning a process to a core is something we will try to avoid in the near future.	2022-04-04 14:31:15 -07:00
Ganesh Maharaj Mahalingam	0d1d72e7a1	Update rook, kata, metrics-server and kube-prometheus (#340 ) Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>	2022-03-29 16:17:07 -07:00
Ganesh Maharaj Mahalingam	dc39af186f	Update CNIs to versions that work well with 1.20.X of Kubernetes Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>	2022-03-04 12:00:27 -07:00
Ganesh Maharaj Mahalingam	6b56a905a6	Update CNIs Canal: v3.18 Cilium: v1.9 Flannel: 0.14.0-rc1 Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>	2021-05-18 09:08:55 -06:00
Ganesh Maharaj Mahalingam	f9b260088d	Update README to take not of k8s-migration package (#334 ) Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>	2021-03-17 13:26:12 -07:00
Mark Horn	5ca64545b1	Disable swap before OS Update (#333 ) Mask all swap targets. Signed-off-by: Mark D Horn <mark.d.horn@intel.com>	2020-12-17 10:38:43 -08:00
Hyunsun Moon	e74b3ca892	Add support of CNI version 0.3.1 to vfioveth Tested with K8S v1.18.6 and the latest multus stable image.	2020-09-10 15:14:19 -07:00
António Meireles	55b2aa2d19	accommodate upstream libvirt boxes changes in a forward and backward way (#330 ) per https://github.com/AntonioMeireles/ClearLinux-packer/issues/24 the ClearLinux libvirt boxes default size was lowered from the previous 40G to a more manageable 5G, having the user the ability to, at box instantiation, to increase that value to whatever is adequate. In order to avoid surprises in this side the Vagrantfile was modded to have hardcoded the previous default value. This change is forward and backward compatible as it will just be ignored when using older boxes ( since the 'new' root volume size just the same as original one) and will resize the box to the expected 40G in newer ones. Signed-off-by: António Meireles <antonio.meireles@reformi.st>	2020-08-19 14:23:51 -07:00
Ganesh Maharaj Mahalingam	aad050f944	Switch private network for Vagrant VMs (#324 ) Previously used IP range seems to be allocated to NASA and probably not a good idea to use it here. Switching the private network to be part of the IPs allocated to private networks. Signed-off-by: Ganesh Maharaj Mahalingam <ganesh.mahalingam@intel.com>	2020-06-15 09:40:22 -07:00
David Lyle	a872ccca6c	adding support for HA clusters, using HAProxy (#317 ) * adding support for HA clusters, using HAProxy Signed-off-by: David Lyle <dklyle0@gmail.com> * fixing typo * fixing load balancer port default value	2020-06-15 09:38:34 -07:00
Saikrishna Edupuganti	1525407bd4	Update multi-net components to latest releases (#329 ) * Update multi-net components to latest releases Multus CNI 3.4.2 SR-IOV CNI 2.3 SR-IOV DP 3.2 Tested as per the README. Works fine. Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com> * Provide dpdk-stable 3 LTS and 1 recent release Testpmd manifests of the last 3 LTS and 1 latest release from stable repo to help test actual DPDK app instead of sleep. Currently 17.11 and 19.11 are the only functioning ones without privileged. ``` NAME READY STATUS RESTARTS AGE dpdk-1711 1/1 Running 0 3m5s dpdk-1811 0/1 Error 0 3m5s dpdk-1911 1/1 Running 0 3m5s dpdk-2002 0/1 Error 0 3m5s ``` ``` EAL: PCI device 0000:07:06.4 on NUMA socket 0 EAL: probe driver: 8086:154c net_i40e_vf EAL: Getting a vfio_dev_fd for 0000:07:06.4 failed EAL: Requested device 0000:07:06.4 cannot be used … testpmd: No probed ethernet devices EAL: Error - exiting with code: 1 Cause: Invalid port 1 ``` Signed-off-by: Saikrishna Edupuganti <saikrishna.edupuganti@intel.com>	2020-06-05 09:06:08 -07:00