Add control plane nodes
There is a single control plane node in the burrito cluster and we want to add two control plane nodes.
This is a guide to add the control plane nodes in the existing burrito cluster.
This is the current host inventory.:
extend-control1 ip=192.168.21.131 ansible_connection=local ansible_python_interpreter=/usr/bin/python3
extend-compute ip=192.168.21.134
extend-storage ip=192.168.21.135
# ceph nodes
[mons]
extend-storage
[mgrs]
extend-storage
[osds]
extend-storage
[rgws]
extend-storage
[clients]
extend-control1
extend-compute
# kubernetes nodes
[kube_control_plane]
extend-control1
[kube_node]
extend-control1
extend-compute
# openstack nodes
[controller-node]
extend-control1
[network-node]
extend-control1
[compute-node]
extend-compute
###################################################
## Do not touch below if you are not an expert!!! #
###################################################
There is only one control plane node. I will add two control plane nodes.
Edit hosts inventory file
Add extend-control{2,3} in hosts.:
$ diff -u hosts.old hosts
--- hosts.old 2024-10-16 11:09:39.412476505 +0900
+++ hosts 2024-10-16 11:09:32.814476505 +0900
@@ -1,4 +1,6 @@
extend-control1 ip=192.168.21.131 ansible_connection=local ansible_python_interpreter=/usr/bin/python3
+extend-control2 ip=192.168.21.132
+extend-control3 ip=192.168.21.133
extend-compute ip=192.168.21.134
extend-storage ip=192.168.21.135
@@ -16,23 +18,23 @@
extend-storage
[clients]
-extend-control1
+extend-control[1:3]
extend-compute
# kubernetes nodes
[kube_control_plane]
-extend-control1
+extend-control[1:3]
[kube_node]
-extend-control1
+extend-control[1:3]
extend-compute
# openstack nodes
[controller-node]
-extend-control1
+extend-control[1:3]
[network-node]
-extend-control1
+extend-control[1:3]
[compute-node]
extend-compute
###################################################
## Do not touch below if you are not an expert!!! #
###################################################
Preflight
Run the preflight playbook with –limit parameter.:
$ ./run.sh preflight --limit=extend-control2,extend-control3
Check burrito repo is set up.:
[clex@extend-control2 ~]$ sudo dnf repoinfo
Last metadata expiration check: 0:03:01 ago on Wed 16 Oct 2024 11:28:42 AM KST.
Repo-id : burrito
Repo-name : Burrito BaseOS
Repo-revision : 1713854508
Repo-updated : Tue 23 Apr 2024 03:41:48 PM KST
Repo-pkgs : 620
Repo-available-pkgs: 620
Repo-size : 816 M
Repo-baseurl : http://192.168.21.131:8001/BaseOS
Repo-expire : 172,800 second(s) (last: Wed 16 Oct 2024 11:28:42 AM KST)
Repo-filename : /etc/yum.repos.d/burrito.repo
Total packages: 620
Check time is synced.:
[clex@extend-control2 ~]$ chronyc tracking
Reference ID : C0A81583 (extend-control1)
Stratum : 9
Ref time (UTC) : Wed Oct 16 02:31:17 2024
System time : 0.000000000 seconds fast of NTP time
Last offset : -0.445706338 seconds
RMS offset : 0.445706338 seconds
Frequency : 0.125 ppm fast
Residual freq : +0.000 ppm
Skew : 41.850 ppm
Root delay : 0.000386291 seconds
Root dispersion : 0.002296808 seconds
Update interval : 0.0 seconds
Leap status : Normal
HA
Run the ha playbook to install keepalived and haproxy on new nodes.:
$ ./run.sh ha
Check keepalived and haproxy service are running on the new nodes.:
$ sudo systemctl status keepalived haproxy
Check the keepalived VIP is on the first control plane node.:
FIRST_CONTROL_PLANE_NODE$ ip -br a s dev MGMT_IFACE
MGMT_IFACE is the management interface name (e.g. eth1).
The keepalived VIP could be moved to the other control plane node. If it is moved, move it back to the first control plane node by restarting keepalived service on the node.:
$ sudo systemctl restart keepalived.service
Ceph
If ceph is in storage backends, run the ceph playbook with ‘ceph_client’ tag to install ceph client on the new nodes.:
$ ./run.sh ceph --tags=ceph_client
Check ‘ceph -s’ command works on the new nodes.:
$ sudo ceph -s
cluster:
id: 8d902f73-3445-449e-9246-03b8b459821f
health: HEALTH_OK
services:
mon: 1 daemons, quorum extend-storage (age 17h)
mgr: extend-storage(active, since 17h)
osd: 3 osds: 3 up (since 17h), 3 in (since 17h)
rgw: 1 daemon active (1 hosts, 1 zones)
data:
pools: 10 pools, 289 pgs
objects: 2.31k objects, 6.8 GiB
usage: 15 GiB used, 285 GiB / 300 GiB avail
pgs: 289 active+clean
io:
client: 61 KiB/s wr, 0 op/s rd, 9 op/s wr
K8S
Before running the k8s playbook, we need to change kube-apiserver parameter in the first control plane node.:
$ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
...
- --anonymous-auth=true
Wait until kube-apiserver is restarted on each control node.
Check if we can connect to kube-apiserver on the first control plane node.:
$ curl -sk https://THE_FIRST_CONTROL_PLANE_NODE_IP:6443/healthz
ok
Run a k8s playbook.:
$ ./run.sh k8s --extra-vars="registry_enabled="
Check the node list.:
$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
extend-compute Ready <none> 4h19m v1.28.3
extend-control1 Ready control-plane 4h20m v1.28.3
extend-control2 Ready control-plane 110m v1.28.3
extend-control3 Ready control-plane 110m v1.28.3
NetApp
If netapp is in storage backends, run the netapp playbook.:
$ ./run.sh netapp
Patch
Run the patch playbook.:
$ ./run.sh patch
Landing
Run the landing playbook.:
$ ./run.sh landing --tags=genesisregistry
Check the genesis registry service is running on the added nodes.:
$ sudo systemctl status genesis_registry.service
Run the localrepo_haproxy_setup playbook.:
$ ./run.sh localrepo_haproxy_setup
Check the localrepo.cfg file is in /etc/haproxy/conf.d/.:
$ sudo ls -1 /etc/haproxy/conf.d/localrepo.cfg
/etc/haproxy/conf.d/localrepo.cfg
Burrito.system
Run the burrito playbook with –tags=system.:
$ ./run.sh burrito --tags=system
OpenStack
Reinstall each openstack component.
There are two types of replicas - the HA replica and the quorum replica.
The HA replica type sets up two pods for high availability. The quorum replica type sets up three pods for quorum membership.
The mariadb and rabbitmq are the quorum replica type. The others are the HA replica type except the ingress.
The ingress is a special replica type that works like a daemonset.
Install ingress.:
$ ./scripts/burrito.sh install ingress
Check if there are three ingress pods.:
root@btx-0:/# k get po -l application=ingress,component=server
NAME READY STATUS RESTARTS AGE
ingress-0 1/1 Running 0 24h
ingress-1 1/1 Running 0 2m4s
ingress-2 1/1 Running 0 86s
Install mariadb.:
$ ./scripts/burrito.sh install mariadb
Check if there are three mariadb server pods.:
root@btx-0:/# k get po -l application=mariadb,component=server
NAME READY STATUS RESTARTS AGE
mariadb-server-0 1/1 Running 0 76s
mariadb-server-1 1/1 Running 0 3m27s
mariadb-server-2 1/1 Running 0 3m27s
Install rabbitmq.:
$ ./scripts/burrito.sh install rabbitmq
Check if there are three rabbitmq pods.:
root@btx-0:/# k get po -l application=rabbitmq,component=server
NAME READY STATUS RESTARTS AGE
rabbitmq-rabbitmq-0 1/1 Running 0 25h
rabbitmq-rabbitmq-1 1/1 Running 0 4m26s
rabbitmq-rabbitmq-2 1/1 Running 0 4m26s
Install keystone.:
$ ./scripts/burrito.sh install keystone
Check if there are two keystone-api pods.:
root@btx-0:/# k get po -l application=keystone,component=api
NAME READY STATUS RESTARTS AGE
keystone-api-667dfbb9bd-bjt6f 1/1 Running 0 112s
keystone-api-667dfbb9bd-f5kjn 1/1 Running 0 112s
Install glance.:
$ ./scripts/burrito.sh install glance
Check if there are two glance-api pods.:
root@btx-0:/# k get po -l application=glance,component=api
NAME READY STATUS RESTARTS AGE
glance-api-0 2/2 Running 0 61m
glance-api-1 2/2 Running 0 62m
Install neutron.:
$ ./scripts/burrito.sh install neutron
Check if there are two neutron server pods.:
root@btx-0:/# k get po -l application=neutron,component=server
NAME READY STATUS RESTARTS AGE
neutron-server-567dfbfd84-p8vdr 2/2 Running 0 128m
neutron-server-567dfbfd84-wjsmr 2/2 Running 0 128m
Install nova.:
$ ./scripts/burrito.sh install nova
Check if there are two nova-api pods.:
root@btx-0:/# k get po -l application=nova,component=os-api
NAME READY STATUS RESTARTS AGE
nova-api-osapi-7d95bf7f85-h2prv 1/1 Running 0 6m26s
nova-api-osapi-7d95bf7f85-twhvg 1/1 Running 0 6m26s
Install cinder.:
$ ./scripts/burrito.sh install cinder
Check if there are two cinder-api pods.:
root@btx-0:/# k get po -l application=cinder,component=api
NAME READY STATUS RESTARTS AGE
cinder-api-7549d5dbb7-4j5tt 1/1 Running 0 2m10s
cinder-api-7549d5dbb7-v9mw7 1/1 Running 0 2m10s
Install horizon.:
$ ./scripts/burrito.sh install horizon
Check if there are two horizon pods.:
root@btx-0:/# k get po -l application=horizon,component=server
NAME READY STATUS RESTARTS AGE
horizon-56454f565f-5tdgv 1/1 Running 0 2m27s
horizon-56454f565f-vc2vg 1/1 Running 0 2d
We have finished adding the control plane nodes in burrito cluster.