Upgrade from 2.0.x

This is a guide to upgrade Burrito Begonia 2.0.x to 2.1.1

I assume Burrito Begonia 2.0.x is already installed and running. This guide will show you how to upgrade it to Burrito Begonia 2.1.1.

Here is the example node ip address table.

Hostname	management IP
control1	192.168.21.111
control2	192.168.21.112
control3	192.168.21.113
compute1	192.168.21.114
compute2	192.168.21.115

KeepAlived VIP: 192.168.21.110

This is a k8s cluster version table.

Components	Begonia 2.0.x	Begonia 2.1.1
containerd	v1.7.13	v1.7.16
kubernetes	v1.29.2	v1.30.3
kubeadm	v1.29.2	v1.30.3
etcd	v3.5.10	v3.5.12
calico	v3.26.4	v3.27.3
coredns	v1.11.1	same as left
helm	v3.14.2	same as left
nerdctl	1.7.1	1.7.4
crictl	v1.29.0	v1.30.0
nodelocaldns	1.22.28	same as left
cert-manager	v1.13.2	v1.14.7
metallb	v0.13.9	same as left
registry	2.8.3	same as left

Modify kube-apiserver manifest

First, we need to modify kube-apiserver manifest.

Modify –anonymous-auth to true in /etc/kubernetes/manifests/kube-apiserver.yaml on every control node.:

$ sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
...
    - --anonymous-auth=true

Wait until kube-apiserver is restarted on each control node.

Check if we can connect to each kube-apiserver.:

$ curl -sk https://192.168.21.111:6443/healthz
ok
$ curl -sk https://192.168.21.112:6443/healthz
ok
$ curl -sk https://192.168.21.113:6443/healthz
ok

Prepare 2.1.1 iso

We will use Burrito 2.1.1 iso to upgrade the existing Burrito 2.0.x cluster.

Mount burrito-2.1.1_8.10.iso in /mnt.:

$ sudo mount -o loop,ro burrito-2.1.1_8.10.iso /mnt

Unarchive burrito-2.1.1 tarball from the iso.:

$ tar xzf /mnt/burrito-2.1.1.tar.gz

Remove localrepo.cfg and registry.cfg in /etc/haproxy/conf.d/.:

$ sudo rm /etc/haproxy/conf.d/{localrepo,registry}.cfg

These haproxy configuration files will be recreated while upgrading.

Reload haproxy.service on the first control node.:

$ sudo systemctl reload haproxy.service

Run prepare.sh script.:

$ cd burrito-2.1.1
$ ./prepare.sh offline

Copy the patched vars.yml.sample to vars.yml.:

$ cp vars.yml.sample vars.yml

Copy files from the existing burrito dir (e.g. $HOME/burrito-2.0.x).:

$ cp $HOME/burrito-2.0.x/.vaultpass .
$ cp $HOME/burrito-2.0.x/group_vars/all/vault.yml group_vars/all/
$ cp $HOME/burrito-2.0.x/.offline_flag .

Edit hosts and vars.yml for your environment.:

$ vi hosts
$ vi vars.yml

Edit group_vars/all/*.yml if you modified them for your environment.:

$ vi group_vars/all/netapp_vars.yml
$ vi group_vars/all/ceph_vars.yml

Check the node connectivity.:

$ ./run.sh ping

Check if keepalived_vip(192.168.21.110) is on the first control node.:

$ ip -br a s dev eth1
eth1             UP             192.168.21.111/24 192.168.21.110/32 fe80::5054:ff:feeb:2b8b/64

If it is not, move keepalived_vip to the first control node by restarting keepalived service. For example, if keepalived_vip is on the second control node, restart keepalived service on the second control node.:

$ sudo systemctl restart keepalived.service

Then the keepalived_vip will be moved to the first control node.

Remove registry, localrepo, and asklepios pods.:

$ sudo kubectl delete deploy registry localrepo asklepios -n kube-system
deployment.apps "registry" deleted
deployment.apps "localrepo" deleted
deployment.apps "asklepios" deleted

These pods will be recreated while upgrading.

Run preflight playbook.:

$ ./run.sh preflight

Run HA playbook.:

$ ./run.sh ha

You are ready to upgrade kubernetes cluster.

Upgrade kubernetes

Run k8s playbook with upgrade_cluster_setup=true.:

$ ./run.sh k8s -e upgrade_cluster_setup=true

It will take a long time. It took about 50 minutes in my testbed.

Check if the kubernetes version is v1.30.3.:

$ sudo kubectl version
Client Version: v1.30.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.3

Run storage playbook.:

$ ./run.sh storage

Run patch playbook.:

$ ./run.sh patch

Run registry playbook.:

$ ./run.sh registry

Check the new images(e.g. kube-apiserver:v1.30.3) are added to the local registry.:

$ curl -sk https://192.168.21.110:32680/v2/kube-apiserver/tags/list
{"name":"kube-apiserver","tags":["v1.30.3","v1.29.2"]}

Run landing playbook.:

$ ./run.sh landing

Check the new images (e.g. kube-apiserver:v1.29.2) are added to the genesis registry.:

$ curl -sk https://192.168.21.110:6000/v2/kube-apiserver/tags/list
{"name":"kube-apiserver","tags":["v1.29.2","v1.30.3"]}

Run burrito playbook with system tag to update /etc/hosts file.:

$ ./run.sh burrito --tags=system

Kubernetes upgrade is done!