Burrito Installation

This is a guide to install Burrito in online environment.

Supported OS

  • Rocky Linux 8.x

System requirements

This is the minimum system requirements to install Burrito.

node role

CPU (ea)

Memory (GB)

Disk (GB)

Extra Disks

control

8

16

50

N/A

compute

4

8

50

N/A

storage

4

8

50

3 ea x 50GB

If you have more resources, consider allocating more resources to each node.

Networks

The standard number of networks for burrito is 5.

  • service network: Public service network (e.g. 192.168.20.0/24)

  • management network: Management and internal network (e.g. 192.168.21.0/24)

  • provider network: OpenStack provider network (e.g. 192.168.22.0/24)

  • overlay network: OpenStack overlay network (e.g. 192.168.23.0/24)

  • storage network: Ceph public/cluster network (e.g. 192.168.24.0/24)

If you do not know what each network is for, consult openstack experts.

Reference network architecture

This is the reference network architecture.

  • control/compute machines have all 5 networks.

  • No ip address is assigned on the provider network.

  • storage machines have 2 networks (management and storage)

hostname

service

management

provider

overlay

storage

.

eth0

eth1

eth2

eth3

eth4

.

192.168.20.x

192.168.21.x

192.168.22.x

192.168.23.x

192.168.24.x

control1

.101

.101

(no ip)

.101

.101

control2

.102

.102

(no ip)

.102

.102

control3

.103

.103

(no ip)

.103

.103

compute1

.104

.104

(no ip)

.104

.104

compute2

.105

.105

(no ip)

.105

.105

storage1

.106

.106

storage2

.107

.107

storage3

.108

.108

  • KeepAlived VIP on management: 192.168.21.100

  • KeepAlived VIP on service: 192.168.20.100

Pre-requisites

  • Rocky Linux 8.x is installed on every node.

  • The python3 package should be already installed on every node.

  • The first node in control group is the ansible deployer.

  • Ansible user in every node has a sudo privilege.

  • All nodes should be in /etc/hosts on the deployer node.

Here is the example of /etc/hosts on the deployer node.:

127.0.0.1 localhost
192.168.21.101 control1
192.168.21.102 control2
192.168.21.103 control3
192.168.21.104 compute1
192.168.21.105 compute2
192.168.21.106 storage1
192.168.21.107 storage2
192.168.21.108 storage3

Prepare

Install git package on the deployer if not already installed.:

$ sudo dnf -y install git

Get the burrito source.:

$ git clone --recursive https://github.com/iorchard/burrito.git

Go to burrito directory.:

$ cd burrito

Run prepare.sh script.:

$ ./prepare.sh

inventory hosts and variables

There are 4 groups of hosts in burrito.

  • Control node: runs kubernetes and openstack control-plane components.

  • Network node: runs kubernetes worker and openstack network services.

  • Compute node: runs kubernetes worker and openstack hypervisor and network agent to operate instances.

  • Storage node: runs Ceph storage services - monitor, manager, osd, rados gateway.

Network node is optional. Control node usually acts as both control and network node.

Edit inventory hosts

There are sample inventory files.

  • hosts.sample (default):

    This is a sample file using ceph as the storage backend.

  • hosts_powerflex.sample:

    This is a sample file using powerflex as the storage backend.

  • hosts_powerflex_hci.sample:

    This is a sample file using powerflex HCI (Hyper-Converged Infrastructure).

  • hosts_hitachi.sample:

    This is a sample file using hitachi as the storage backend. But burrito does not support hitachi storage for online installation.

Warning

You need to get the powerflex rpm packages from Dell if you want to install powerflex in burrito.

When you run prepare.sh script, the default hosts.sample is copied to hosts file.

If you want to use powerflex, copy one of powerflex inventory files.:

$ cp hosts_powerflex_hci.sample hosts

Here are the sample inventory files.

the default inventory file
 1control1 ip=192.168.21.101 ansible_connection=local ansible_python_interpreter=/usr/bin/python3
 2control2 ip=192.168.21.102
 3control3 ip=192.168.21.103
 4compute1 ip=192.168.21.104
 5compute2 ip=192.168.21.105
 6storage1 ip=192.168.21.106
 7storage2 ip=192.168.21.107
 8storage3 ip=192.168.21.108
 9
10# ceph nodes
11[mons]
12storage[1:3]
13
14[mgrs]
15storage[1:3]
16
17[osds]
18storage[1:3]
19
20[rgws]
21storage[1:3]
22
23[clients]
24control[1:3]
25compute[1:2]
26
27# kubernetes nodes
28[kube_control_plane]
29control[1:3]
30
31[kube_node]
32control[1:3]
33compute[1:2]
34
35# openstack nodes
36[controller-node]
37control[1:3]
38
39[network-node]
40control[1:3]
41
42[compute-node]
43compute[1:2]
44
45###################################################
46## Do not touch below if you are not an expert!!! #
47###################################################
the powerflex inventory file
 1control1 ip=192.168.21.101 ansible_connection=local ansible_python_interpreter=/usr/bin/python3
 2control2 ip=192.168.21.102
 3control3 ip=192.168.21.103
 4compute1 ip=192.168.21.104
 5compute2 ip=192.168.21.105
 6storage1 ip=192.168.21.106
 7storage2 ip=192.168.21.107
 8storage3 ip=192.168.21.108
 9
10# ceph nodes
11[mons]
12[mgrs]
13[osds]
14[rgws]
15[clients]
16
17# powerflex nodes
18[mdm]
19storage[1:3]
20
21[sds]
22storage[1:3]
23
24[sdc]
25control[1:3]
26compute[1:2]
27
28[gateway]
29storage[1:2]
30
31[presentation]
32storage3
33
34# kubernetes nodes
35[kube_control_plane]
36control[1:3]
37
38[kube_node]
39control[1:3]
40compute[1:2]
41
42# openstack nodes
43[controller-node]
44control[1:3]
45
46[network-node]
47control[1:3]
48
49[compute-node]
50compute[1:2]
51
52###################################################
53## Do not touch below if you are not an expert!!! #
54###################################################
the powerflex HCI inventory file
 1pfx-1 ip=192.168.21.131 ansible_connection=local ansible_python_interpreter=/usr/bin/python3
 2pfx-2 ip=192.168.21.132
 3pfx-3 ip=192.168.21.133
 4
 5# ceph nodes
 6[mons]
 7[mgrs]
 8[osds]
 9[rgws]
10[clients]
11
12# powerflex nodes
13[mdm]
14pfx-[1:3]
15
16[sds]
17pfx-[1:3]
18
19[sdc]
20pfx-[1:3]
21
22[gateway]
23pfx-[1:2]
24
25[presentation]
26pfx-3
27
28# kubernetes nodes
29[kube_control_plane]
30pfx-[1:3]
31
32[kube_node]
33pfx-[1:3]
34
35# openstack nodes
36[controller-node]
37pfx-[1:3]
38
39[network-node]
40pfx-[1:3]
41
42[compute-node]
43pfx-[1:3]
44
45###################################################
46## Do not touch below if you are not an expert!!! #
47###################################################

Warning

Beware that control nodes are in network-node group since there is no network node in these sample files.

Edit vars.yml

 1---
 2### define network interface names
 3# set overlay_iface_name to null if you do not want to set up overlay network.
 4# then, only provider network will be set up.
 5svc_iface_name: eth0
 6mgmt_iface_name: eth1
 7provider_iface_name: eth2
 8overlay_iface_name: eth3
 9storage_iface_name: eth4
10
11### ntp
12# Specify time servers for control nodes.
13# You can use the default ntp.org servers or time servers in your network.
14# If servers are offline and there is no time server in your network,
15#   set ntp_servers to empty list.
16#   Then, the control nodes will be the ntp servers for other nodes.
17# ntp_servers: []
18ntp_servers:
19  - 0.pool.ntp.org
20  - 1.pool.ntp.org
21  - 2.pool.ntp.org
22
23### keepalived VIP on management network (mandatory)
24keepalived_vip: ""
25# keepalived VIP on service network (optional)
26# Set this if you do not have a direct access to management network
27# so you need to access horizon dashboard through service network.
28keepalived_vip_svc: ""
29
30### metallb
31# To use metallb LoadBalancer, set this to true
32metallb_enabled: false
33# set up MetalLB LoadBalancer IP range or cidr notation
34# IP range: 192.168.20.95-192.168.20.98 (4 IPs can be assigned.)
35# CIDR: 192.168.20.128/26 (192.168.20.128 - 191 can be assigned.)
36# Only one IP: 192.168.20.95/32
37metallb_ip_range:
38  - "192.168.20.95-192.168.20.98"
39
40### HA tuning
41# ha levels: moderato, allegro, and vivace
42# moderato: default liveness update and failover response
43# allegro: faster liveness update and failover response
44# vivace: fastest liveness update and failover response
45ha_level: "moderato"
46k8s_ha_level: "moderato"
47
48### storage
49# storage backends
50# If there are multiple backends, the first one is the default backend.
51storage_backends:
52  - ceph
53  - netapp
54  - powerflex
55  - hitachi
56
57# ceph: set ceph configuration in group_vars/all/ceph_vars.yml
58# netapp: set netapp configuration in group_vars/all/netapp_vars.yml
59# powerflex: set powerflex configuration in group_vars/all/powerflex_vars.yml
60# hitachi: set hitachi configuration in group_vars/all/hitachi_vars.yml
61
62###################################################
63## Do not edit below if you are not an expert!!!  #
64###################################################

Description of each variable

*_iface_name

Set each network interface name.

If you want to set up only provider network, set overlay_iface_name to null. Then, openstack neutron will disable self-service(overlay) network.

ntp_servers (default: {0,1,2}.pool.ntp.org)

Specify time servers for control nodes. You can use the default ntp.org server or time servers in your network.

If servers are offline and there is no time server in your network, set ntp_servers to empty list(ntp_servers: []). Then the control nodes will be the ntp servers for other nodes.

keepalived_vip (mandatory)

Assign VIP address on management network for LoadBalancing and High Availability to internal services. This is mandatory.

keepalived_vip_svc (optional)

Assign VIP address on service network for horizon dashboard service. Set this if you do not have a direct access to management network.

If it is not assigned, you have to connect to horizon dashboard via keepalived_vip on management network.

metallb_enabled (default: false)

Set true to use metallb LoadBalancer. (See ` what is metallb? <https://metallb.universe.tf/>`_)

metallb_ip_range

Set metallb LoadBalancer IP range or cidr notation.

  • IP range: 192.168.20.95-192.168.20.98 (4 IPs can be assigned.)

  • CIDR: 192.168.20.128/26 (192.168.20.128 - 191 can be assigned.)

  • Only one IP: 192.168.20.95/32 (192.168.20.95 can be assigned.)

ha_level

Set KeepAlived/HAProxy HA level. It should be one of moderato(default), allegro, and vivace. Each level sets the following parameters.

  • interval: health check interval in seconds

  • timeout: health check timeout in seconds

  • rise: required number of success

  • fall: required number of failure

k8s_ha_level

Set kubernetes HA level. It should be one of moderato(default), allegro, and vivace. Each level sets the following parameters.

  • node_status_update_frequency: Specifies how often kubelet posts node status to master.

  • node_monitor_period: The period for syncing NodeStatus in NodeController.

  • node_monitor_grace_period: Amount of time which we allow running Node to be unresponsive before marking it unhealthy.

  • not_ready_toleration_seconds: the tolerationSeconds of the toleration for notReady:NoExecute that is added by default to every pod that does not already have such a toleration

  • unreachable_toleration_seconds: the tolerationSeconds of the toleration for unreachable:NoExecute that is added by default to every pod that does not already have such a toleration

  • kubelet_shutdown_grace_period: the total duration that the node should delay the shutdown by

  • kubelet_shutdown_grace_period_critical_pods: the duration used to terminate critical pods during a node shutdown

storage_backends

Burrito supports the following storage backends - ceph, netapp, powerflex, and hitachi (hitachi storage cannot be used for online installation).

If there are multiple backends, the first one is the default backend. It means the default storageclass, glance store and the default cinder volume type is the first backend.

The Persistent Volumes in k8s are created on the default backend if you do not specify the storageclass name.

The volumes in openstack are created on the default backend if you do not specify the volume type.

storage variables

ceph

If ceph is in storage_backends, run lsblk command on storage nodes to get the device names.

storage1$ lsblk -p
NAME        MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
/dev/sda      8:0    0  50G  0 disk
└─/dev/sda1   8:1    0  50G  0 part /
/dev/sdb      8:16   0  50G  0 disk
/dev/sdc      8:32   0  50G  0 disk
/dev/sdd      8:48   0  50G  0 disk

In this case, /dev/sda is the OS disk and /dev/sd{b,c,d} are for ceph OSD disks.

Edit group_vars/all/ceph_vars.yml.

1---
2# ceph config
3lvm_volumes:
4  - data: /dev/sdb
5  - data: /dev/sdc
6  - data: /dev/sdd
7...

netapp

If netapp is in storage_backends, edit group_vars/all/netapp_vars.yml.

 1---
 2netapp:
 3  - name: netapp1
 4    managementLIF: "192.168.100.230"
 5    dataLIF: "192.168.140.19"
 6    svm: "svm01"
 7    username: "admin"
 8    password: "<netapp_admin_password>"
 9    nfsMountOptions: "nfsvers=4,lookupcache=pos"
10    shares:
11      - /dev03
12...

You can add nfsvers in nfsMountOptions to use the specific nfs version.

For example, if you want to use nfs version 4.0, put nfsvers=4.0 in nfsMountOptions (nfsMountOptions: “nfsvers=4.0,lookupcache=pos”). Then, you should check if nfs version 4 is enabled in NetApp NFS storage.

If you do not know what these variables are, contact a Netapp engineer.

powerflex

If powerflex is in storage_backends, run lsblk command on storage nodes to get the device names.

1storage1$ lsblk -p
2NAME        MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
3/dev/sda      8:0    0  50G  0 disk
4└─/dev/sda1   8:1    0  50G  0 part /
5/dev/sdb      8:16   0  50G  0 disk
6/dev/sdc      8:32   0  50G  0 disk
7/dev/sdd      8:48   0  50G  0 disk

In this case, /dev/sda is the OS disk and /dev/sd{b,c,d} are for powerflex SDS disks.

Edit group_vars/all/powerflex_vars.yml and add /dev/sd{b,c,d} in it.

 1# MDM VIPs on storage networks
 2mdm_ip:
 3  - "192.168.24.100"
 4storage_iface_names:
 5  - eth4
 6sds_devices:
 7  - /dev/sdb
 8  - /dev/sdc
 9  - /dev/sdd
10
11#
12# Do Not Edit below
13#

If you do not know what these variables are, contact a Dell engineer.

Create a vault secret file

Create a vault file to encrypt passwords.:

$ ./run.sh vault
<user> password:
openstack admin password:
Encryption successful

Enter <user> password for ssh connection to other nodes.

Enter openstack admin password which will be used when you connect to openstack horizon dashboard.

Check the connectivity

Check the connections to other nodes.:

$ ./run.sh ping

It should show SUCCESS on all nodes.

Install

There should be no failed tasks in PLAY RECAP on each playbook run.

For example:

PLAY RECAP *****************************************************************
control1                   : ok=20   changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
control2                   : ok=19   changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
control3                   : ok=19   changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Each step has a verification process, so be sure to verify before proceeding to the next step.

Warning

Never proceed to the next step if the verification fails.

Step.1 Preflight

The Preflight installation step implements the following tasks.

  • Set up a local yum repository.

  • Configure NTP time servers and clients.

  • Deploy the public ssh key to other nodes (if deploy_ssh_key is true).

Install

Run a preflight playbook.:

$ ./run.sh preflight

Verify

Check if the ntp servers and clients are configured.

When you set ntp_servers to the default ntp servers, each control node should have the ntp servers on the internet.:

control1$ chronyc sources
MS Name/IP address         Stratum Poll Reach LastRx Last sample
=========================================================================
^* send.mx.cdnetworks.com  2  10   377    98  -1096us[-1049us] +/-   49ms
^- 121.162.54.1            3   6   377     1  -4196us[-4196us] +/-   38ms
^+ 106.247.248.106         2  10   377    50  +2862us[+2862us] +/-   61ms

Compute/storage nodes should have control nodes as time servers.:

$ chronyc sources
MS Name/IP address      Stratum Poll Reach LastRx Last sample
========================================================================
^* control1             8   6   377    46    -15us[  -44us] +/-  212us
^- control2             9   6   377    47    -57us[  -86us] +/-  513us
^- control3             9   6   377    47    -97us[ -126us] +/-  674us

Step.2 HA

The HA installation step implements the following tasks.

  • Set up KeepAlived service.

  • Set up HAProxy service.

KeepAlived and HAProxy services are the vital services for burrito platform.

Ceph Rados Gateway service is dependent of them.

Install

Run a HA stack playbook.:

$ ./run.sh ha

Verify

Check if keepalived and haproxy are running on control nodes.:

$ sudo systemctl status keepalived haproxy
keepalived.service - LVS and VRRP High Availability Monitor
...
   Active: active (running) since Wed 2023-05-31 17:29:05 KST; 6min ago
...
haproxy.service - HAProxy Load Balancer
...
   Active: active (running) since Wed 2023-05-31 17:28:52 KST; 8min ago

Check if keepalived_vip is created on the management interface in the first control node.:

$ ip -br -4 address show dev eth1
eth1             UP             192.168.21.101/24 192.168.21.100/32

Check if keepalived_vip_svc is created on the service interface in the first control node if you set it up.:

$ ip -br -4 address show dev eth0
eth0             UP             192.168.20.101/24 192.168.20.100/32

Step.3 Ceph

Skip this step if ceph is not in storage_backends.

The Ceph installation step implements the following tasks.

  • Install ceph server and client packages in storage nodes.

  • Install ceph client packages in other nodes.

  • Set up ceph monitor, manager, osd, rados gateway services on storage nodes.

Install

Run a ceph playbook if ceph is in storage_backends.:

$ ./run.sh ceph

Verify

Check ceph health after running ceph playbook.:

$ sudo ceph health
HEALTH_OK

It should show HEALTH_OK.

To get the detailed health status, run sudo ceph -s command. It will show the output like this.:

$ sudo ceph -s
  cluster:
    id:     cd7bdd5a-1814-4e6a-9e07-c2bdc3f53fea
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum storage1,storage2,storage3 (age 17h)
    mgr: storage2(active, since 17h), standbys: storage1, storage3
    osd: 9 osds: 9 up (since 17h), 9 in (since 17h)
    rgw: 3 daemons active (3 hosts, 1 zones)

  data:
    pools:   10 pools, 513 pgs
    objects: 2.54k objects, 7.3 GiB
    usage:   19 GiB used, 431 GiB / 450 GiB avail
    pgs:     513 active+clean

There are 4 services - mon, mgr, osd, and rgw.

Sometimes it could show HEALTH_WARN <something> have recently crashed. Don’t worry. it is mostly harmless warning.

List the crashes.:

$ sudo ceph crash ls

Archive all crashes.:

$ sudo ceph crash archive-all

Then, check ceph health again. It will show HEALTH_OK now.

Step.4 Kubernetes

The Kubernetes installation step implements the following tasks.

  • Install kubernetes binaries in kubernetes nodes.

  • Set up kubernetes control plane.

  • Set up kubernete worker nodes.

Install

Run a k8s playbook.:

$ ./run.sh k8s

Verify

Check if all nodes are in ready state.:

$ sudo kubectl get nodes
NAME       STATUS   ROLES           AGE   VERSION
compute1   Ready    <none>          15m   v1.28.3
compute2   Ready    <none>          15m   v1.28.3
control1   Ready    control-plane   17m   v1.28.3
control2   Ready    control-plane   16m   v1.28.3
control3   Ready    control-plane   16m   v1.28.3

Step.5.1 Netapp

Skip this step if netapp is not in storage_backends.

The Netapp installation step implements the following tasks.

  • Install trident components in trident namespace.

  • Set up a netapp backend.

  • Create a netapp storageclass.

Install

Run a netapp playbook.:

$ ./run.sh netapp

Verify

Check if all pods are running and ready in trident namespace.:

$ sudo kubectl get pods -n trident
NAME                           READY   STATUS    RESTARTS   AGE
trident-csi-6b96bb4f87-tw22r   6/6     Running   0          43s
trident-csi-84g2x              2/2     Running   0          42s
trident-csi-f6m8w              2/2     Running   0          42s
trident-csi-klj7h              2/2     Running   0          42s
trident-csi-kv9mw              2/2     Running   0          42s
trident-csi-r8gqv              2/2     Running   0          43s

Check if netapp storageclass is created.:

$ sudo kubectl get storageclass netapp
NAME               PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
netapp (default)   csi.trident.netapp.io   Delete          Immediate           true                   20h

Step.5.2 Powerflex

Skip this step if powerflex is not in storage_backends.

The powerflex installation step implements the following tasks.

  • Install powerflex rpm packages.

  • Create powerflex MDM cluster.

  • Configure gateway and presentation services.

  • Set up Protection Domain, Storage Pool, and SDS devices.

  • Install vxflexos controller and node in vxflexos namespace.

  • Create a powerflex storageclass.

Prepare

To install powerflex, you need to have powerflex rpm packages.

Create the rpm package tarball powerflex_pkgs.tar.gz in /mnt.

$ ls
EMC-ScaleIO-gateway-3.6-700.103.x86_64.rpm
EMC-ScaleIO-lia-3.6-700.103.el8.x86_64.rpm
EMC-ScaleIO-mdm-3.6-700.103.el8.x86_64.rpm
EMC-ScaleIO-mgmt-server-3.6-700.101.noarch.rpm
EMC-ScaleIO-sdc-3.6-700.103.el8.x86_64.rpm
EMC-ScaleIO-sds-3.6-700.103.el8.x86_64.rpm
$ sudo tar cvzf /mnt/powerflex_pkgs.tar.gz EMC-*.rpm

Warning

The tarball should be placed in /mnt.

Install

Run a powerflex playbook.:

$ ./run.sh powerflex

Verify

Check if all pods are running and ready in vxflexos namespace.:

$ sudo kubectl get pods -n vxflexos
NAME                                   READY   STATUS    RESTARTS   AGE
vxflexos-controller-744989794d-92bvf   5/5     Running   0          18h
vxflexos-controller-744989794d-gblz2   5/5     Running   0          18h
vxflexos-node-dh55h                    2/2     Running   0          18h
vxflexos-node-k7kpb                    2/2     Running   0          18h
vxflexos-node-tk7hd                    2/2     Running   0          18h

Check if powerflex storageclass is created.:

$ sudo kubectl get storageclass powerflex
NAME                  PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
powerflex (default)   csi-vxflexos.dellemc.com   Delete          WaitForFirstConsumer   true                   20h

Step.6 Patch

The Patch installation step implements the following tasks.

  • Install ceph-csi driver if ceph is in storage_backends.

  • Patch kube-apiserver.

Install

Run a patch playbook.:

$ ./run.sh patch

Verify

It will take some time to restart kube-apiserver after the patch.

Check if all pods are running and ready in kube-system namespace.

pod list in kube-system namespace
$ sudo kubectl get pods -n kube-system
NAME                                       READY STATUS    RESTARTS      AGE
calico-kube-controllers-67c66cdbfb-rz8lz   1/1   Running   0             60m
calico-node-28k2c                          1/1   Running   0             60m
calico-node-7cj6z                          1/1   Running   0             60m
calico-node-99s5j                          1/1   Running   0             60m
calico-node-tnmht                          1/1   Running   0             60m
calico-node-zmpxs                          1/1   Running   0             60m
coredns-748d85fb6d-c8cj2                   1/1   Running   1 (28s ago)   59m
coredns-748d85fb6d-gfv98                   1/1   Running   1 (27s ago)   59m
dns-autoscaler-795478c785-hrjqr            1/1   Running   1 (32s ago)   59m
kube-apiserver-control1                    1/1   Running   0             33s
kube-apiserver-control2                    1/1   Running   0             34s
kube-apiserver-control3                    1/1   Running   0             35s
kube-controller-manager-control1           1/1   Running   1             62m
kube-controller-manager-control2           1/1   Running   1             62m
kube-controller-manager-control3           1/1   Running   1             62m
kube-proxy-jjq5l                           1/1   Running   0             61m
kube-proxy-k4kxq                           1/1   Running   0             61m
kube-proxy-lqtgc                           1/1   Running   0             61m
kube-proxy-qhdzh                           1/1   Running   0             61m
kube-proxy-vxrg8                           1/1   Running   0             61m
kube-scheduler-control1                    1/1   Running   2             62m
kube-scheduler-control2                    1/1   Running   1             62m
kube-scheduler-control3                    1/1   Running   1             62m
nginx-proxy-compute1                       1/1   Running   0             60m
nginx-proxy-compute2                       1/1   Running   0             60m
nodelocaldns-5dbbw                         1/1   Running   0             59m
nodelocaldns-cq2sd                         1/1   Running   0             59m
nodelocaldns-dzcjr                         1/1   Running   0             59m
nodelocaldns-plhwm                         1/1   Running   0             59m
nodelocaldns-vlb8w                         1/1   Running   0             59m

Step.7 Landing

The Landing installation step implements the following tasks.

  • Install Graceful Node Shutdown Helper (GNSH).

Install

Run landing playbook.:

$ ./run.sh landing

Verify

Check if the Graceful Node Shutdown Helper (GNSH) service is running.:

$ sudo systemctl status gnsh.service
 gnsh.service - Graceful Node Shutdown Helper
   Loaded: loaded (/etc/systemd/system/gnsh.service; enabled; vendor preset: di>
   Active: active (exited) since Tue 2023-11-07 13:58:34 KST; 25min ago
  Process: 435851 ExecStart=/usr/bin/gnsh start (code=exited, status=0/SUCCESS)
 Main PID: 435851 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 100633)
   Memory: 0B
   CGroup: /system.slice/gnsh.service

Nov 07 13:58:34 control1 systemd[1]: Starting Graceful Node Shutdown Helper...
Nov 07 13:58:34 control1 gnsh[435851]: Uncordon my node control1.
Nov 07 13:58:34 control1 gnsh[435853]: node/control1 already uncordoned
Nov 07 13:58:34 control1 systemd[1]: Started Graceful Node Shutdown Helper.

Congratulations!

You’ve just finished the installation of burrito kubernetes platform.

Next you will install OpenStack on burrito kubernetes platform.

Step.8 Burrito

The Burrito installation step implements the following tasks.

  • Create a rados gateway user (default: cloudpc) and a client configuration (s3cfg).

  • Deploy nova vnc TLS certificate.

  • Deploy openstack components.

  • Create a nova ssh keypair and copy them on every compute nodes.

Install

Run a burrito playbook.:

$ sudo helm plugin install https://github.com/databus23/helm-diff
$ ./run.sh burrito

Verify

Check if all pods are running and ready in openstack namespace.:

$ sudo kubectl get pods -n openstack
NAME                                   READY   STATUS      RESTARTS   AGE
barbican-api-664986fd5-jkp9x           1/1     Running     0          4m23s
...
rabbitmq-rabbitmq-0                    1/1     Running     0          27m
rabbitmq-rabbitmq-1                    1/1     Running     0          27m
rabbitmq-rabbitmq-2                    1/1     Running     0          27m

Congratulations!

You’ve just finished the OpenStack installation on burrito kubernetes platform.

Horizon

The horizon dashboard listens on tcp 31000 on control nodes.

Here is how to connect to the horizon dashboard on your browser.

  1. Open your browser.

  2. If keepalived_vip_svc is set, go to https://<keepalived_vip_svc>:31000/

  3. If keepalived_vip_svc is not set, go to https://<keepalived_vip>:31000/

  4. Accept the self-signed TLS certificate and log in. The admin password is the one you set when you run vault.sh script (openstack admin password:).

Next, perform the basic openstack operation test using btx (burrito toolbox).

BTX

BTX is a toolbox for burrito platform. It should be already up and running.:

$ sudo kubectl -n openstack get pods -l application=btx
NAME    READY   STATUS    RESTARTS   AGE
btx-0   1/1     Running   0          36m

Let’s go into btx shell (bts).:

$ . ~/.btx.env
$ bts

Check openstack volume service status.:

root@btx-0:/# openstack volume service list
+------------------+------------------------------+------+---------+-------+----------------------------+
| Binary           | Host                         | Zone | Status  | State | Updated At                 |
+------------------+------------------------------+------+---------+-------+----------------------------+
| cinder-scheduler | cinder-volume-worker         | nova | enabled | up    | 2023-05-31T12:05:02.000000 |
| cinder-volume    | cinder-volume-worker@rbd1    | nova | enabled | up    | 2023-05-31T12:05:02.000000 |
| cinder-volume    | cinder-volume-worker@netapp1 | nova | enabled | up    | 2023-05-31T12:05:07.000000 |
+------------------+------------------------------+------+---------+-------+----------------------------+

Check openstack network agent status.:

root@btx-0:/# openstack network agent list
+--------------------------------------+--------------------+----------+-------------------+-------+-------+---------------------------+
| ID                                   | Agent Type         | Host     | Availability Zone | Alive | State | Binary                    |
+--------------------------------------+--------------------+----------+-------------------+-------+-------+---------------------------+
| 0b4ddf14-d593-44bb-a0aa-2776dfc20dc9 | Metadata agent     | control1 | None              | :-)   | UP    | neutron-metadata-agent    |
| 189c6f4a-4fad-4962-8439-0daf400fcae0 | DHCP agent         | control3 | nova              | :-)   | UP    | neutron-dhcp-agent        |
| 22b0d873-4192-41ad-831b-0d468fa2e411 | Metadata agent     | control3 | None              | :-)   | UP    | neutron-metadata-agent    |
| 4e51b0a0-e38a-402e-bbbd-5b759130220f | Linux bridge agent | compute1 | None              | :-)   | UP    | neutron-linuxbridge-agent |
| 56e43554-47bc-45c8-8c46-fb2aa0557cc0 | DHCP agent         | control1 | nova              | :-)   | UP    | neutron-dhcp-agent        |
| 7f51c2b7-b9e3-4218-9c7b-94076d2b162a | Linux bridge agent | compute2 | None              | :-)   | UP    | neutron-linuxbridge-agent |
| 95d09bfd-0d71-40d4-a5c2-d46eb640e967 | DHCP agent         | control2 | nova              | :-)   | UP    | neutron-dhcp-agent        |
| b76707f2-f13c-4f68-b769-fab8043621c7 | Linux bridge agent | control3 | None              | :-)   | UP    | neutron-linuxbridge-agent |
| c3a6a32c-cbb5-406c-9b2f-de3734234c46 | Linux bridge agent | control1 | None              | :-)   | UP    | neutron-linuxbridge-agent |
| c7187dc2-eea3-4fb6-a3f6-1919b82ced5b | Linux bridge agent | control2 | None              | :-)   | UP    | neutron-linuxbridge-agent |
| f0a396d3-8200-41c3-9057-5d609204be3f | Metadata agent     | control2 | None              | :-)   | UP    | neutron-metadata-agent    |
+--------------------------------------+--------------------+----------+-------------------+-------+-------+---------------------------+
  • All agents should be :-) and UP.

  • If you set overlay_iface_name to null, there is no ‘L3 agent’ in Agent Type column.

  • If you set is_ovs to false, there should be ‘Linux bridge agent’ in Agent Type column.

  • If you set is_ovs to true, there should be ‘Open vSwitch agent’ in Agent Type column.

Check openstack compute service status.:

root@btx-0:/# openstack compute service list
+--------------------------------------+----------------+---------------------------------+----------+---------+-------+----------------------------+
| ID                                   | Binary         | Host                            | Zone     | Status  | State | Updated At                 |
+--------------------------------------+----------------+---------------------------------+----------+---------+-------+----------------------------+
| b31c814b-d210-4e52-9d6e-59090f8a641a | nova-scheduler | nova-scheduler-5bcc764f79-wkfgl | internal | enabled | up    | 2023-05-31T12:16:20.000000 |
| 872555ad-dd52-46ce-be01-1ec7f8af9cd9 | nova-conductor | nova-conductor-56dfd9749-fn9xb  | internal | enabled | up    | 2023-05-31T12:16:21.000000 |
| ff3710b8-f110-4949-b578-b09a1dbc19bb | nova-scheduler | nova-scheduler-5bcc764f79-5hcvx | internal | enabled | up    | 2023-05-31T12:16:21.000000 |
| d6831741-677e-471f-a019-66b46150cbcc | nova-scheduler | nova-scheduler-5bcc764f79-sfclc | internal | enabled | up    | 2023-05-31T12:16:20.000000 |
| 792ec442-5e04-4a5f-9646-7cb0001dfb9c | nova-conductor | nova-conductor-56dfd9749-s5c6j  | internal | enabled | up    | 2023-05-31T12:16:21.000000 |
| 848f1573-3706-49ab-8c57-d6edf1631dce | nova-conductor | nova-conductor-56dfd9749-dfkgd  | internal | enabled | up    | 2023-05-31T12:16:21.000000 |
| c5217922-bc1d-446e-a951-a4871d6020e3 | nova-compute   | compute2                        | nova     | enabled | up    | 2023-05-31T12:16:25.000000 |
| 5f8cbde0-3c5f-404c-b31e-da443c1f14fd | nova-compute   | compute1                        | nova     | enabled | up    | 2023-05-31T12:16:25.000000 |
+--------------------------------------+----------------+---------------------------------+----------+---------+-------+----------------------------+
  • All services should be enabled and up.

  • Each compute node should have nova-compute service.

Test

The command “btx –test”

  • Creates a provider network and subnet. When it creates a provider network, it will ask for an address pool range.

  • Creates a cirros image.

  • Adds security group rules.

  • Creates a flavor.

  • Creates an instance.

  • Creates a volume.

  • Attaches a volume to an instance.

If everything goes well, the output looks like this.:

$ btx --test
...
Creating provider network...
Type the provider network address (e.g. 192.168.22.0/24): 192.168.22.0/24
Okay. I got the provider network address: 192.168.22.0/24
The first IP address to allocate (e.g. 192.168.22.100): 192.168.22.100
Okay. I got the first address in the pool: 192.168.22.100
The last IP address to allocate (e.g. 192.168.22.200): 192.168.22.108
Okay. I got the last address of provider network pool: 192.168.22.108
...
Instance status
+------------------+------------------------------------------------------------------------------------+
| Field            | Value                                                                              |
+------------------+------------------------------------------------------------------------------------+
| addresses        | public-net=192.168.22.104                                                          |
| flavor           | disk='1', ephemeral='0', , original_name='m1.tiny', ram='512', swap='0', vcpus='1' |
| image            | cirros (0b2787c1-fdb3-4a3c-ba9d-80208346a85c)                                      |
| name             | test                                                                               |
| status           | ACTIVE                                                                             |
| volumes_attached | delete_on_termination='False', id='76edcae9-4b17-4081-8a23-26e4ad13787f'           |
+------------------+------------------------------------------------------------------------------------+

Connect to the instance via provider network ip using ssh on the machine that has a provider network access.:

(a node on provider network)$ ssh cirros@192.168.22.104
cirros@192.168.22.104's password:
$ ip address show dev eth0
2: eth0:<BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast qlen 1000
    link/ether fa:16:3e:ed:bc:7b brd ff:ff:ff:ff:ff:ff
    inet 192.168.22.104/24 brd 192.168.22.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feed:bc7b/64 scope link
       valid_lft forever preferred_lft forever

Password is the default cirros password. (hint: password seems to be created by someone who loves Chicago Cubs baseball team.)