Chapter 8. Monitoring high availbility services

Galera CR’s status records the status of a Galera cluster’s startup. The CR’s conditions report the status of the prerequisites that Galera pods need to start as shown in the following example:

Status:
  Conditions:
	Last Transition Time:  2024-04-22T07:32:06Z
	Message:           	Setup complete
	Reason:            	Ready
	Status:            	True
	Type:              	Ready
	Last Transition Time:  2024-04-22T07:31:49Z
	Message:           	Deployment completed
	Reason:            	Ready
	Status:            	True
	Type:              	DeploymentReady
	Last Transition Time:  2024-04-22T07:31:11Z
	Message:           	Exposing service completed
	Reason:            	Ready
	Status:            	True
	Type:              	ExposeServiceReady
	Last Transition Time:  2024-04-22T07:31:11Z
	Message:           	Input data complete
	Reason:            	Ready
	Status:            	True
	Type:              	InputReady
	Last Transition Time:  2024-04-22T07:31:11Z
	Message:           	RoleBinding created
	Reason:            	Ready
	Status:            	True
	Type:              	RoleBindingReady
	Last Transition Time:  2024-04-22T07:31:11Z
	Message:           	Role created
	Reason:            	Ready
	Status:            	True
	Type:              	RoleReady
	Last Transition Time:  2024-04-22T07:31:11Z
	Message:           	ServiceAccount created
	Reason:            	Ready
	Status:            	True
	Type:              	ServiceAccountReady
	Last Transition Time:  2024-04-22T07:31:11Z
	Message:           	Service config create completed
	Reason:            	Ready
	Status:            	True
	Type:              	ServiceConfigReady
	Last Transition Time:  2024-04-22T07:31:11Z
	Message:           	Input data complete
	Reason:            	Ready
	Status:            	True
	Type:              	TLSInputReady

When the mariadb operator bootstraps a Galera cluster, it gathers information from every database replica, and then stores it in transient attributes. The transient attributes appear in the Galera CR’s status if the cluster is being inspected while the Galera cluster is stopped and being restarted:

Status:
  Attributes:
	openstack-galera-0:
  	Seqno:  1232
	openstack-galera-1:
  	Container ID:  cri-o://f56ec2389e878b462a54f5255dad83db29daf4d8e8cda338904bfd353b370165
  	Gcomm:     	gcomm://
  	Seqno:     	1232
	openstack-galera-2:
  	Seqno: 	1231
  Bootstrapped:  false

Before starting a Galera Cluster, the MariaDB operator starts all Galera pod replicas in a waiting state. Even if you can see the pods using oc get pods command, they have not started mysqld servers yet. The mariadb operator introspects the content of each pod’s database copy to extract the database sequence number (Seqno). Once the mariadb operator retrieves all of the pods’ Seqno information, it decides which pod holds the most recent version of the database and bootstraps a new Galera cluster from this pod. This pod starts a mysqld server and a transient attribute Gcomm:// appears in the Galera CR’s status. When the first mysqld server is ready to serve traffic, the attribute Bootstrapped becomes true, and transient Attributes for this pod are removed from the Galera CR’s status.

The state and the service availability of the rabbitmq-cluster operator and the rabbitmq clusters are exposed in the output of the Rabbitmq CR.

$ oc get pods -l app.kubernetes.io/name=rabbitmq
NAME            	READY   STATUS	RESTARTS  	AGE
rabbitmq-server-0   1/1 	Running   0                46h
rabbitmq-server-1   1/1 	Running   0                46h
rabbitmq-server-2   1/1 	Running   0                46h

The rabbitmq-cluster operator creates two Openstack service objects for a rabbitmq cluster. One service provides a DNS name resolution to the rabbitmq servers for internal rabbitmq communication. The RabbitMQ messaging service is exposed using an Openshift service managed by MetalLB:

$ oc get service -l app.kubernetes.io/name=rabbitmq
NAME         	TYPE       	CLUSTER-IP  	EXTERNAL-IP   PORT(S)                   	AGE
rabbitmq     	LoadBalancer   172.30.170.63   172.17.0.85   5671:31331/TCP,15671:31404/TCP,15691:31453/TCP   47h
rabbitmq-nodes   ClusterIP  	None        	<none>    	4369/TCP,25672/TCP                           	47h

For example, this MetalLB-managed service is called rabbitmq. This service acts as a load balancer across the RabbitMQ pods. It has an IP address that listens to the internal API network so it is accessible from the Openstack dataplane and controlplane. The MetalLB receives incoming traffic from the internal API on 172.17.0.85, and forwards it to the service’s IP 172.30.170.63 which balances traffic to rabbitmq pods:

$ oc describe service rabbitmq
Name:                 	rabbitmq
Namespace:            	openstack
Labels:               	app.kubernetes.io/component=rabbitmq
                      	app.kubernetes.io/name=rabbitmq
                      	app.kubernetes.io/part-of=rabbitmq
Annotations:          	dnsmasq.network.openstack.org/hostname: rabbitmq.openstack.svc
                      	metallb.universe.tf/address-pool: internalapi
                      	metallb.universe.tf/ip-allocated-from-pool: internalapi
                      	metallb.universe.tf/loadBalancerIPs: 172.17.0.85
Selector:             	app.kubernetes.io/name=rabbitmq
Type:                 	LoadBalancer
IP Family Policy:     	SingleStack
IP Families:          	IPv4
IP:                   	172.30.170.63
IPs:                  	172.30.170.63
LoadBalancer Ingress: 	172.17.0.85
Port:                 	amqps  5671/TCP
TargetPort:           	5671/TCP
NodePort:             	amqps  31331/TCP
Endpoints:            	192.168.16.69:5671,192.168.20.54:5671,192.168.24.45:5671

$ oc get memcached
NAME        READY   MESSAGE
memcached   True	Setup complete

When a memached CR is marked as Ready, its associated pods are started and ready to accept traffic. For example, here is a memcached cluster that is ready to accept traffic:

$ oc get pods -l memcached/name=memcached
NAME      	READY   STATUS	RESTARTS   AGE
memcached-0   1/1 	Running   0      	2d4h
memcached-1   1/1 	Running   0      	15m
memcached-2   1/1 	Running   0      	15m

$ oc get service memcached
NAME        TYPE        CLUSTER-IP   EXTERNAL-IP     PORT(S)           	  AGE
memcached   ClusterIP   None     	    <none>          11211/TCP,11212/TCP   2d4h

The memcached pods are accessed directly by name through the Openstack components. The memcached service is used only to maintain a list of DNS records for each memcached pod.

The following example shows the basic information that you can retrieve about an operator:

Name:           	openstack-galera
Namespace:      	openstack
CreationTimestamp:  Thu, 21 Mar 2024 08:39:59 -0400
Selector:       	app=galera,cr=galera-openstack,galera/name=openstack,galera/namespace=openstack,galera/uid=1c93b3a3-1ac3-4f18-984d-34e9ce9dc12f,owner=mariadb-operator
Labels:         	<none>
Annotations:    	<none>
Replicas:       	3 desired | 3 total
Update Strategy:	RollingUpdate
  Partition:    	0
Pods Status:    	3 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       	app=galera
                	cr=galera-openstack
                	galera/name=openstack
                	galera/namespace=openstack
                	galera/uid=1c93b3a3-1ac3-4f18-984d-34e9ce9dc12f
                	owner=mariadb-operator
  Service Account:  galera-openstack
  Init Containers:
   mysql-bootstrap:
	Image:  	quay.io/podified-antelope-centos9/openstack-mariadb@sha256:7fa37f7dcdd850fb6e401c4d5f76d16ad53ecdd14d6a130dbf61f02b819dcdf6
	Port:   	<none>
	Host Port:  <none>
	Command:
  	bash
  	/var/lib/operator-scripts/mysql_bootstrap.sh
	Environment:
  	KOLLA_BOOTSTRAP:    	True
  	KOLLA_CONFIG_STRATEGY:  COPY_ALWAYS
  	DB_ROOT_PASSWORD:   	<set to the key 'DbRootPassword' in secret 'osp-secret'>  Optional: false
	Mounts:
  	/var/lib/config-data/default from config-data-default (ro)
  	/var/lib/config-data/generated from config-data-generated (rw)
  	/var/lib/kolla/config_files from kolla-config (ro)
  	/var/lib/mysql from mysql-db (rw)
  	/var/lib/operator-scripts from operator-scripts (ro)
  	/var/lib/secrets from secrets (ro)
... [cont]

The following example shows the information about the actual container that you can retrieve about an operator:

 Containers:
   galera:
	Image:   	quay.io/podified-antelope-centos9/openstack-mariadb@sha256:7fa37f7dcdd850fb6e401c4d5f76d16ad53ecdd14d6a130dbf61f02b819dcdf6
	Ports:   	3306/TCP, 4567/TCP
	Host Ports:  0/TCP, 0/TCP
	Command:
  	/usr/bin/dumb-init
  	--
  	/usr/local/bin/kolla_start
	Liveness:   exec [/bin/bash /var/lib/operator-scripts/mysql_probe.sh liveness] delay=0s timeout=1s period=10s #success=1 #failure=3
	Readiness:  exec [/bin/bash /var/lib/operator-scripts/mysql_probe.sh readiness] delay=0s timeout=1s period=10s #success=1 #failure=3
	Startup:	exec [/bin/bash /var/lib/operator-scripts/mysql_probe.sh startup] delay=0s timeout=1s period=10s #success=1 #failure=30
	Environment:
  	CR_CONFIG_HASH:     	n558hf6h557hcfh589h688h684hb6h687h679h659h554h64fh77h76h568h695h5b6h8fh79h5c8h648h674hdch556h56bh655h64bh655h66ch5h5c4q
  	KOLLA_CONFIG_STRATEGY:  COPY_ALWAYS
  	DB_ROOT_PASSWORD:   	<set to the key 'DbRootPassword' in secret 'osp-secret'>  Optional: false
	Mounts:
  	/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem from combined-ca-bundle (ro,path="tls-ca-bundle.pem")
  	/var/lib/config-data/default from config-data-default (ro)
  	/var/lib/config-data/generated from config-data-generated (rw)
  	/var/lib/config-data/tls/certs/galera.crt from galera-tls-certs (ro,path="tls.crt")
  	/var/lib/config-data/tls/private/galera.key from galera-tls-certs (ro,path="tls.key")
  	/var/lib/kolla/config_files from kolla-config (ro)
  	/var/lib/mysql from mysql-db (rw)
  	/var/lib/operator-scripts from operator-scripts (ro)
  	/var/lib/secrets from secrets (ro)
... [cont]

The following example shows the information about the volumes of a service that you can retrieve about an operator:

 Volumes:
   secrets:
	Type:    	Secret (a volume populated by a Secret)
	SecretName:  osp-secret
	Optional:	false
   kolla-config:
	Type:  	ConfigMap (a volume populated by a ConfigMap)
	Name:  	openstack-config-data
	Optional:  false
   config-data-generated:
	Type:   	EmptyDir (a temporary directory that shares a pod's lifetime)
	Medium:
	SizeLimit:  <unset>
   config-data-default:
	Type:  	ConfigMap (a volume populated by a ConfigMap)
	Name:  	openstack-config-data
	Optional:  false
   operator-scripts:
	Type:  	ConfigMap (a volume populated by a ConfigMap)
	Name:  	openstack-scripts
	Optional:  false
   galera-tls-certs:
	Type:    	Secret (a volume populated by a Secret)
	SecretName:  cert-galera-openstack-svc
	Optional:	false
   combined-ca-bundle:
	Type:    	Secret (a volume populated by a Secret)
	SecretName:  combined-ca-bundle
	Optional:	false
Volume Claims:
  Name:      	mysql-db
  StorageClass:  local-storage
  Labels:    	app=galera
             	cr=galera-openstack
             	galera/name=openstack
             	galera/namespace=openstack
             	galera/uid=1c93b3a3-1ac3-4f18-984d-34e9ce9dc12f
             	owner=mariadb-operator
  Annotations:   <none>
  Capacity:  	5G
  Access Modes:  [ReadWriteOnce]
... [cont]

The following example shows the Event details that you can retrieve about an operator:

Events:
  Type	Reason        	Age   From                	Message
  ----	------        	----  ----                	-------
  Normal  SuccessfulCreate  179m  statefulset-controller  create Claim mysql-db-openstack-galera-0 Pod openstack-galera-0 in statefulset openstack-galera success
  Normal  SuccessfulCreate  179m  statefulset-controller  create Pod openstack-galera-0 in statefulset openstack-galera successful
  Normal  SuccessfulCreate  179m  statefulset-controller  create Claim mysql-db-openstack-galera-1 Pod openstack-galera-1 in statefulset openstack-galera success
  Normal  SuccessfulCreate  179m  statefulset-controller  create Claim mysql-db-openstack-galera-2 Pod openstack-galera-2 in statefulset openstack-galera success
  Normal  SuccessfulCreate  179m  statefulset-controller  create Pod openstack-galera-1 in statefulset openstack-galera successful
  Normal  SuccessfulCreate  179m  statefulset-controller  create Pod openstack-galera-2 in statefulset openstack-galera successful

$ oc describe pod/rabbitmq-server-0
Name:         	rabbitmq-server-0
Namespace:    	openstack
Priority:     	0
Service Account:  rabbitmq-server
Node:         	master-2/192.168.111.22
Start Time:   	Thu, 21 Mar 2024 08:39:57 -0400
Labels:       	app.kubernetes.io/component=rabbitmq
              	app.kubernetes.io/name=rabbitmq
              	app.kubernetes.io/part-of=rabbitmq
              	controller-revision-hash=rabbitmq-server-5c886b79b4
              	statefulset.kubernetes.io/pod-name=rabbitmq-server-0
Annotations:  	k8s.ovn.org/pod-networks:
                	{"default":{"ip_addresses":["192.168.16.35/22"],"mac_address":"0a:58:c0:a8:10:23","gateway_ips":["192.168.16.1"],"routes":[{"dest":"192.16...
              	k8s.v1.cni.cncf.io/network-status:
                	[{
                    	"name": "ovn-kubernetes",
                    	"interface": "eth0",
                    	"ips": [
                        	"192.168.16.35"
                    	],
                    	"mac": "0a:58:c0:a8:10:23",
                    	"default": true,
                    	"dns": {}
                	}]
              	openshift.io/scc: restricted-v2
              	seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:       	Running
...

By default, The Taint-Based Evictions feature evicts pods from a node that experiences specific conditions like not-ready and unreachable. When a node experiences one of these conditions, OCP adds taints to the node, evicts the pods, and then reschedules the pods on different nodes.

Also, Taint-Based Evictions have a NoExecute effect. Any pod that does not tolerate the taint is evicted immediately and any pod that does tolerate the taint will never be evicted, unless the pod uses the tolerationSeconds parameter.

Use the tolerationSeconds parameter to specify how long a pod stays bound to a node that has a node condition. If the condition still exists after the tolerationSeconds period, the taint remains on the node and the pods with a matching toleration are evicted. If the condition clears before the tolerationSeconds period, pods with matching tolerations are not removed.

OpenShift Container Platform adds a toleration for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable with tolerationSeconds=300, unless the Pod configuration specifies either toleration.

For more information about Remediation, fencing, and maintenance, see Remediation, fencing, and maintenance