Chapter 3. RHOSO RabbitMQ clusters

The RabbitMQ interface for OpenStack services in Red Hat OpenStack Services on OpenShift (RHOSO) has been expanded to improve the isolation and scalability of RabbitMQ communication.

The initial implementation of the RabbitMQ interface for OpenStack services consisted of a single setting in the OpenStackControlPlane CR to specify the RabbitMQ cluster. The RabbitMQ interface has been revised into an expandable structure, which in addition to specifying the RabbitMQ cluster, you can specify a dedicated RabbitMQ user and a dedicated RabbitMQ virtual host for both of the communication methods facilitated by RabbitMQ.

OpenStack services use RabbitMQ for the following two distinct messaging purposes:

In a typical RHOSO deployment, the OpenStack services typically use the same RabbitMQ cluster for both RPC and notifications. Therefore you can set global defaults for the messagingBus and notificationsBus RabbitMQ settings in the OpenStackControlPlane CR to specify the default RabbitMQ cluster:

apiVersion: core.openstack.org/v1beta1
kind: OpenStackControlPlane
metadata:
  name: openstack
spec:
  messagingBus:
    cluster: <rabbitmq-cluster>

  notificationsBus:
    cluster: <rabbitmq-cluster>

where:

By default, RHOSO creates a shared RabbitMQ user for all OpenStack services. For security and isolation, you can create dedicated RabbitMQ users for your OpenStack services. For more information, see the Additional resources section below.

To specify service-specific RabbitMQ clusters or to create dedicated RabbitMQ users, you can add and configure one or both of the following standard RabbitMQ configuration sections to the template of each RHOSO service in the OpenStackControlPlane CR, depending upon the messaging requirements of each service:

...
  template:
    messagingBus:
      cluster: <rabbitmq-rpc-cluster>
      user: <rabbitmq-rpc-user>
      vhost: <rabbitmq-rpc-vhost-name>

where:

...
  template:
    notificationsBus:
      cluster: <rabbitmq-notif-cluster>
      user: <rabbitmq-notif-user>
      vhost: <rabbitmq-notif-vhost-name>

where:

Your deployment might require service specific RabbitMQ clusters in the following scenarios:

...
  nova:
  ...
    template:
      cellTemplates:
        cell0:
          ...
          messagingBus:
            cluster: rabbitmq  # Cell0 uses default RabbitMQ cluster
        cell1:
          ...
          messagingBus:
            cluster: rabbitmq-cell1  # Cell1 uses dedicated RabbitMQ cluster

  neutron:
    template:
      messagingBus:
        cluster: rabbitmq-neutron  # Dedicated RabbitMQ cluster for the Networking service
        vhost: neutron

The following example configures a dedicated RabbitMQ virtual host for the Block Storage (cinder) OpenStack service:

spec:
  messagingBus:
    cluster: rabbitmq  # Default for most services
...

  cinder:
    template:
      messagingBus:
        vhost: cinder-vhost  # Block Storage Cinder uses dedicated vhost

You can use the following commands to retrieve information about the state and service availability of the RabbitMQ clusters and their pods:

You can use the following command to determine the status of the RabbitMQ pods:

$ oc get pods -l app.kubernetes.io/name=rabbitmq
NAME            	READY   STATUS	RESTARTS  	AGE
rabbitmq-server-0   1/1 	Running   0                46h
rabbitmq-server-1   1/1 	Running   0                46h
rabbitmq-server-2   1/1 	Running   0                46h

The RabbitMQ service Operator rabbitmq-cluster-operator creates the following two OpenStack service objects for a RabbitMQ cluster:

You can use the following command to determine the type of these services and their ports:

$ oc get service -l app.kubernetes.io/name=rabbitmq
NAME         	TYPE       	CLUSTER-IP  	EXTERNAL-IP   PORT(S)                   	AGE
rabbitmq     	LoadBalancer   172.30.170.63   172.17.0.85   5671:31331/TCP,15671:31404/TCP,15691:31453/TCP   47h
rabbitmq-nodes   ClusterIP  	None        	<none>    	4369/TCP,25672/TCP                           	47h

The MetalLB-managed service, called rabbitmq, is a LoadBalancer across the RabbitMQ pods. It has an IP address that listens to the internal API network so it is accessible from the Red Hat OpenStack Services on OpenShift (RHOSO) data plane and control plane.

You can use the following command to determine the configuration of the rabbitmq service:

$ oc describe service rabbitmq
Name:                 	rabbitmq
Namespace:            	openstack
Labels:               	app.kubernetes.io/component=rabbitmq
                      	app.kubernetes.io/name=rabbitmq
                      	app.kubernetes.io/part-of=rabbitmq
Annotations:          	dnsmasq.network.openstack.org/hostname: rabbitmq.openstack.svc
                      	metallb.universe.tf/address-pool: internalapi
                      	metallb.universe.tf/ip-allocated-from-pool: internalapi
                      	metallb.universe.tf/loadBalancerIPs: 172.17.0.85
Selector:             	app.kubernetes.io/name=rabbitmq
Type:                 	LoadBalancer
IP Family Policy:     	SingleStack
IP Families:          	IPv4
IP:                   	172.30.170.63
IPs:                  	172.30.170.63
LoadBalancer Ingress: 	172.17.0.85
Port:                 	amqps  5671/TCP
TargetPort:           	5671/TCP
NodePort:             	amqps  31331/TCP
Endpoints:            	192.168.16.69:5671,192.168.20.54:5671,192.168.24.45:5671

In this example, this MetalLB-managed service called rabbitmq, receives incoming traffic from the internal API on 172.17.0.85, which is known as the LoadBalancer Ingress, and forwards it to its IP address of 172.30.170.63 which balances traffic to the RabbitMQ pods.