Chapter 6. Monitoring the Load-balancing service

The Load-balancing service (octavia) controller pods require network connectivity across the OpenStack cloud to monitor and manage amphora load-balancer virtual machines (VMs) using the load-balancing networks.

The load-balancing networks are:

An OpenStack router routes packets between the management network and the controller network with both the control plane pods and load-balancer VMs having routes configured to direct traffic through the router for those networks.

Running the following command, oc rsh openstack client openstack network list -f yaml yields output similar to the following:

- ID: 2e4fc309-546b-4ac8-9eae-aa8d70a27a9b
  Name: octavia-provider-net
  Subnets:
  - eea45073-6e56-47fd-9153-12f7f49bc115
- ID: 77881d3f-04b0-46cb-931f-d54003cce9f0
  Name: lb-mgmt-net
  Subnets:
  - e4ab96af-8077-4971-baa4-e0d40a16f55a

The octavia-provider-net network is the external provider network, and uses the octavia network attachment interface as the physical network. This network is limited to the OpenShift control plane. The lb-mgmt-net network is a self-serve tenant network that connects the Octavia amphora instances.

The amphora controllers do not have direct access to the lb-mgmt-net network. The controllers access the lb-mgmt-net network through the octavia network attachment and a router that the octavia-operator manages. You can view the subnets by running the command, oc rsh openstackclient subnet list -f yaml:

- ID: e4ab96af-8077-4971-baa4-e0d40a16f55a
  Name: lb-mgmt-subnet
  Network: 77881d3f-04b0-46cb-931f-d54003cce9f0
  Subnet: 172.24.0.0/16
- ID: eea45073-6e56-47fd-9153-12f7f49bc115
  Name: octavia-provider-subnet
  Network: 2e4fc309-546b-4ac8-9eae-aa8d70a27a9b
  Subnet: 172.23.0.0/24

The subnet CIDR for octavia-provider-subnet originates from the octavia network attachment. The Subnet CIDR of lb-mgmt-subnet originates from the dst field of the octavia network attachment routes.

The octavia-link-router manages the routing between the octavia-provider-net and lb-mgmt-net networks. To view the routers, run the command,oc rsh openstackclient openstack router list -f yaml:

- ID: 371d800c-c803-4210-836b-eb468654462a
  Name: octavia-link-router
  Project: dc65b54e9cba475ba0adba7f898060f2
  State: true
  Status: ACTIVE

You can view the configuration of the octavia-link-router by running the command, oc rsh openstackclient openstack router show -f yaml octavia-link-router:

admin_state_up: true
availability_zone_hints: []
availability_zones: []
created_at: '2024-06-11T17:20:57Z'
description: ''
enable_ndp_proxy: null
external_gateway_info:
  enable_snat: false
  external_fixed_ips:
  - ip_address: 172.23.0.150
    subnet_id: eea45073-6e56-47fd-9153-12f7f49bc115
  network_id: 2e4fc309-546b-4ac8-9eae-aa8d70a27a9b
flavor_id: null
id: 371d800c-c803-4210-836b-eb468654462a
interfaces_info:
- ip_address: 172.24.1.89
  port_id: 1a44e94d-f44a-4752-81db-bc5402857a08
  subnet_id: e4ab96af-8077-4971-baa4-e0d40a16f55a
name: octavia-link-router
project_id: dc65b54e9cba475ba0adba7f898060f2
revision_number: 4
routes: []
status: ACTIVE
tags: []
tenant_id: dc65b54e9cba475ba0adba7f898060f2
updated_at: '2024-06-11T17:21:01Z'

The external_gateway_info of the router corresponds to the gw field of the routes provided in the network attachment.

Notice that source network address translation is disabled. This is important as the amphora controllers communicate with the amphora by using the addresses on the lb-mgmt-net network that OpenStack allocates, not a floating IP address. The routes of the network attachment direct traffic from the amphora controllers to the router, and the host routes on the lb-mgmt-net subnet establish the reverse route. This host route uses the ip_address of the port in interfaces_info as the next_hop and the Subnet of the octavia-provider-subnet as the Destination to be routed to.

To view the host routes for the lb-mgmt-subnet, run the command, oc rsh openstackclient openstack subnet show lb-mgmt-subnet -c host_routes -f yaml:

host_routes:
- destination: 172.23.0.0/24
  nexthop: 172.24.1.89

The port used to connect lb-mgmt-subnet to the router is named lb-mgmt-router-port and you can view the details by running the command, oc rsh openstackclient openstack port show lb-mgmt-router-port -f yaml. Note that the port_id in the router’s interface_info can be used instead of the port name.

admin_state_up: true
allowed_address_pairs: []
binding_host_id: ''
binding_profile: {}
binding_vif_details: {}
binding_vif_type: unbound
binding_vnic_type: normal
created_at: '2024-06-11T17:20:41Z'
data_plane_status: null
description: ''
device_id: 371d800c-c803-4210-836b-eb468654462a
device_owner: network:router_interface
device_profile: null
dns_assignment:
- fqdn: host-172-24-1-89.openstackgate.local.
  hostname: host-172-24-1-89
  ip_address: 172.24.1.89
dns_domain: ''
dns_name: ''
extra_dhcp_opts: []
fixed_ips:
- ip_address: 172.24.1.89
  subnet_id: e4ab96af-8077-4971-baa4-e0d40a16f55a
id: 1a44e94d-f44a-4752-81db-bc5402857a08
ip_allocation: immediate
mac_address: fa:16:3e:ba:be:ee
name: lb-mgmt-router-port
network_id: 77881d3f-04b0-46cb-931f-d54003cce9f0
numa_affinity_policy: null
port_security_enabled: true
project_id: dc65b54e9cba475ba0adba7f898060f2
propagate_uplink_status: null
qos_network_policy_id: null
qos_policy_id: null
resource_request: null
revision_number: 3
security_group_ids:
- 055686ce-fb2d-409b-ab74-85df9ab3a9e0
- 5c41444b-0863-4609-9335-d5a66bdbcad8
status: ACTIVE
tags: []
trunk_details: null
updated_at: '2024-06-11T17:21:03Z'

Notice the following about these fields and their values:

In RHOSO environments, the Load-balancing service (octavia) monitors the load-balancing instances (amphorae) and initiates failovers and replacements if the amphorae malfunction.

Any time a failover occurs, the Load-balancing service logs the failover in the corresponding health manager log on the controller in /var/log/containers/octavia.

Use log analytics to monitor failover trends to address problems early. Problems such as Networking service (neutron) connectivity issues, denial of service attacks, and Compute service (nova) malfunctions often lead to higher failover rates for load balancers.

In Red Hat OpenStack Services on OpenShift (RHOSO) environments, the Load-balancing service (octavia) uses the health information from the underlying load balancing subsystems to determine the health of members of the load-balancing pool.

Health information is streamed to the Load-balancing service database, and made available by the status tree or other API methods. For critical applications, you must poll for health information in regular intervals.

In Red Hat OpenStack Services on OpenShift (RHOSO) environments, you can monitor the provisioning status of a load balancer and send alerts if the provisioning status is ERROR. Do not configure an alert to trigger when an application is making regular changes to the pool and enters several PENDING stages.

The provisioning status of load-balancer objects reflect the ability of the control plane to contact and successfully provision a create, update, and delete request. The operating status of a load-balancer object reports on the current functionality of the load balancer.

For example, a load balancer might have a provisioning status of ERROR, but an operating status of ONLINE. This might be caused by a Networking service (neutron) failure that blocked that last requested update to the load-balancer configuration from successfully completing. In this case, the load balancer continues to process traffic through the load balancer, but might not have applied the latest configuration updates yet.

You can monitor the operational status of your load balancer and its child objects in your Red Hat OpenStack Services on OpenShift (RHOSO) environment.

You can also use an external monitoring service that connects to your load-balancer listeners and monitors them from outside of the cloud. An external monitoring service indicates if there is a failure outside of the Load-balancing service (octavia) that might impact the functionality of your load balancer, such as router failures, network connectivity issues, and so on.

A Red Hat OpenStack Services on OpenShift (RHOSO) Load-balancing service (octavia) health monitor is a process that does periodic health checks on each back end member server to pre-emptively detect failed servers and temporarily pull them out of the pool.

If the health monitor detects a failed server, it removes the server from the pool and marks the member in ERROR. After you have corrected the server and it is functional again, the health monitor automatically changes the status of the member from ERROR to ONLINE, and resumes passing traffic to it.

Always use health monitors in production load balancers. If you do not have a health monitor, failed servers are not removed from the pool. This can lead to service disruption for web clients.

There are several types of health monitors, as briefly described here:

Use Load-balancing service (octavia) health monitors to avoid service disruptions. The health monitors run periodic health checks on each back end server to pre-emptively detect failed servers and temporarily remove the servers from the pool.

You can modify the configuration for Red Hat OpenStack Services on OpenShift (RHOSO) Load-balancing service (octavia) health monitors when you want to change the interval for sending probes to members, the connection timeout interval, the HTTP method for requests, and so on.

You can remove a Red Hat OpenStack Services on OpenShift (RHOSO) Load-balancing service (octavia) health monitor.

In RHOSO environments, when you write the code that generates the health check in your web application, you must consider situations that can cause cause your checks to fail.

Follow these guidelines when developing your health checks: