Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 13. Troubleshooting and maintaining the Load-balancing service

To diagnose and maintain the Red Hat OpenStack Services on OpenShift (RHOSO) Load-balancing service (octavia), use OpenStack client commands to show status, migrate instances, and access logs. For additional troubleshooting, you can SSH into one or more Load-balancing service instances (amphorae).

13.1. Verifying the load balancer
Copy link

You can troubleshoot the Red Hat OpenStack Services on OpenShift (RHOSO) Load-balancing service (octavia) and its various components by viewing the output of the load balancer show and list commands.

Prerequisites

  • The administrator has created a project for you and has provided you with a clouds.yaml file for you to access the cloud.

  • The python-openstackclient package resides on your workstation. 

    $ dnf list installed python-openstackclient

Procedure

  1. Confirm that the system OS_CLOUD variable is set for your cloud: 

    $ echo $OS_CLOUD
my_cloud

    Reset the variable if necessary: 

    $ export OS_CLOUD=my_other_cloud

    As an alternative, you can specify the cloud name by adding the --os-cloud <cloud_name> option each time you run an openstack command.

  2. Verify the load balancer (lb1) settings.

    Note

    Values inside parentheses are sample values that are used in the example commands in this procedure. Substitute these sample values with values that are appropriate for your site.

    Example
    $ openstack loadbalancer show lb1
    Sample output
    +---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2024-02-17T15:59:18                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 265d0b71-c073-40f4-9718-8a182c6d53ca |
| listeners           | 5aaa67da-350d-4125-9022-238e0f7b7f6f |
| name                | lb1                                  |
| operating_status    | ONLINE                               |
| pools               | 48f6664c-b192-4763-846a-da568354da4a |
| project_id          | 52376c9c5c2e434283266ae7cacd3a9c     |
| provider            | amphora                              |
| provisioning_status | ACTIVE                               |
| updated_at          | 2024-02-17T16:01:21                  |
| vip_address         | 192.0.2.177                          |
| vip_network_id      | afeaf55e-7128-4dff-80e2-98f8d1f2f44c |
| vip_port_id         | 94a12275-1505-4cdc-80c9-4432767a980f |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 06ffa90e-2b86-4fe3-9731-c7839b0be6de |
+---------------------+--------------------------------------+

  3. Using the loadbalancer ID (265d0b71-c073-40f4-9718-8a182c6d53ca) from the previous step, obtain the ID of the amphora associated with the load balancer (lb1).

    Example
    $ openstack loadbalancer amphora list | grep 265d0b71-c073-40f4-9718-8a182c6d53ca
    Sample output
    | 1afabefd-ba09-49e1-8c39-41770aa25070 | 265d0b71-c073-40f4-9718-8a182c6d53ca | ALLOCATED | STANDALONE | 198.51.100.7  | 192.0.2.177   |

  4. Using the amphora ID (1afabefd-ba09-49e1-8c39-41770aa25070) from the previous step, view amphora information.

    Example
    $ openstack loadbalancer amphora show 1afabefd-ba09-49e1-8c39-41770aa25070
    Sample output
    +-----------------+--------------------------------------+
| Field           | Value                                |
+-----------------+--------------------------------------+
| id              | 1afabefd-ba09-49e1-8c39-41770aa25070 |
| loadbalancer_id | 265d0b71-c073-40f4-9718-8a182c6d53ca |
| compute_id      | ba9fc1c4-8aee-47ad-b47f-98f12ea7b200 |
| lb_network_ip   | 198.51.100.7                         |
| vrrp_ip         | 192.0.2.36                           |
| ha_ip           | 192.0.2.177                          |
| vrrp_port_id    | 07dcd894-487a-48dc-b0ec-7324fe5d2082 |
| ha_port_id      | 94a12275-1505-4cdc-80c9-4432767a980f |
| cert_expiration | 2026-03-19T15:59:23                  |
| cert_busy       | False                                |
| role            | STANDALONE                           |
| status          | ALLOCATED                            |
| vrrp_interface  | None                                 |
| vrrp_id         | 1                                    |
| vrrp_priority   | None                                 |
| cached_zone     | nova                                 |
| created_at      | 2024-02-17T15:59:22                  |
| updated_at      | 2024-02-17T16:00:50                  |
| image_id        | 53001253-5005-4891-bb61-8784ae85e962 |
| compute_flavor  | 65                                   |
+-----------------+--------------------------------------+

  5. View the listener (listener1) details.

    Example
    $ openstack loadbalancer listener show listener1
    Sample output
    +-----------------------------+--------------------------------------+
| Field                       | Value                                |
+-----------------------------+--------------------------------------+
| admin_state_up              | True                                 |
| connection_limit            | -1                                   |
| created_at                  | 2024-02-17T16:00:59                  |
| default_pool_id             | 48f6664c-b192-4763-846a-da568354da4a |
| default_tls_container_ref   | None                                 |
| description                 |                                      |
| id                          | 5aaa67da-350d-4125-9022-238e0f7b7f6f |
| insert_headers              | None                                 |
| l7policies                  |                                      |
| loadbalancers               | 265d0b71-c073-40f4-9718-8a182c6d53ca |
| name                        | listener1                            |
| operating_status            | ONLINE                               |
| project_id                  | 52376c9c5c2e434283266ae7cacd3a9c     |
| protocol                    | HTTP                                 |
| protocol_port               | 80                                   |
| provisioning_status         | ACTIVE                               |
| sni_container_refs          | []                                   |
| timeout_client_data         | 50000                                |
| timeout_member_connect      | 5000                                 |
| timeout_member_data         | 50000                                |
| timeout_tcp_inspect         | 0                                    |
| updated_at                  | 2024-02-17T16:01:21                  |
| client_ca_tls_container_ref | None                                 |
| client_authentication       | NONE                                 |
| client_crl_container_ref    | None                                 |
| allowed_cidrs               | None                                 |
+-----------------------------+--------------------------------------+

  6. View the pool (pool1) and load-balancer members.

    Example
    $ openstack loadbalancer pool show pool1
    Sample output
    +----------------------+--------------------------------------+
| Field                | Value                                |
+----------------------+--------------------------------------+
| admin_state_up       | True                                 |
| created_at           | 2024-02-17T16:01:08                  |
| description          |                                      |
| healthmonitor_id     | 4b24180f-74c7-47d2-b0a2-4783ada9a4f0 |
| id                   | 48f6664c-b192-4763-846a-da568354da4a |
| lb_algorithm         | ROUND_ROBIN                          |
| listeners            | 5aaa67da-350d-4125-9022-238e0f7b7f6f |
| loadbalancers        | 265d0b71-c073-40f4-9718-8a182c6d53ca |
| members              | b92694bd-3407-461a-92f2-90fb2c4aedd1 |
|                      | 4ccdd1cf-736d-4b31-b67c-81d5f49e528d |
| name                 | pool1                                |
| operating_status     | ONLINE                               |
| project_id           | 52376c9c5c2e434283266ae7cacd3a9c     |
| protocol             | HTTP                                 |
| provisioning_status  | ACTIVE                               |
| session_persistence  | None                                 |
| updated_at           | 2024-02-17T16:05:21                  |
| tls_container_ref    | None                                 |
| ca_tls_container_ref | None                                 |
| crl_container_ref    | None                                 |
| tls_enabled          | False                                |
+----------------------+--------------------------------------+

  7. Verify HTTPS traffic flows across a load balancer whose listener is configured for HTTPS or TERMINATED_HTTPS protocols by connecting to the VIP address (192.0.2.177) of the load balancer.

    Tip

    Obtain the load-balancer VIP address by using the command, openstack loadbalancer show <load_balancer_name>.

    Note

    Security groups implemented for the load balancer VIP only allow data traffic for the required protocols and ports. For this reason you cannot ping load balancer VIPs, because ICMP traffic is blocked.

    Example
    $ curl -v https://192.0.2.177 --insecure
    Sample output
    * About to connect() to 192.0.2.177 port 443 (#0)
*   Trying 192.0.2.177...
* Connected to 192.0.2.177 (192.0.2.177) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* 	subject: CN=www.example.com,O=Dis,L=Springfield,ST=Denial,C=US
* 	start date: Jan 15 09:21:45 2024 GMT
* 	expire date: Jan 15 09:21:45 2027 GMT
* 	common name: www.example.com
* 	issuer: CN=www.example.com,O=Dis,L=Springfield,ST=Denial,C=US
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 192.0.2.177
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Length: 30
<
* Connection #0 to host 192.0.2.177 left intact

In some cases you must migrate a Load-balancing service instance (amphora). For example, if the host is being shut down for maintenance.

Prerequisites

  • You have the oc command line tool installed on your workstation.
  • You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges.

Procedure

  1. Access the remote shell for the OpenStackClient pod from your workstation: 

    $ oc rsh -n openstack openstackclient

  2. Locate the ID of the amphora that you want to migrate. You need to provide the ID in a later step. 

    $ openstack loadbalancer amphora list

  3. To prevent the Compute scheduler service from scheduling any new amphorae to the Compute node being evacuated, disable the Compute node (compute-host-1).

    Note

    Values inside parentheses are sample values that are used in the example commands in this procedure. Substitute these sample values with values that are appropriate for your site.

    Example
    $ openstack compute service set compute-host-1 nova-compute --disable

  4. Fail over the amphora by using the amphora ID (ea17210a-1076-48ff-8a1f-ced49ccb5e53) that you obtained.

    Example
    $ openstack loadbalancer amphora failover ea17210a-1076-48ff-8a1f-ced49ccb5e53

  5. Exit the openstackclient pod: 

    $ exit

13.3. Showing listener statistics
Copy link

Using the OpenStack Client, you can obtain statistics about the listener for a particular Red Hat OpenStack Services on OpenShift (RHOSO) loadbalancer:

  • current active connections (active_connections).
  • total bytes received (bytes_in).
  • total bytes sent (bytes_out).
  • total requests that were unable to be fulfilled (request_errors).
  • total connections handled (total_connections).

Prerequisites

  • The administrator has created a project for you and has provided you with a clouds.yaml file for you to access the cloud.

  • The python-openstackclient package resides on your workstation. 

    $ dnf list installed python-openstackclient

Procedure

  1. Confirm that the system OS_CLOUD variable is set for your cloud: 

    $ echo $OS_CLOUD
my_cloud

    Reset the variable if necessary: 

    $ export OS_CLOUD=my_other_cloud

    As an alternative, you can specify the cloud name by adding the --os-cloud <cloud_name> option each time you run an openstack command.

  2. View the stats for the listener (listener1).

    Note

    Values inside parentheses are sample values that are used in the example commands in this procedure. Substitute these sample values with values that are appropriate for your site.

    Example
    $ openstack loadbalancer listener stats show listener1
    Tip

    If you do not know the name of the listener, enter the command loadbalancer listener list.

    Sample output
    +--------------------+-------+
| Field              | Value |
+--------------------+-------+
| active_connections | 0     |
| bytes_in           | 0     |
| bytes_out          | 0     |
| request_errors     | 0     |
| total_connections  | 0     |
+--------------------+-------+

13.4. Interpreting listener request errors
Copy link

You can obtain statistics about the listener for a particular Red Hat OpenStack Services on OpenShift (RHOSO) loadbalancer.

One of the statistics tracked by the RHOSO loadbalancer, request_errors, is only counting errors that occurred in the request from the end user connecting to the load balancer. The request_errors variable is not measuring errors reported by the member server.

For example, if a tenant connects through the RHOSO Load-balancing service (octavia) to a web server that returns an HTTP status code of 400 (Bad Request), this error is not collected by the Load-balancing service. Loadbalancers do not inspect the content of data traffic. In this example, the loadbalancer interprets this flow as successful because it transported information between the user and the web server correctly.

The following conditions can cause the request_errors variable to increment:

  • early termination from the client, before the request has been sent.
  • read error from the client.
  • client timeout.
  • client closed the connection.
  • various bad requests from the client.

In Red Hat OpenStack Services on OpenShift (RHOSO) 18.0.10, you can use a new OVN database synchronization tool to fix OVN load balancers that experience problems caused by:

  • Inconsistencies between Octavia and OVN.
  • Restoration or recreation of the OVN database.
  • Migration or repair of Load-balancing service (octavia) resources.
  • Failure of the OVN database cluster

The new tool, octavia-ovn-db-sync-util is run on the command-line to synchronize the state of Load-balancing service (octavia) resources, with the OVN databases.

Important

The octavia-ovn-db-sync-util is designed to only work on load balancers that use the OVN provider driver. Do not use octavia-ovn-db-sync-util on load balancers that use the amphora provider driver.

Prerequisites

  • You have the oc command line tool installed on your workstation.
  • You are logged on to a workstation that has access to the RHOSO control plane as a user with cluster-admin privileges.

Procedure

  • Run octavia-ovn-db-sync-util: 

    $ oc exec deployments/octavia-api -- octavia-ovn-db-sync-util
    Sample output

    You should see output similar to the following: 

    INFO ovn_octavia_provider.cmd.octavia_ovn_db_sync_util [-] OVN Octavia DB sync start.
INFO ovn_octavia_provider.driver [-] Starting sync OVN DB with Loadbalancer filter {'provider': 'ovn'}
INFO ovn_octavia_provider.driver [-] Starting sync OVN DB with Loadbalancer lb1
DEBUG ovn_octavia_provider.driver [-] OVN loadbalancer 5bcaab92-3f8e-4460-b34d-4437a86909ef not found. Start create process. {{(pid=837681) _ensure_loadbalancer /opt/stack/ovn-octavia-provider/ovn_octavia_provider/driver.py:684}}
DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbCreateCommand(_result=None, table=Load_Balancer, columns={'name': '5bcaab92-3f8e-4460-b34d-4437a86909ef', 'protocol': [], 'external_ids': {'neutron:vip': '192.168.100.188', 'neutron:vip_port_id': 'e60041e8-01e8-459b-956e-a55608eb5255', 'enabled': 'True'}, 'selection_fields': ['ip_src', 'ip_dst', 'tp_src', 'tp_dst']}, row=False) {{(pid=837681) do_commit /opt/stack/ovn-octavia-provider/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}}
DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): LsLbAddCommand(_result=None, switch=000a1a3e-edff-45ad-9241-5ab8894ac0e0, lb=d69e29cd-0069-4d7f-a1ed-08c246bfb3da, may_exist=True) {{(pid=837681) do_commit /opt/stack/ovn-octavia-provider/venv/lib/python3.12/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:89}}
INFO ovn_octavia_provider.driver [-] Starting sync floating IP for loadbalancer 5bcaab92-3f8e-4460-b34d-4437a86909ef
WARNING ovn_octavia_provider.driver [-] Floating IP not found for loadbalancer 5bcaab92-3f8e-4460-b34d-4437a86909ef
...

Verification

  • When you see the following output, the database synchronization for your OVN load balancers is complete:

    Sample output
    INFO ovn_octavia_provider.cmd.octavia_ovn_db_sync_util [-] OVN Octavia DB sync finish.
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top