High Availability Deployment and Usage

Procedure

1. In a running HA environment, log in to the undercloud as the stack user.

2. Identify the IP addresses of your overcloud nodes:

   $ source ~/stackrc
   (undercloud) $ openstack server list
    +-------+------------------------+---+----------------------+---+
    | ID    | Name                   |...| Networks             |...|
    +-------+------------------------+---+----------------------+---+
    | d1... | overcloud-controller-0 |...| ctlplane=*10.200.0.11* |...|
    ...

3. Log in to one of the overcloud nodes:

   (undercloud) $ ssh tripleo-admin@<node_IP>

   Replace <node_ip> with the IP address of the node that you want to log in to.

Procedure

1. Log in to any Controller node as the tripleo-admin user.

   $ ssh tripleo-admin@overcloud-controller-0

2. Run the pcs status command:

   [tripleo-admin@overcloud-controller-0 ~]  $ sudo pcs status

   Example output:

   Cluster name: tripleo_cluster
   Stack: corosync
   Current DC: overcloud-controller-1 (version 2.0.1-4.el9-0eb7991564) - partition with quorum
   
   Last updated: Thu Feb  8 14:29:21 2018
   Last change: Sat Feb  3 11:37:17 2018 by root via cibadmin on overcloud-controller-2
   
   12 nodes configured
   37 resources configured
   
   Online: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]
   GuestOnline: [ galera-bundle-0@overcloud-controller-0 galera-bundle-1@overcloud-controller-1 galera-bundle-2@overcloud-controller-2 rabbitmq-bundle-0@overcloud-controller-0 rabbitmq-bundle-1@overcloud-controller-1 rabbitmq-bundle-2@overcloud-controller-2 redis-bundle-0@overcloud-controller-0 redis-bundle-1@overcloud-controller-1 redis-bundle-2@overcloud-controller-2 ]
   
   Full list of resources:
   [...]

   The main sections of the output show the following information about the cluster:

   • Cluster name: Name of the cluster.
   • [NUM] nodes configured: Number of nodes that are configured for the cluster.
   • [NUM] resources configured: Number of resources that are configured for the cluster.
   • Online: Names of the Controller nodes that are currently online.
   • GuestOnline: Names of the guest nodes that are currently online. Each guest node consists of a complex Bundle Set resource. For more information about bundle sets, see Section 3.1, “Pacemaker resource bundles and containers”.

Procedure

1. Log in to any Controller node as the tripleo-admin user.

   $ ssh tripleo-admin@overcloud-controller-0

2. Use one of the following options:

   • Show all resources that use virtual IPs by running the pcs resource show command with the --full option:

     $ sudo pcs resource show --full

     Example output:

      ip-10.200.0.6	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1
      ip-192.168.1.150	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0
      ip-172.16.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-1
      ip-172.16.0.11	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-0
      ip-172.18.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2
      ip-172.19.0.10	(ocf::heartbeat:IPaddr2):	Started overcloud-controller-2

     Each IP address is initially attached to a specific Controller node. For example, 192.168.1.150 is started on overcloud-controller-0. However, if that Controller node fails, the IP address is reassigned to other Controller nodes in the cluster.

     The following table describes the IP addresses in the example output and shows the original allocation of each IP address.

     Table 3.1. IP address description and allocation source

     IP Address	Description	Allocated From
     10.200.0.6	Controller virtual IP address	Part of the dhcp_start and dhcp_end range set to 10.200.0.5-10.200.0.24 in the undercloud.conf file
     192.168.1.150	Public IP address	ExternalAllocationPools attribute in the network-environment.yaml file
     172.16.0.10	Provides access to OpenStack API services on a Controller node	InternalApiAllocationPools in the network-environment.yaml file
     172.16.0.11	Provides access to Redis service on a Controller node	InternalApiAllocationPools in the network-environment.yaml file
     172.18.0.10	Storage virtual IP address that provides access to the Glance API and to Swift Proxy services	StorageAllocationPools attribute in the network-environment.yaml file
     172.19.0.10	Provides access to storage management	StorageMgmtAlloctionPools in the network-environment.yaml file

   • View a specific VIP address by running the pcs resource show command with the name of the resource that uses that VIP, in this example ip-192.168.1.150:

     $ sudo pcs resource show ip-192.168.1.150

     Example output:

      Resource: ip-192.168.1.150 (class=ocf provider=heartbeat type=IPaddr2)
       Attributes: ip=192.168.1.150 cidr_netmask=32
       Operations: start interval=0s timeout=20s (ip-192.168.1.150-start-timeout-20s)
                   stop interval=0s timeout=20s (ip-192.168.1.150-stop-timeout-20s)
                   monitor interval=10s timeout=20s (ip-192.168.1.150-monitor-interval-10s)

Procedure

1. Log in to the Controller node that is assigned to the IP address you want to view and run the ip addr show command on the network interface, in this example vlan100:

   $ ip addr show vlan100

   Example output:

     9: vlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
       link/ether be:ab:aa:37:34:e7 brd ff:ff:ff:ff:ff:ff
       inet *192.168.1.151/24* brd 192.168.1.255 scope global vlan100
          valid_lft forever preferred_lft forever
       inet *192.168.1.150/32* brd 192.168.1.255 scope global vlan100
          valid_lft forever preferred_lft forever

2. Run the netstat command to show all processes that listen to the IP address, in this example 192.168.1.150.haproxy:

   $ sudo netstat -tupln | grep "192.168.1.150.haproxy"

   Example output:

   tcp        0      0 192.168.1.150:8778          0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:8042          0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:9292          0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:8080          0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:80            0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:8977          0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:6080          0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:9696          0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:8000          0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:8004          0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:8774          0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:5000          0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:8776          0.0.0.0:*               LISTEN      61029/haproxy
   tcp        0      0 192.168.1.150:8041          0.0.0.0:*               LISTEN      61029/haproxy

   Note

   Processes that are listening to all local addresses, such as 0.0.0.0, are also available through 192.168.1.150. These processes include sshd, mysqld, dhclient, ntpd.

3. View the default port number assignments and the services they listen to by opening the configuration file for the HA service, in this example /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg:

   • TCP port 6080: nova_novncproxy
   • TCP port 9696: neutron
   • TCP port 8000: heat_cfn
   • TCP port 80: horizon

   • TCP port 8776: cinder

     In this example, most services that are defined in the haproxy.cfg file listen to the 192.168.1.150 IP address on all three Controller nodes. However, only the controller-0 node is listening externally to the 192.168.1.150 IP address.

     Therefore, if the controller-0 node fails, HAProxy only needs to re-assign 192.168.1.150 to another Controller node and all other services will already be running on the fallback Controller node.

Procedure

1. Log in to any Controller node as the tripleo-admin user.

   $ ssh tripleo-admin@overcloud-controller-0

2. Run the pcs status command:

   [tripleo-admin@overcloud-controller-0 ~]  $ sudo pcs status

   Example output:

    my-ipmilan-for-controller-0	(stonith:fence_ipmilan): Started my-ipmilan-for-controller-0
    my-ipmilan-for-controller-1	(stonith:fence_ipmilan): Started my-ipmilan-for-controller-1
    my-ipmilan-for-controller-2	(stonith:fence_ipmilan): Started my-ipmilan-for-controller-2
   
   PCSD Status:
     overcloud-controller-0: Online
     overcloud-controller-1: Online
     overcloud-controller-2: Online
   
   Daemon Status:
     corosync: active/enabled
     pacemaker: active/enabled openstack-cinder-volume        (systemd:openstack-cinder-volume):      Started overcloud-controller-0
     pcsd: active/enabled

   The output shows the following sections of the pcs status command output:

   • my-ipmilan-for-controller: Shows the type of fencing for each Controller node (stonith:fence_ipmilan) and whether or not the IPMI service is stopped or running.
   • PCSD Status: Shows that all three Controller nodes are currently online.
   • Daemon Status: Shows the status of the three Pacemaker daemons: corosync, pacemaker, and pcsd. In the example, all three services are active and enabled.

Procedure

1. Log in to each Controller node as the tripleo-admin user.

2. Verify that the cluster is running:

   $ sudo pcs status

   Example output:

   Cluster name: openstackHA
   Last updated: Wed Jun 24 12:40:27 2015
   Last change: Wed Jun 24 11:36:18 2015
   Stack: corosync
   Current DC: lb-c1a2 (2) - partition with quorum
   Version: 1.1.12-a14efad
   3 Nodes configured
   141 Resources configured

3. Verify that STONITH is disabled:

   $ sudo pcs property show

   Example output:

   Cluster Properties:
   cluster-infrastructure: corosync
   cluster-name: openstackHA
   dc-version: 1.1.12-a14efad
   have-watchdog: false
   stonith-enabled: false

4. Depending on the fencing agent that you want to use, choose one of the following options:

   • If you use the IPMI or RHV fencing agent, generate the fencing.yaml environment file:

     (undercloud) $ openstack overcloud generate fencing --output fencing.yaml nodes.json

     Note

     This command converts ilo and drac power management details to IPMI equivalents.

   • If you use a different fencing agent, such as STONITH Block Device (SBD), fence_kdump, or Redfish, or if you use pre-provisioned nodes, create the fencing.yaml file manually.

5. SBD fencing only: Add the following parameter to the fencing.yaml file:

   parameter_defaults:
     ExtraConfig:
       pacemaker::corosync::enable_sbd: true

   Note

   This step is applicable to initial overcloud deployments only. For more information about how to enable SBD fencing on an existing overcloud, see Enabling sbd fencing in RHEL 7 and 8.

6. Multi-layered fencing only: Add the level-specific parameters to the generated fencing.yaml file:

   parameter_defaults:
     EnableFencing: true
     FencingConfig:
       devices:
         level1:
         - agent: [VALUE]
           host_mac: aa:bb:cc:dd:ee:ff
           params:
             <parameter>: <value>
         level2:
         - agent: fence_agent2
           host_mac: aa:bb:cc:dd:ee:ff
           params:
             <parameter>: <value>

   Replace <parameter> and <value> with the actual parameters and values that the fencing agent requires.

7. Run the overcloud deploy command and include the fencing.yaml file and any other environment files that are relevant for your deployment:

   openstack overcloud deploy --templates \
   -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
   -e ~/templates/network-environment.yaml \
   -e ~/templates/storage-environment.yaml --ntp-server pool.ntp.org --neutron-network-type vxlan --neutron-tunnel-types vxlan \
   -e fencing.yaml

8. SBD fencing only: Set the watchdog timer device interval and check that the interval is set correctly.

   # pcs property set stonith-watchdog-timeout=<interval>
   # pcs property show

Verification

1. Log in to the overcloud as the heat-admin user and ensure that Pacemaker is configured as the resource manager:

   $ source stackrc
   $ openstack server list | grep controller
   $ ssh tripleo-admin@<controller-x_ip>
   $ sudo pcs status | grep fence
   stonith-overcloud-controller-x (stonith:fence_ipmilan): Started overcloud-controller-y

   In this example, Pacemaker is configured to use a STONITH resource for each of the Controller nodes that are specified in the fencing.yaml file.

   Note

   You must not configure the fence-resource process on the same node that it controls.

2. Check the fencing resource attributes. The STONITH attribute values must match the values in the fencing.yaml file:

   $ sudo pcs stonith show <stonith-resource-controller-x>

Procedure

1. Log in to a Controller node as the stack user and source the credentials file:

   $ source stackrc
   $ openstack server list | grep controller
   $ ssh tripleo-admin@<controller-x_ip>

2. Change to the root user and close all connections to the Controller node:

   $ sudo -i
   iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT &&
   iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT &&
   iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 5016 -j ACCEPT &&
   iptables -A INPUT -p udp -m state --state NEW -m udp --dport 5016 -j ACCEPT &&
   iptables -A INPUT ! -i lo -j REJECT --reject-with icmp-host-prohibited &&
   iptables -A OUTPUT -p tcp --sport 22 -j ACCEPT &&
   iptables -A OUTPUT -p tcp --sport 5016 -j ACCEPT &&
   iptables -A OUTPUT -p udp --sport 5016 -j ACCEPT &&
   iptables -A OUTPUT ! -o lo -j REJECT --reject-with icmp-host-prohibited

3. From a different Controller node, locate the fencing event in the Pacemaker log file:

   $ ssh tripleo-admin@<controller-x_ip>
   $ less /var/log/cluster/corosync.log
   (less): /fenc*

   If the STONITH service performed the fencing action on the Controller, the log file shows a fencing event.

4. Wait a few minutes and then verify that the restarted Controller node is running in the cluster again by running the pcs status command. If you can see the Controller node that you restarted in the output, fencing functions correctly.

Procedure

1. Open the /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg file in any Controller node where HAProxy is installed.

2. Locate the listen haproxy.stats section:

   listen haproxy.stats
     bind 10.200.0.6:1993
     mode http
     stats enable
     stats uri /
     stats auth admin:<haproxy-stats-password>

3. In a Web browser, navigate to 10.200.0.6:1993 and enter the credentials from the stats auth row to view the HAProxy Stats page.

Procedure

1. From a Controller node, get the MariaDB database root password by running the hiera command.

   $ sudo hiera -c /etc/puppet/hiera.yaml "mysql::server::root_password"
   *[MYSQL-HIERA-PASSWORD]*

2. Get the name of the MariaDB container that runs on the node.

   $ sudo podman ps | grep -i galera
   a403d96c5026  undercloud.ctlplane.localdomain:8787/rhosp-rhel9/openstack-mariadb:16.0-106            /bin/bash /usr/lo...  3 hours ago  Up 3 hours ago         galera-bundle-podman-0

3. Get the write-set replication information from the MariaDB database on each node.

   $ sudo podman exec galera-bundle-podman-0 sudo mysql -B --password="[MYSQL-HIERA-PASSWORD]" -e "SHOW GLOBAL STATUS LIKE 'wsrep_%';"
       +----------------------------+----------+
       | Variable_name              | Value    |
       +----------------------------+----------+
       | wsrep_applier_thread_count | 1        |
       | wsrep_apply_oooe           | 0.018672 |
       | wsrep_apply_oool           | 0.000630 |
       | wsrep_apply_window         | 1.021942 |
       | ...                        | ...      |
       +----------------------------+----------+

   Each relevant variable uses the prefix wsrep.

4. Verify the health and integrity of the MariaDB Galera cluster by checking that the cluster is reporting the correct number of nodes.

Procedure

1. Log in to the Controller node on which the resource is failing.

2. Run the pcs status command with the grep option to get the status of the service:

   # sudo pcs status | grep cinder
     Podman container: openstack-cinder-volume [192.168.24.1:8787/rh-osbs/rhosp161-openstack-cinder-volume:pcmklatest]
       openstack-cinder-volume-podman-0     (ocf::heartbeat:podman):        Started controller-1

3. View the log events for the resource:

   # sudo less /var/log/containers/stdouts/openstack-cinder-volume.log
   [...]
   2021-04-12T12:32:17.607179705+00:00 stderr F ++ cat /run_command
   2021-04-12T12:32:17.609648533+00:00 stderr F + CMD='/usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf'
   2021-04-12T12:32:17.609648533+00:00 stderr F + ARGS=
   2021-04-12T12:32:17.609648533+00:00 stderr F + [[ ! -n '' ]]
   2021-04-12T12:32:17.609648533+00:00 stderr F + . kolla_extend_start
   2021-04-12T12:32:17.611214130+00:00 stderr F +++ stat -c %U:%G /var/lib/cinder
   2021-04-12T12:32:17.616637578+00:00 stderr F ++ [[ cinder:kolla != \c\i\n\d\e\r\:\k\o\l\l\a ]]
   2021-04-12T12:32:17.616722778+00:00 stderr F + echo 'Running command: '\''/usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf'\'''
   2021-04-12T12:32:17.616751172+00:00 stdout F Running command: '/usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf'
   2021-04-12T12:32:17.616775368+00:00 stderr F + exec /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf

4. Correct the failed resource based on the information from the output and from the logs.

5. Run the pcs resource cleanup command to reset the status and the fail count of the resource.

   $ sudo pcs resource cleanup openstack-cinder-volume
     Resource: openstack-cinder-volume successfully cleaned up

Procedure

1. Run the systemctl status command to show the resource status and recent log events:

   [tripleo-admin@controller-0 ~]$ sudo systemctl status tripleo_nova_conductor
   ● tripleo_nova_conductor.service - nova_conductor container
      Loaded: loaded (/etc/systemd/system/tripleo_nova_conductor.service; enabled; vendor preset: disabled)
      Active: active (running) since Mon 2021-04-12 10:54:46 UTC; 1h 38min ago
    Main PID: 5125 (conmon)
       Tasks: 2 (limit: 126564)
      Memory: 1.2M
      CGroup: /system.slice/tripleo_nova_conductor.service
              └─5125 /usr/bin/conmon --api-version 1 -c cc3c63b54e0864c94ac54a5789be96aea1dd60b2f3216b37c3e020c76e7887d4 -u cc3c63b54e0864c94ac54a5789be96aea1dd60b2f3216b37c3e020c76e7887d4 -r /usr/bin/runc -b /var/lib/containers/storage/overlay-containers/cc3c63b54e0864c94ac54a5789be96aea1dd60b2f3216b37c3e02>
   
   Apr 12 10:54:42 controller-0.redhat.local systemd[1]: Starting nova_conductor container...
   Apr 12 10:54:46 controller-0.redhat.local podman[2855]: nova_conductor
   Apr 12 10:54:46 controller-0.redhat.local systemd[1]: Started nova_conductor container.

2. View the log events for the resource:

   # sudo less /var/log/containers/tripleo_nova_conductor.log

3. Correct the failed resource based on the information from the output and from the logs.

4. Restart the resource and check the status of the service:

   # systemctl restart tripleo_nova_conductor
   # systemctl status tripleo_nova_conductor
   ● tripleo_nova_conductor.service - nova_conductor container
      Loaded: loaded (/etc/systemd/system/tripleo_nova_conductor.service; enabled; vendor preset: disabled)
      Active: active (running) since Thu 2021-04-22 14:28:35 UTC; 7s ago
     Process: 518937 ExecStopPost=/usr/bin/podman stop -t 10 nova_conductor (code=exited, status=0/SUCCESS)
     Process: 518653 ExecStop=/usr/bin/podman stop -t 10 nova_conductor (code=exited, status=0/SUCCESS)
     Process: 519063 ExecStart=/usr/bin/podman start nova_conductor (code=exited, status=0/SUCCESS)
    Main PID: 519198 (conmon)
       Tasks: 2 (limit: 126564)
      Memory: 1.1M
      CGroup: /system.slice/tripleo_nova_conductor.service
              └─519198 /usr/bin/conmon --api-version 1 -c 0d6583beb20508e6bacccd5fea169a2fe949471207cb7d4650fec5f3638c2ce6 -u 0d6583beb20508e6bacccd5fea169a2fe949471207cb7d4650fec5f3638c2ce6 -r /usr/bin/runc -b /var/lib/containe>
   
   Apr 22 14:28:34 controller-0.redhat.local systemd[1]: Starting nova_conductor container...
   Apr 22 14:28:35 controller-0.redhat.local podman[519063]: nova_conductor
   Apr 22 14:28:35 controller-0.redhat.local systemd[1]: Started nova_conductor container.