Red Hat Summit이 콘텐츠는 선택한 언어로 제공되지 않습니다.
Chapter 4. Managing high availability services with Pacemaker
The Pacemaker service manages core container and active-passive services, such as Galera, RabbitMQ, Redis, and HAProxy. You use Pacemaker to view and manage general information about the managed services, virtual IP addresses, power management, and fencing.
For more information about Pacemaker in Red Hat Enterprise Linux, see Configuring and Managing High Availability Clusters in the Red Hat Enterprise Linux documentation.
4.1. Resource bundles and containers
Pacemaker manages Red Hat OpenStack Platform (RHOSP) services as Bundle Set resources, or bundles. Most of these services are active-active services that start in the same way and always run on each Controller node.
Pacemaker manages the following resource types:
- Bundle
- A bundle resource configures and replicates the same container on all Controller nodes, maps the necessary storage paths to the container directories, and sets specific attributes related to the resource itself.
- Container
-
A container can run different kinds of resources, from simple
systemdservices like HAProxy to complex services like Galera, which requires specific resource agents that control and set the state of the service on the different nodes.
-
You cannot use
podmanorsystemctlto manage bundles or containers. You can use the commands to check the status of the services, but you must use Pacemaker to perform actions on these services. -
Podman containers that Pacemaker controls have a RestartPolicy set to
noby Podman. This is to ensure that Pacemaker, and not Podman, controls the container start and stop actions.
Simple Bundle Set resources (simple bundles)
A simple Bundle Set resource, or simple bundle, is a set of containers that each include the same Pacemaker services that you want to deploy across the Controller nodes.
The following example shows a list of simple bundles from the output of the pcs status command:
Podman container set: haproxy-bundle [192.168.24.1:8787/rhosp-rhel8/openstack-haproxy:pcmklatest] haproxy-bundle-podman-0 (ocf::heartbeat:podman): Started overcloud-controller-0 haproxy-bundle-podman-1 (ocf::heartbeat:podman): Started overcloud-controller-1 haproxy-bundle-podman-2 (ocf::heartbeat:podman): Started overcloud-controller-2
Podman container set: haproxy-bundle [192.168.24.1:8787/rhosp-rhel8/openstack-haproxy:pcmklatest]
haproxy-bundle-podman-0 (ocf::heartbeat:podman): Started overcloud-controller-0
haproxy-bundle-podman-1 (ocf::heartbeat:podman): Started overcloud-controller-1
haproxy-bundle-podman-2 (ocf::heartbeat:podman): Started overcloud-controller-2For each bundle, you can see the following details:
- The name that Pacemaker assigns to the service.
- The reference to the container that is associated with the bundle.
- The list and status of replicas that are running on the different Controller nodes.
The following example shows the settings for the haproxy-bundle simple bundle:
The example shows the following information about the containers in the bundle:
-
image: Image used by the container, which refers to the local registry of the undercloud. -
network: Container network type, which is"host"in the example. -
options: Specific options for the container. -
replicas: Indicates how many copies of the container must run in the cluster. Each bundle includes three containers, one for each Controller node. -
run-command: System command used to spawn the container. -
Storage Mapping: Mapping of the local path on each host to the container. To check the haproxy configuration from the host, open the
/var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfgfile instead of the/etc/haproxy/haproxy.cfgfile.
Although HAProxy provides high availability services by load balancing traffic to selected services, you configure HAProxy as a highly available service by managing it as a Pacemaker bundle service.
Complex Bundle Set resources (complex bundles)
Complex Bundle Set resources, or complex bundles, are Pacemaker services that specify a resource configuration in addition to the basic container configuration that is included in simple bundles.
This configuration is needed to manage Multi-State resources, which are services that can have different states depending on the Controller node they run on.
This example shows a list of complex bundles from the output of the pcs status command:
This output shows the following information about each complex bundle:
- RabbitMQ: All three Controller nodes run a standalone instance of the service, similar to a simple bundle.
- Galera: All three Controller nodes are running as Galera masters under the same constraints.
- Redis: The overcloud-controller-0 container is running as the master, while the other two Controller nodes are running as slaves. Each container type might run under different constraints.
The following example shows the settings for the galera-bundle complex bundle:
This output shows that, unlike in a simple bundle, the galera-bundle resource includes explicit resource configuration that determines all aspects of the multi-state resource.
Although a service can run on multiple Controller nodes at the same time, the Controller node itself might not be listening at the IP address that is required to reach those services. For information about how to check the IP address of a service, see Section 4.4, “Viewing virtual IP addresses”.
4.2. Viewing general Pacemaker information
To view general Pacemaker information, use the pcs status command.
Procedure
Log in to any Controller node as the heat-admin user.
ssh heat-admin@overcloud-controller-0
$ ssh heat-admin@overcloud-controller-0Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the
pcs statuscommand:[heat-admin@overcloud-controller-0 ~] $ sudo pcs status
[heat-admin@overcloud-controller-0 ~] $ sudo pcs statusCopy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The main sections of the output show the following information about the cluster:
-
Cluster name: Name of the cluster. -
[NUM] nodes configured: Number of nodes that are configured for the cluster. -
[NUM] resources configured: Number of resources that are configured for the cluster. -
Online: Names of the Controller nodes that are currently online. -
GuestOnline: Names of the guest nodes that are currently online. Each guest node consists of a complex Bundle Set resource. For more information about bundle sets, see Section 4.1, “Resource bundles and containers”.
-
4.3. Viewing bundle status
You can check the status of a bundle from an undercloud node or log in to one of the Controller nodes to check the bundle status directly.
Check bundle status from an undercloud node
Run the following command:
sudo podman exec -it haproxy-bundle-podman-0 ps -efww | grep haproxy*
$ sudo podman exec -it haproxy-bundle-podman-0 ps -efww | grep haproxy*Example output:
root 7 1 0 06:08 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ws haproxy 11 7 0 06:08 ? 00:00:17 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ws
root 7 1 0 06:08 ? 00:00:00 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ws
haproxy 11 7 0 06:08 ? 00:00:17 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -Ws
The output shows that the haproxy process is running inside the container.
Check bundle status from a Controller node
Log in to a Controller node and run the following command:
ps -ef | grep haproxy*
$ ps -ef | grep haproxy*Example output:
4.4. Viewing virtual IP addresses
Each IPaddr2 resource sets a virtual IP address that clients use to request access to a service. If the Controller node with that IP address fails, the IPaddr2 resource reassigns the IP address to a different Controller node.
Show all virtual IP addresses
Run the pcs resource show command with the --full option to display all resources that use the VirtualIP type:
sudo pcs resource show --full
$ sudo pcs resource show --fullThe following example output shows each Controller node that is currently set to listen to a particular virtual IP address:
Each IP address is initially attached to a specific Controller node. For example, 192.168.1.150 is started on overcloud-controller-0. However, if that Controller node fails, the IP address is reassigned to other Controller nodes in the cluster.
The following table describes the IP addresses in the example output and shows the original allocation of each IP address.
| IP Address | Description | Allocated From |
|---|---|---|
|
| Public IP address |
|
|
| Controller virtual IP address |
Part of the |
|
| Provides access to OpenStack API services on a Controller node |
|
|
| Storage virtual IP address that provides access to the Glance API and to Swift Proxy services |
|
|
| Provides access to Redis service on a Controller node |
|
|
| Provides access to storage management |
|
View a specific IP address
Run the pcs resource show command.
sudo pcs resource show ip-192.168.1.150
$ sudo pcs resource show ip-192.168.1.150Example output:
Resource: ip-192.168.1.150 (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=192.168.1.150 cidr_netmask=32
Operations: start interval=0s timeout=20s (ip-192.168.1.150-start-timeout-20s)
stop interval=0s timeout=20s (ip-192.168.1.150-stop-timeout-20s)
monitor interval=10s timeout=20s (ip-192.168.1.150-monitor-interval-10s)
Resource: ip-192.168.1.150 (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=192.168.1.150 cidr_netmask=32
Operations: start interval=0s timeout=20s (ip-192.168.1.150-start-timeout-20s)
stop interval=0s timeout=20s (ip-192.168.1.150-stop-timeout-20s)
monitor interval=10s timeout=20s (ip-192.168.1.150-monitor-interval-10s)View network information for a specific IP address
- Log in to the Controller node that is assigned to the IP address you want to view.
Run the
ip addr showcommand to view network interface information.ip addr show vlan100
$ ip addr show vlan100Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Run the
netstatcommand to show all processes that listen to the IP address.sudo netstat -tupln | grep "192.168.1.150.haproxy"
$ sudo netstat -tupln | grep "192.168.1.150.haproxy"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Example output:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow NoteProcesses that are listening to all local addresses, such as
0.0.0.0, are also available through192.168.1.150. These processes includesshd,mysqld,dhclient,ntpd.
View port number assignments
Open the /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg file to see default port number assignments.
The following example shows the port numbers and the services that they listen to:
- TCP port 6080: nova_novncproxy
- TCP port 9696: neutron
- TCP port 8000: heat_cfn
- TCP port 80: horizon
- TCP port 8776: cinder
In this example, most services that are defined in the haproxy.cfg file listen to the 192.168.1.150 IP address on all three Controller nodes. However, only the controller-0 node is listening externally to the 192.168.1.150 IP address.
Therefore, if the controller-0 node fails, HAProxy only needs to re-assign 192.168.1.150 to another Controller node and all other services will already be running on the fallback Controller node.
4.5. Viewing Pacemaker status and power management information
The last sections of the pcs status output show information about your power management fencing, such as IPMI, and the status of the Pacemaker service itself:
The my-ipmilan-for-controller settings show the type of fencing for each Controller node (stonith:fence_ipmilan) and whether or not the IPMI service is stopped or running. The PCSD Status shows that all three Controller nodes are currently online. The Pacemaker service consists of three daemons: corosync, pacemaker, and pcsd. In the example, all three services are active and enabled.
4.6. Troubleshooting failed Pacemaker resources
If one the Pacemaker resources fails, you can view the Failed Actions section of the pcs status output. In the following example, the openstack-cinder-volume service stopped working on controller-0:
Failed Actions: * openstack-cinder-volume_monitor_60000 on overcloud-controller-0 'not running' (7): call=74, status=complete, exitreason='none', last-rc-change='Wed Dec 14 08:33:14 2016', queued=0ms, exec=0ms
Failed Actions:
* openstack-cinder-volume_monitor_60000 on overcloud-controller-0 'not running' (7): call=74, status=complete, exitreason='none',
last-rc-change='Wed Dec 14 08:33:14 2016', queued=0ms, exec=0msIn this case, you must enable the systemd service openstack-cinder-volume. In other cases, you might need to locate and fix the problem and then clean up the resources. For more information about troubleshooting resource problems, see Chapter 8, Troubleshooting resource problems.
'%20d='M201.5%20305.5c-13.8%200-24.9-11.1-24.9-24.6%200-13.8%2011.1-24.9%2024.9-24.9%2013.6%200%2024.6%2011.1%2024.6%2024.9%200%2013.6-11.1%2024.6-24.6%2024.6zM504%20256c0%20137-111%20248-248%20248S8%20393%208%20256%20119%208%20256%208s248%20111%20248%20248zm-132.3-41.2c-9.4%200-17.7%203.9-23.8%2010-22.4-15.5-52.6-25.5-86.1-26.6l17.4-78.3%2055.4%2012.5c0%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.3%2024.9-24.9s-11.1-24.9-24.9-24.9c-9.7%200-18%205.8-22.1%2013.8l-61.2-13.6c-3-.8-6.1%201.4-6.9%204.4l-19.1%2086.4c-33.2%201.4-63.1%2011.3-85.5%2026.8-6.1-6.4-14.7-10.2-24.1-10.2-34.9%200-46.3%2046.9-14.4%2062.8-1.1%205-1.7%2010.2-1.7%2015.5%200%2052.6%2059.2%2095.2%20132%2095.2%2073.1%200%20132.3-42.6%20132.3-95.2%200-5.3-.6-10.8-1.9-15.8%2031.3-16%2019.8-62.5-14.9-62.5zM302.8%20331c-18.2%2018.2-76.1%2017.9-93.6%200-2.2-2.2-6.1-2.2-8.3%200-2.5%202.5-2.5%206.4%200%208.6%2022.8%2022.8%2087.3%2022.8%20110.2%200%202.5-2.2%202.5-6.1%200-8.6-2.2-2.2-6.1-2.2-8.3%200zm7.7-75c-13.6%200-24.6%2011.1-24.6%2024.9%200%2013.6%2011.1%2024.6%2024.6%2024.6%2013.8%200%2024.9-11.1%2024.9-24.6%200-13.8-11-24.9-24.9-24.9z'/%3e%3c/svg%3e){kind=small}