Configuring and managing high availability clusters

Procedure

1. From the system on which you are running the RHEL web console, log in to the console and install the HA Cluster Management add-on application. See Add-on applications for the RHEL web console in the "Managing systems in the RHEL web console" document for details.

2. On each cluster node, install the Red Hat High Availability fence agents from the High Availability channel.

   # dnf install fence-agents-all

   You can install only the fence agent that you require with the following command.

   # dnf install fence-agents-model

3. On each cluster node, ensure that the pcsd service is running.

   # systemctl status pcsd.service

   If the pcsd service is not running on a cluster node, enter the following command to start the pcsd service and to enable it at system start.

   # systemctl enable --now pcsd.service

4. Ensure you are logged in to the RHEL web console. To use the RHEL web console to create clusters, the user account used to sign in to the web console must have sudo access to the system.

   Note

   The hacluster user account is the Pacemaker service account and you cannot use this account to log in to the RHEL web console.

5. In the RHEL web console, switch to administrative access mode. For information about administrative access mode, see Administrative access in the web console in the "Managing systems in the RHEL web console" document.

   Note

   Only a user with sudo access can create clusters and add nodes to existing ones. After you create a cluster, by default, users in the haclient group can manage the cluster and change permissions. For information about granting different permissions to any other users and groups that require them, or for modifying the default haclient permissions, see Granting HA Cluster Management permissions.

Procedure

1. Log in to the RHEL web console with an account that has sudo access to the system and ensure that you are in administrative access mode. For information about administrative access mode, see Administrative access in the web console in the "Managing systems in the RHEL web console" document.

   Alternatively, log into the RHEL web console with an account that has grant permissions for the cluster you want to manage. The account must be a member of the haclient group to see the HA Cluster Management web console add-on in limited access mode.

2. Select a cluster for which you want to manage permissions from the cluster list.
3. In the cluster detail, click the Permissions tab on the top of the page and select Create Permission.
4. Add, remove, or edit the permissions for a user or group.
5. By default, any user with an account that is a member of the haclient group has read, write and grant permissions. From the Permissions page you can remove this permission if you have administrative access in the web console or if you have grant permissions.

Procedure

1. On each node in the cluster, enable the repository for high availability that corresponds to your system architecture. For example, to enable the high availability repository for an x86_64 system, you can enter the following subscription-manager command:

   # subscription-manager repos --enable=rhel-10-for-x86_64-highavailability-rpms

2. On each node in the cluster, install the Red Hat High Availability Add-On software packages along with all available fence agents from the High Availability channel.

   # dnf install pcs pacemaker fence-agents-all

   Alternatively, you can install the Red Hat High Availability Add-On software packages along with only the fence agent that you require with the following command.

   # dnf install pcs pacemaker fence-agents-model

   The following command displays a list of the available fence agents.

   # rpm -q -a | grep fence
   fence-agents-rhevm-4.0.2-3.el7.x86_64
   fence-agents-ilo-mp-4.0.2-3.el7.x86_64
   fence-agents-ipmilan-4.0.2-3.el7.x86_64
   ...

   Warning

   After you install the Red Hat High Availability Add-On packages, you should ensure that your software update preferences are set so that nothing is installed automatically. Installation on a running cluster can cause unexpected behaviors. For more information, see the Red Hat Knowledgebase article Recommended Practices for Applying Software Updates to a RHEL High Availability or Resilient Storage Cluster.

3. If you are running the firewalld daemon, execute the following commands to enable the ports that are required by the Red Hat High Availability Add-On.

   Note

   You can determine whether the firewalld daemon is installed on your system with the rpm -q firewalld command. If it is installed, you can determine whether it is running with the firewall-cmd --state command.

   # firewall-cmd --permanent --add-service=high-availability
   # firewall-cmd --add-service=high-availability

   Note

   The ideal firewall configuration for cluster components depends on the local environment, where you may need to take into account such considerations as whether the nodes have multiple network interfaces or whether off-host firewalling is present. The example here, which opens the ports that are generally required by a Pacemaker cluster, should be modified to suit local conditions. Enabling ports for the High Availability Add-On shows the ports to enable for the Red Hat High Availability Add-On and provides an explanation for what each port is used for.

4. In order to use pcs to configure the cluster and communicate among the nodes, you must set a password on each node for the user ID hacluster, which is the pcs administration account. It is recommended that the password for user hacluster be the same on each node.

   # passwd hacluster
   Changing password for user hacluster.
   New password:
   Retype new password:
   passwd: all authentication tokens updated successfully.

5. Before the cluster can be configured, the pcsd daemon must be started and enabled to start up on boot on each node. This daemon works with the pcs command to manage configuration across the nodes in the cluster.

   On each node in the cluster, execute the following commands to start the pcsd service and to enable pcsd at system start.

   # systemctl start pcsd.service
   # systemctl enable pcsd.service

Procedure

1. Authenticate the pcs user hacluster for each node in the cluster on the node from which you will be running pcs.

   The following command authenticates user hacluster on z1.example.com for both of the nodes in a two-node cluster that will consist of z1.example.com and z2.example.com.

   [root@z1 ~]# pcs host auth z1.example.com z2.example.com
   Username: hacluster
   Password:
   z1.example.com: Authorized
   z2.example.com: Authorized

2. Execute the following command from z1.example.com to create the two-node cluster my_cluster that consists of nodes z1.example.com and z2.example.com. This will propagate the cluster configuration files to both nodes in the cluster. This command includes the --start option, which will start the cluster services on both nodes in the cluster.

   [root@z1 ~]# pcs cluster setup my_cluster --start z1.example.com z2.example.com

3. Enable the cluster services to run on each node in the cluster when the node is booted.

   Note

   For your particular environment, you can skip this step by keeping the cluster services disabled. If enabled and a node goes down, any issues with your cluster or your resources are resolved before the node rejoins the cluster. If you keep the cluster services disabled, you need to manually start the services when you reboot a node by executing the pcs cluster start command on that node.

   [root@z1 ~]# pcs cluster enable --all

4. Display the status of the cluster you created with the pcs cluster status command. Because there could be a slight delay before the cluster is up and running when you start the cluster services with the --start option of the pcs cluster setup command, you should ensure that the cluster is up and running before performing any subsequent actions on the cluster and its configuration.

   [root@z1 ~]# pcs cluster status
   Cluster Status:
    Stack: corosync
    Current DC: z2.example.com (version 2.0.0-10.el8-b67d8d0de9) - partition with quorum
    Last updated: Thu Oct 11 16:11:18 2018
    Last change: Thu Oct 11 16:11:00 2018 by hacluster via crmd on z2.example.com
    2 Nodes configured
    0 Resources configured
   
   ...

Procedure

1. Create the fencing device. Note that you can use an IP address when specifying the host name for the nodes.

   [root@z1 ~]# pcs stonith create myapc fence_apc_snmp ipaddr="zapc.example.com" pcmk_host_map="z1.example.com:1;z2.example.com:2" login="apc" passwd="apc"

2. Display the parameters of the fence device you created.

   [root@rh7-1 ~]# pcs stonith config myapc
    Resource: myapc (class=stonith type=fence_apc_snmp)
     Attributes: ipaddr=zapc.example.com pcmk_host_map=z1.example.com:1;z2.example.com:2 login=apc passwd=apc
     Operations: monitor interval=60s (myapc-monitor-interval-60s)

   After configuring your fence device, you should test the device. For information about testing a fence device, see Testing a fence device.

   Note

   Do not test your fence device by disabling the network interface, as this will not properly test fencing.

   Note

   Once fencing is configured and a cluster has been started, a network restart will trigger fencing for the node which restarts the network even when the timeout is not exceeded. For this reason, do not restart the network service while the cluster service is running because it will trigger unintentional fencing on the node.

Procedure

1. On both nodes of the cluster, perform the following steps to set the value for the LVM system ID to the value of the uname identifier for the system. The LVM system ID will be used to ensure that only the cluster is capable of activating the volume group.

   1. Set the system_id_source configuration option in the /etc/lvm/lvm.conf configuration file to uname.

      # Configuration option global/system_id_source.
      system_id_source = "uname"

   2. Verify that the LVM system ID on the node matches the uname for the node.

      # lvm systemid
        system ID: z1.example.com
      # uname -n
        z1.example.com

2. Create the LVM volume and create an XFS file system on that volume. Since the /dev/sdb1 partition is storage that is shared, you perform this part of the procedure on one node only.

   1. Create an LVM physical volume on partition /dev/sdb1.

      [root@z1 ~]# pvcreate /dev/sdb1
        Physical volume "/dev/sdb1" successfully created

   2. Create the volume group my_vg that consists of the physical volume /dev/sdb1.

      Specify the --setautoactivation n flag to ensure that volume groups managed by Pacemaker in a cluster will not be automatically activated on startup. If you are using an existing volume group for the LVM volume you are creating, you can reset this flag with the vgchange --setautoactivation n command for the volume group.

      [root@z1 ~]# vgcreate --setautoactivation n my_vg /dev/sdb1
        Volume group "my_vg" successfully created

      Note

      If your LVM volume group contains one or more physical volumes that reside on remote block storage, such as an iSCSI target, Red Hat recommends that you ensure that the service starts before Pacemaker starts. For information about configuring startup order for a remote physical volume used by a Pacemaker cluster, see Configuring startup order for resource dependencies not managed by Pacemaker.

   3. Verify that the new volume group has the system ID of the node on which you are running and from which you created the volume group.

      [root@z1 ~]# vgs -o+systemid
        VG    #PV #LV #SN Attr   VSize  VFree  System ID
        my_vg   1   0   0 wz--n- <1.82t <1.82t z1.example.com

   4. Create a logical volume using the volume group my_vg.

      [root@z1 ~]# lvcreate -L450 -n my_lv my_vg
        Rounding up size to full physical extent 452.00 MiB
        Logical volume "my_lv" created

      You can use the lvs command to display the logical volume.

      [root@z1 ~]# lvs
        LV      VG      Attr      LSize   Pool Origin Data%  Move Log Copy%  Convert
        my_lv   my_vg   -wi-a---- 452.00m
        ...

   5. Create an XFS file system on the logical volume my_lv.

      [root@z1 ~]# mkfs.xfs /dev/my_vg/my_lv
      meta-data=/dev/my_vg/my_lv       isize=512    agcount=4, agsize=28928 blks
               =                       sectsz=512   attr=2, projid32bit=1
      ...

3. If the use of a devices file is enabled with the use_devicesfile = 1 parameter in the lvm.conf file, add the shared device to the devices file on the second node in the cluster. This feature is enabled by default.

   [root@z2 ~]# lvmdevices --adddev /dev/sdb1

Procedure

1. Ensure that the Apache HTTP Server is installed on each node in the cluster. You also need the wget tool installed on the cluster to be able to check the status of the Apache HTTP Server.

   On each node, execute the following command.

   # dnf install -y httpd wget

   If you are running the firewalld daemon, on each node in the cluster enable the ports that are required by the Red Hat High Availability Add-On and enable the ports you will require for running httpd. This example enables the httpd ports for public access, but the specific ports to enable for httpd may vary for production use.

   # firewall-cmd --permanent --add-service=http
   # firewall-cmd --permanent --zone=public --add-service=http
   # firewall-cmd --reload

2. In order for the Apache resource agent to get the status of Apache, on each node in the cluster create the following addition to the existing configuration to enable the status server URL.

   # cat <<-END > /etc/httpd/conf.d/status.conf
   <Location /server-status>
       SetHandler server-status
       Require local
   </Location>
   END

3. Create a web page for Apache to serve up.

   On one node in the cluster, ensure that the logical volume you created in Configuring an LVM volume with an XFS file system in a Pacemaker cluster is activated, mount the file system that you created on that logical volume, create the file index.html on that file system, and then unmount the file system.

   # lvchange -ay my_vg/my_lv
   # mount /dev/my_vg/my_lv /var/www/
   # mkdir /var/www/html
   # mkdir /var/www/cgi-bin
   # mkdir /var/www/error
   # restorecon -R /var/www
   # cat <<-END >/var/www/html/index.html
   <html>
   <body>Hello</body>
   </html>
   END
   # umount /var/www

1. An LVM-activate resource named my_lvm that uses the LVM volume group you created in Configuring an LVM volume with an XFS file system in a Pacemaker cluster.
2. A Filesystem resource named my_fs, that uses the file system device /dev/my_vg/my_lv you created in Configuring an LVM volume with an XFS file system in a Pacemaker cluster.
3. An IPaddr2 resource, which is a floating IP address for the apachegroup resource group. The IP address must not be one already associated with a physical node. If the IPaddr2 resource’s NIC device is not specified, the floating IP must reside on the same network as one of the node’s statically assigned IP addresses, otherwise the NIC device to assign the floating IP address cannot be properly detected.
4. An apache resource named Website that uses the index.html file and the Apache configuration you defined in Configuring an Apache HTTP server.

Procedure

1. The following command creates the LVM-activate resource my_lvm. Because the resource group apachegroup does not yet exist, this command creates the resource group.

   Note

   Do not configure more than one LVM-activate resource that uses the same LVM volume group in an active/passive HA configuration, as this could cause data corruption. Additionally, do not configure an LVM-activate resource as a clone resource in an active/passive HA configuration.

   [root@z1 ~]# pcs resource create my_lvm ocf:heartbeat:LVM-activate vgname=my_vg vg_access_mode=system_id --group apachegroup

   When you create a resource, the resource is started automatically. You can use the following command to confirm that the resource was created and has started.

   # pcs resource status
    Resource Group: apachegroup
        my_lvm	(ocf::heartbeat:LVM-activate):	Started

   You can manually stop and start an individual resource with the pcs resource disable and pcs resource enable commands.

2. The following commands create the remaining resources for the configuration, adding them to the existing resource group apachegroup.

   [root@z1 ~]# pcs resource create my_fs Filesystem device="/dev/my_vg/my_lv" directory="/var/www" fstype="xfs" --group apachegroup
   
   [root@z1 ~]# pcs resource create VirtualIP IPaddr2 ip=198.51.100.3 cidr_netmask=24 --group apachegroup
   
   [root@z1 ~]# pcs resource create Website apache configfile="/etc/httpd/conf/httpd.conf" statusurl="http://127.0.0.1/server-status" --group apachegroup

3. After creating the resources and the resource group that contains them, you can check the status of the cluster. Note that all four resources are running on the same node.

   [root@z1 ~]# pcs status
   Cluster name: my_cluster
   Last updated: Wed Jul 31 16:38:51 2013
   Last change: Wed Jul 31 16:42:14 2013 via crm_attribute on z1.example.com
   Stack: corosync
   Current DC: z2.example.com (2) - partition with quorum
   Version: 1.1.10-5.el7-9abe687
   2 Nodes configured
   6 Resources configured
   
   Online: [ z1.example.com z2.example.com ]
   
   Full list of resources:
    myapc	(stonith:fence_apc_snmp):	Started z1.example.com
    Resource Group: apachegroup
        my_lvm	(ocf::heartbeat:LVM-activate):	Started z1.example.com
        my_fs	(ocf::heartbeat:Filesystem):	Started z1.example.com
        VirtualIP	(ocf::heartbeat:IPaddr2):	Started z1.example.com
        Website	(ocf::heartbeat:apache):	Started z1.example.com

   Note that if you have not configured a fencing device for your cluster, by default the resources do not start.

4. Once the cluster is up and running, you can point a browser to the IP address you defined as the IPaddr2 resource to view the sample display, consisting of the simple word "Hello".

   Hello

   If you find that the resources you configured are not running, you can run the pcs resource debug-start resource command to test the resource configuration.

5. When you use the apache resource agent to manage Apache, it does not use systemd. Because of this, you must edit the logrotate script supplied with Apache so that it does not use systemctl to reload Apache.

   Remove the following line in the /etc/logrotate.d/httpd file on each node in the cluster.

   /bin/systemctl reload httpd.service > /dev/null 2>/dev/null || true

   Replace the line you removed with the following three lines, specifying /var/run/httpd-website.pid as the PID file path where website is the name of the Apache resource. In this example, the Apache resource name is Website.

   /usr/bin/test -f /var/run/httpd-Website.pid >/dev/null 2>/dev/null &&
   /usr/bin/ps -q $(/usr/bin/cat /var/run/httpd-Website.pid) >/dev/null 2>/dev/null &&
   /usr/sbin/httpd -f /etc/httpd/conf/httpd.conf -c "PidFile /var/run/httpd-Website.pid" -k graceful > /dev/null 2>/dev/null || true

Procedure

1. The following command puts node z1.example.com in standby mode.

   [root@z1 ~]# pcs node standby z1.example.com

2. After putting node z1 in standby mode, check the cluster status. Note that the resources should now all be running on z2.

   [root@z1 ~]# pcs status
   Cluster name: my_cluster
   Last updated: Wed Jul 31 17:16:17 2013
   Last change: Wed Jul 31 17:18:34 2013 via crm_attribute on z1.example.com
   Stack: corosync
   Current DC: z2.example.com (2) - partition with quorum
   Version: 1.1.10-5.el7-9abe687
   2 Nodes configured
   6 Resources configured
   
   Node z1.example.com (1): standby
   Online: [ z2.example.com ]
   
   Full list of resources:
   
    myapc	(stonith:fence_apc_snmp):	Started z1.example.com
    Resource Group: apachegroup
        my_lvm	(ocf::heartbeat:LVM-activate):	Started z2.example.com
        my_fs	(ocf::heartbeat:Filesystem):	Started z2.example.com
        VirtualIP	(ocf::heartbeat:IPaddr2):	Started z2.example.com
        Website	(ocf::heartbeat:apache):	Started z2.example.com

   The web site at the defined IP address should still display, without interruption.

3. To remove z1 from standby mode, enter the following command.

   [root@z1 ~]# pcs node unstandby z1.example.com

   Note

   Removing a node from standby mode does not in itself cause the resources to fail back over to that node. This will depend on the resource-stickiness value for the resources. For information about the resource-stickiness meta attribute, see Configuring a resource to prefer its current node.

Procedure

1. On both nodes of the cluster, perform the following steps to set the value for the LVM system ID to the value of the uname identifier for the system. The LVM system ID will be used to ensure that only the cluster is capable of activating the volume group.

   1. Set the system_id_source configuration option in the /etc/lvm/lvm.conf configuration file to uname.

      # Configuration option global/system_id_source.
      system_id_source = "uname"

   2. Verify that the LVM system ID on the node matches the uname for the node.

      # lvm systemid
        system ID: z1.example.com
      # uname -n
        z1.example.com

2. Create the LVM volume and create an XFS file system on that volume. Since the /dev/sdb1 partition is storage that is shared, you perform this part of the procedure on one node only.

   1. Create an LVM physical volume on partition /dev/sdb1.

      [root@z1 ~]# pvcreate /dev/sdb1
        Physical volume "/dev/sdb1" successfully created

   2. Create the volume group my_vg that consists of the physical volume /dev/sdb1.

      Specify the --setautoactivation n flag to ensure that volume groups managed by Pacemaker in a cluster will not be automatically activated on startup. If you are using an existing volume group for the LVM volume you are creating, you can reset this flag with the vgchange --setautoactivation n command for the volume group.

      [root@z1 ~]# vgcreate --setautoactivation n my_vg /dev/sdb1
        Volume group "my_vg" successfully created

      Note

      If your LVM volume group contains one or more physical volumes that reside on remote block storage, such as an iSCSI target, Red Hat recommends that you ensure that the service starts before Pacemaker starts. For information about configuring startup order for a remote physical volume used by a Pacemaker cluster, see Configuring startup order for resource dependencies not managed by Pacemaker.

   3. Verify that the new volume group has the system ID of the node on which you are running and from which you created the volume group.

      [root@z1 ~]# vgs -o+systemid
        VG    #PV #LV #SN Attr   VSize  VFree  System ID
        my_vg   1   0   0 wz--n- <1.82t <1.82t z1.example.com

   4. Create a logical volume using the volume group my_vg.

      [root@z1 ~]# lvcreate -L450 -n my_lv my_vg
        Rounding up size to full physical extent 452.00 MiB
        Logical volume "my_lv" created

      You can use the lvs command to display the logical volume.

      [root@z1 ~]# lvs
        LV      VG      Attr      LSize   Pool Origin Data%  Move Log Copy%  Convert
        my_lv   my_vg   -wi-a---- 452.00m
        ...

   5. Create an XFS file system on the logical volume my_lv.

      [root@z1 ~]# mkfs.xfs /dev/my_vg/my_lv
      meta-data=/dev/my_vg/my_lv       isize=512    agcount=4, agsize=28928 blks
               =                       sectsz=512   attr=2, projid32bit=1
      ...

3. If the use of a devices file is enabled with the use_devicesfile = 1 parameter in the lvm.conf file, add the shared device to the devices file on the second node in the cluster. This feature is enabled by default.

   [root@z2 ~]# lvmdevices --adddev /dev/sdb1

Procedure

1. On both nodes in the cluster, create the /nfsshare directory.

   # mkdir /nfsshare

2. On one node in the cluster, perform the following procedure.

   1. Ensure that the logical volume you you created in Configuring an LVM volume with an XFS file system in a Pacemaker cluster is activated, then mount the file system you created on the logical volume on the /nfsshare directory.

      [root@z1 ~]# lvchange -ay my_vg/my_lv
      [root@z1 ~]# mount /dev/my_vg/my_lv /nfsshare

   2. Create an exports directory tree on the /nfsshare directory.

      [root@z1 ~]# mkdir -p /nfsshare/exports
      [root@z1 ~]# mkdir -p /nfsshare/exports/export1
      [root@z1 ~]# mkdir -p /nfsshare/exports/export2

   3. Place files in the exports directory for the NFS clients to access. For this example, we are creating test files named clientdatafile1 and clientdatafile2.

      [root@z1 ~]# touch /nfsshare/exports/export1/clientdatafile1
      [root@z1 ~]# touch /nfsshare/exports/export2/clientdatafile2

   4. Unmount the file system and deactivate the LVM volume group.

      [root@z1 ~]# umount /dev/my_vg/my_lv
      [root@z1 ~]# vgchange -an my_vg

Procedure

1. Create the LVM-activate resource named my_lvm. Because the resource group nfsgroup does not yet exist, this command creates the resource group.

   Warning

   Do not configure more than one LVM-activate resource that uses the same LVM volume group in an active/passive HA configuration, as this risks data corruption. Additionally, do not configure an LVM-activate resource as a clone resource in an active/passive HA configuration.

   [root@z1 ~]# pcs resource create my_lvm ocf:heartbeat:LVM-activate vgname=my_vg vg_access_mode=system_id --group nfsgroup

2. Check the status of the cluster to verify that the resource is running.

   root@z1 ~]#  pcs status
   Cluster name: my_cluster
   Last updated: Thu Jan  8 11:13:17 2015
   Last change: Thu Jan  8 11:13:08 2015
   Stack: corosync
   Current DC: z2.example.com (2) - partition with quorum
   Version: 1.1.12-a14efad
   2 Nodes configured
   3 Resources configured
   
   Online: [ z1.example.com z2.example.com ]
   
   Full list of resources:
    myapc  (stonith:fence_apc_snmp):       Started z1.example.com
    Resource Group: nfsgroup
        my_lvm     (ocf::heartbeat:LVM-activate):   Started z1.example.com
   
   PCSD Status:
     z1.example.com: Online
     z2.example.com: Online
   
   Daemon Status:
     corosync: active/enabled
     pacemaker: active/enabled
     pcsd: active/enabled

3. Configure a Filesystem resource for the cluster.

   The following command configures an XFS Filesystem resource named nfsshare as part of the nfsgroup resource group. This file system uses the LVM volume group and XFS file system you created in Configuring an LVM volume with an XFS file system in a cluster and will be mounted on the /nfsshare directory you created in Configuring an NFS share.

   [root@z1 ~]# pcs resource create nfsshare Filesystem device=/dev/my_vg/my_lv directory=/nfsshare fstype=xfs --group nfsgroup

   You can specify mount options as part of the resource configuration for a Filesystem resource with the options=options parameter. Run the pcs resource describe Filesystem command for full configuration options.

4. Verify that the my_lvm and nfsshare resources are running.

   [root@z1 ~]# pcs status
   ...
   Full list of resources:
    myapc  (stonith:fence_apc_snmp):       Started z1.example.com
    Resource Group: nfsgroup
        my_lvm     (ocf::heartbeat:LVM-activate):   Started z1.example.com
        nfsshare   (ocf::heartbeat:Filesystem):    Started z1.example.com
   ...

5. Create the nfsserver resource named nfs-daemon as part of the resource group nfsgroup.

   Note

   The nfsserver resource allows you to specify an nfs_shared_infodir parameter, which is a directory that NFS servers use to store NFS-related stateful information.

   It is recommended that this attribute be set to a subdirectory of one of the Filesystem resources you created in this collection of exports. This ensures that the NFS servers are storing their stateful information on a device that will become available to another node if this resource group needs to relocate. In this example;

   • /nfsshare is the shared-storage directory managed by the Filesystem resource
   • /nfsshare/exports/export1 and /nfsshare/exports/export2 are the export directories
   • /nfsshare/nfsinfo is the shared-information directory for the nfsserver resource

   [root@z1 ~]# pcs resource create nfs-daemon nfsserver nfs_shared_infodir=/nfsshare/nfsinfo nfs_no_notify=true --group nfsgroup
   
   [root@z1 ~]# pcs status
   ...

6. Add the exportfs resources to export the /nfsshare/exports directory. These resources are part of the resource group nfsgroup. This builds a virtual directory for NFSv4 clients. NFSv3 clients can access these exports as well.

   Note

   The fsid=0 option is required only if you want to create a virtual directory for NFSv4 clients. For more information, see the Red Hat Knowledgebase solution How do I configure the fsid option in an NFS server’s /etc/exports file?.

   [root@z1 ~]# pcs resource create nfs-root exportfs clientspec=192.168.122.0/255.255.255.0 options=rw,sync,no_root_squash directory=/nfsshare/exports fsid=0 --group nfsgroup
   
   [root@z1 ~]# pcs resource create nfs-export1 exportfs clientspec=192.168.122.0/255.255.255.0 options=rw,sync,no_root_squash directory=/nfsshare/exports/export1 fsid=1 --group nfsgroup
   
   [root@z1 ~]# pcs resource create nfs-export2 exportfs clientspec=192.168.122.0/255.255.255.0 options=rw,sync,no_root_squash directory=/nfsshare/exports/export2 fsid=2 --group nfsgroup

7. Add the floating IP address resource that NFS clients will use to access the NFS share. This resource is part of the resource group nfsgroup. For this example deployment, we are using 192.168.122.200 as the floating IP address.

   [root@z1 ~]# pcs resource create nfs_ip IPaddr2 ip=192.168.122.200 cidr_netmask=24 --group nfsgroup

8. Add an nfsnotify resource for sending NFSv3 reboot notifications once the entire NFS deployment has initialized. This resource is part of the resource group nfsgroup.

   Note

   For the NFS notification to be processed correctly, the floating IP address must have a host name associated with it that is consistent on both the NFS servers and the NFS client.

   [root@z1 ~]# pcs resource create nfs-notify nfsnotify source_host=192.168.122.200 --group nfsgroup

9. After creating the resources and the resource constraints, you can check the status of the cluster. Note that all resources are running on the same node.

   [root@z1 ~]# pcs status
   ...
   Full list of resources:
    myapc  (stonith:fence_apc_snmp):       Started z1.example.com
    Resource Group: nfsgroup
        my_lvm     (ocf::heartbeat:LVM-activate):   Started z1.example.com
        nfsshare   (ocf::heartbeat:Filesystem):    Started z1.example.com
        nfs-daemon (ocf::heartbeat:nfsserver):     Started z1.example.com
        nfs-root   (ocf::heartbeat:exportfs):      Started z1.example.com
        nfs-export1        (ocf::heartbeat:exportfs):      Started z1.example.com
        nfs-export2        (ocf::heartbeat:exportfs):      Started z1.example.com
        nfs_ip     (ocf::heartbeat:IPaddr2):       Started  z1.example.com
        nfs-notify (ocf::heartbeat:nfsnotify):     Started z1.example.com
   ...

1. If you are running the firewalld daemon on your cluster nodes, ensure that the ports that your system requires for NFS access are enabled on all nodes.

2. On a node outside of the cluster, residing in the same network as the deployment, verify that the NFS share can be seen by mounting the NFS share. For this example, we are using the 192.168.122.0/24 network.

   # showmount -e 192.168.122.200
   Export list for 192.168.122.200:
   /nfsshare/exports/export1 192.168.122.0/255.255.255.0
   /nfsshare/exports         192.168.122.0/255.255.255.0
   /nfsshare/exports/export2 192.168.122.0/255.255.255.0

3. To verify that you can mount the NFS share with NFSv4, mount the NFS share to a directory on the client node. After mounting, verify that the contents of the export directories are visible. Unmount the share after testing.

   # mkdir nfsshare
   # mount -o "vers=4" 192.168.122.200:export1 nfsshare
   # ls nfsshare
   clientdatafile1
   # umount nfsshare

4. Verify that you can mount the NFS share with NFSv3. After mounting, verify that the test file clientdatafile1 is visible. Unlike NFSv4, since NFSv3 does not use the virtual file system, you must mount a specific export. Unmount the share after testing.

   # mkdir nfsshare
   # mount -o "vers=3" 192.168.122.200:/nfsshare/exports/export2 nfsshare
   # ls nfsshare
   clientdatafile2
   # umount nfsshare

1. On a node outside of the cluster, mount the NFS share and verify access to the clientdatafile1 file you created in Configuring an NFS share.

   # mkdir nfsshare
   # mount -o "vers=4" 192.168.122.200:export1 nfsshare
   # ls nfsshare
   clientdatafile1

2. From a node within the cluster, determine which node in the cluster is running nfsgroup. In this example, nfsgroup is running on z1.example.com.

   [root@z1 ~]# pcs status
   ...
   Full list of resources:
    myapc  (stonith:fence_apc_snmp):       Started z1.example.com
    Resource Group: nfsgroup
        my_lvm     (ocf::heartbeat:LVM-activate):   Started z1.example.com
        nfsshare   (ocf::heartbeat:Filesystem):    Started z1.example.com
        nfs-daemon (ocf::heartbeat:nfsserver):     Started z1.example.com
        nfs-root   (ocf::heartbeat:exportfs):      Started z1.example.com
        nfs-export1        (ocf::heartbeat:exportfs):      Started z1.example.com
        nfs-export2        (ocf::heartbeat:exportfs):      Started z1.example.com
        nfs_ip     (ocf::heartbeat:IPaddr2):       Started  z1.example.com
        nfs-notify (ocf::heartbeat:nfsnotify):     Started z1.example.com
   ...

3. From a node within the cluster, put the node that is running nfsgroup in standby mode.

   [root@z1 ~]# pcs node standby z1.example.com

4. Verify that nfsgroup successfully starts on the other cluster node.

   [root@z1 ~]# pcs status
   ...
   Full list of resources:
    Resource Group: nfsgroup
        my_lvm     (ocf::heartbeat:LVM-activate):   Started z2.example.com
        nfsshare   (ocf::heartbeat:Filesystem):    Started z2.example.com
        nfs-daemon (ocf::heartbeat:nfsserver):     Started z2.example.com
        nfs-root   (ocf::heartbeat:exportfs):      Started z2.example.com
        nfs-export1        (ocf::heartbeat:exportfs):      Started z2.example.com
        nfs-export2        (ocf::heartbeat:exportfs):      Started z2.example.com
        nfs_ip     (ocf::heartbeat:IPaddr2):       Started  z2.example.com
        nfs-notify (ocf::heartbeat:nfsnotify):     Started z2.example.com
   ...

5. From the node outside the cluster on which you have mounted the NFS share, verify that this outside node still continues to have access to the test file within the NFS mount.

   # ls nfsshare
   clientdatafile1

   Service will be lost briefly for the client during the failover but the client should recover it with no user intervention. By default, clients using NFSv4 may take up to 90 seconds to recover the mount; this 90 seconds represents the NFSv4 file lease grace period observed by the server on startup. NFSv3 clients should recover access to the mount in a matter of a few seconds.

6. From a node within the cluster, remove the node that was initially running nfsgroup from standby mode.

   Note

   Removing a node from standby mode does not in itself cause the resources to fail back over to that node. This will depend on the resource-stickiness value for the resources. For information about the resource-stickiness meta attribute, see Configuring a resource to prefer its current node.

   [root@z1 ~]# pcs node unstandby z1.example.com

Procedure

1. Use SSH, Telnet, HTTP, or whatever remote protocol is used to connect to the device to manually log in and test the fence device or see what output is given. For example, if you will be configuring fencing for an IPMI-enabled device,then try to log in remotely with ipmitool. Take note of the options used when logging in manually because those options might be needed when using the fencing agent.

   If you are unable to log in to the fence device, verify that the device is pingable, there is nothing such as a firewall configuration that is preventing access to the fence device, remote access is enabled on the fencing device, and the credentials are correct.

2. Run the fence agent manually, using the fence agent script. This does not require that the cluster services are running, so you can perform this step before the device is configured in the cluster. This can ensure that the fence device is responding properly before proceeding.

   Note

   These examples use the fence_ipmilan fence agent script for an iLO device. The actual fence agent you will use and the command that calls that agent will depend on your server hardware. You should consult the man page for the fence agent you are using to determine which options to specify. You will usually need to know the login and password for the fence device and other information related to the fence device.

   The following example shows the format you would use to run the fence_ipmilan fence agent script with -o status parameter to check the status of the fence device interface on another node without actually fencing it. This allows you to test the device and get it working before attempting to reboot the node. When running this command, you specify the name and password of an iLO user that has power on and off permissions for the iLO device.

   # fence_ipmilan -a ipaddress -l username -p password -o status

   The following example shows the format you would use to run the fence_ipmilan fence agent script with the -o reboot parameter. Running this command on one node reboots the node managed by this iLO device.

   # fence_ipmilan -a ipaddress -l username -p password -o reboot

   If the fence agent failed to properly do a status, off, on, or reboot action, you should check the hardware, the configuration of the fence device, and the syntax of your commands. In addition, you can run the fence agent script with the debug output enabled. The debug output is useful for some fencing agents to see where in the sequence of events the fencing agent script is failing when logging into the fence device.

   # fence_ipmilan -a ipaddress -l username -p password -o status -D /tmp/$(hostname)-fence_agent.debug

   When diagnosing a failure that has occurred, you should ensure that the options you specified when manually logging in to the fence device are identical to what you passed on to the fence agent with the fence agent script.

   For fence agents that support an encrypted connection, you may see an error due to certificate validation failing, requiring that you trust the host or that you use the fence agent’s ssl-insecure parameter. Similarly, if SSL/TLS is disabled on the target device, you may need to account for this when setting the SSL parameters for the fence agent.

   Note

   If the fence agent that is being tested is a fence_drac, fence_ilo, or some other fencing agent for a systems management device that continues to fail, then fall back to trying fence_ipmilan. Most systems management cards support IPMI remote login and the only supported fencing agent is fence_ipmilan.

3. Once the fence device has been configured in the cluster with the same options that worked manually and the cluster has been started, test fencing with the pcs stonith fence command from any node (or even multiple times from different nodes), as in the following example. The pcs stonith fence command reads the cluster configuration from the CIB and calls the fence agent as configured to execute the fence action. This verifies that the cluster configuration is correct.

   # pcs stonith fence node_name

   If the pcs stonith fence command works properly, that means the fencing configuration for the cluster should work when a fence event occurs. If the command fails, it means that cluster management cannot invoke the fence device through the configuration it has retrieved. Check for the following issues and update your cluster configuration as needed.

   • Check your fence configuration. For example, if you have used a host map you should ensure that the system can find the node using the host name you have provided.
   • Check whether the password and user name for the device include any special characters that could be misinterpreted by the bash shell. Making sure that you enter passwords and user names surrounded by quotation marks could address this issue.
   • Check whether you can connect to the device using the exact IP address or host name you specified in the pcs stonith command. For example, if you give the host name in the stonith command but test by using the IP address, that is not a valid test.

   • If the protocol that your fence device uses is accessible to you, use that protocol to try to connect to the device. For example many agents use ssh or telnet. You should try to connect to the device with the credentials you provided when configuring the device, to see if you get a valid prompt and can log in to the device.

     If you determine that all your parameters are appropriate but you still have trouble connecting to your fence device, you can check the logging on the fence device itself, if the device provides that, which will show if the user has connected and what command the user issued. You can also search through the /var/log/messages file for instances of stonith and error, which could give some idea of what is transpiring, but some agents can provide additional information.

4. Once the fence device tests are working and the cluster is up and running, test an actual failure. To do this, take an action in the cluster that should initiate a token loss.

   • Take down a network. How you take a network depends on your specific configuration. In many cases, you can physically pull the network or power cables out of the host. For information about simulating a network failure, see the Red Hat Knowledgebase solution What is the proper way to simulate a network failure on a RHEL Cluster?.

     Note

     Disabling the network interface on the local host rather than physically disconnecting the network or power cables is not recommended as a test of fencing because it does not accurately simulate a typical real-world failure.

   • Block corosync traffic both inbound and outbound using the local firewall.

     The following example blocks corosync, assuming the default corosync port is used, firewalld is used as the local firewall, and the network interface used by corosync is in the default firewall zone:

     # firewall-cmd --direct --add-rule ipv4 filter OUTPUT 2 -p udp --dport=5405 -j DROP
     # firewall-cmd --add-rich-rule='rule family="ipv4" port port="5405" protocol="udp" drop

   • Simulate a crash and panic your machine with sysrq-trigger. Note, however, that triggering a kernel panic can cause data loss; it is recommended that you disable your cluster resources first.

     # echo c > /proc/sysrq-trigger

Procedure

1. Add a fencing level of 1 for fence device my_ilo on node rh7-2.

   # pcs stonith level add 1 rh7-2 my_ilo

2. Add a fencing level of 2 for fence device my_apc on node rh7-2.

   # pcs stonith level add 2 rh7-2 my_apc

3. List the currently configured fencing levels.

   # pcs stonith level
    Node: rh7-2
     Level 1 - my_ilo
     Level 2 - my_apc

Procedure

1. Create the first fence device.

   # pcs stonith create apc1 fence_apc_snmp ipaddr=apc1.example.com login=user passwd='7a4D#1j!pz864' pcmk_host_map="node1.example.com:1;node2.example.com:2"

2. Create the second fence device.

   # pcs stonith create apc2 fence_apc_snmp ipaddr=apc2.example.com login=user passwd='7a4D#1j!pz864' pcmk_host_map="node1.example.com:1;node2.example.com:2"

3. Specify that both devices are required to fence the node.

   # pcs stonith level add 1 node1.example.com apc1,apc2
   # pcs stonith level add 1 node2.example.com apc1,apc2

Procedure

1. Create a resource with a resource-stickiness value of 50.

   # pcs resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.120 meta resource-stickiness=50

2. For the existing resource named dummy_resource, set the failure-timeout meta option to 20 seconds, so that the resource can attempt to restart on the same node in 20 seconds.

   # pcs resource meta dummy_resource failure-timeout=20s

3. Display the values for the resource to verify that failure-timeout=20s is set.

   # pcs resource config dummy_resource
    Resource: dummy_resource (class=ocf provider=heartbeat type=Dummy)
     Meta Attrs: failure-timeout=20s
     ...

Procedure

1. Reset the default values for or any ocf:pacemaker:pgsql resource.

   # pcs resource defaults set create id=set-1 score=100 meta resource-stickiness=10 rule resource ocf:pacemaker:pgsql

2. Display the configured resource default values in plain text.

   # pcs resource defaults config
   Meta Attrs: build-resource-defaults
     resource-stickiness=1
   Meta Attrs: set-1 score=100
     resource-stickiness=10
     Rule: boolean-op=and score=INFINITY
       Expression: resource ocf:pacemaker:pgsql

3. Display the pcs resource defaults commands created from the current cluster defaults configuration.

   # pcs resource defaults config --output-format=cmd
   pcs -- resource defaults set create id=build-resource-defaults \
     meta resource-stickiness=1;
   pcs -- resource defaults set create id=set-1 score=100 \
     meta resource-stickiness=10 \
     rule 'resource ocf:pacemaker:pgsql'

4. Display the configured resource default values in JSON format.

   # pcs resource defaults config --output-format=json
   {"instance_attributes": [], "meta_attributes": [{"id": "build-resource-defaults", "options": {}, "rule": null, "nvpairs": [{"id": "build-resource-stickiness", "name": "resource-stickiness", "value": "1"}]}, {"id": "set-1", "options": {"score": "100"}, "rule": {"id": "set-1-rule", "type": "RULE", "in_effect": "UNKNOWN", "options": {"boolean-op": "and", "score": "INFINITY"}, "date_spec": null, "duration": null, "expressions": [{"id": "set-1-rule-rsc-ocf-pacemaker-pgsql", "type": "RSC_EXPRESSION", "in_effect": "UNKNOWN", "options": {"class": "ocf", "provider": "pacemaker", "type": "pgsql"}, "date_spec": null, "duration": null, "expressions": [], "as_string": "resource ocf:pacemaker:pgsql"}], "as_string": "resource ocf:pacemaker:pgsql"}, "nvpairs": [{"id": "set-1-resource-stickiness", "name": "resource-stickiness", "value": "10"}]}]}

Procedure

1. Create an LVM-activate resource.

   # pcs resource create my_lvm ocf:heartbeat:LVM-activate vgname=my_vg vg_access_mode=system_id --group apachegroup

2. Create a Filesystem resource.

   # pcs resource create my_fs Filesystem device="/dev/my_vg/my_lv" directory="/var/www" fstype="xfs" --group apachegroup

3. Create an IPaddr2 resource.

   # pcs resource create VirtualIP IPaddr2 ip=198.51.100.3 cidr_netmask=24 --group apachegroup

4. Create an Apache resource.

   # pcs resource create Website apache configfile="/etc/httpd/conf/httpd.conf" statusurl="http://127.0.0.1/server-status" --group apachegroup

5. Display the pcs commands you can use to re-create the four resources you created on a different system.

   # pcs resource config --output-format=cmd
   pcs resource create --no-default-ops --force -- my_lvm ocf:heartbeat:LVM-activate \
     vg_access_mode=system_id vgname=my_vg \
     op \
       monitor interval=30s id=my_lvm-monitor-interval-30s timeout=90s \
       start interval=0s id=my_lvm-start-interval-0s timeout=90s \
       stop interval=0s id=my_lvm-stop-interval-0s timeout=90s;
   pcs resource create --no-default-ops --force -- my_fs ocf:heartbeat:Filesystem \
     device=/dev/my_vg/my_lv directory=/var/www fstype=xfs \
     op \
       monitor interval=20s id=my_fs-monitor-interval-20s timeout=40s \
       start interval=0s id=my_fs-start-interval-0s timeout=60s \
       stop interval=0s id=my_fs-stop-interval-0s timeout=60s;
   pcs resource create --no-default-ops --force -- VirtualIP ocf:heartbeat:IPaddr2 \
     cidr_netmask=24 ip=198.51.100.3 \
     op \
       monitor interval=10s id=VirtualIP-monitor-interval-10s timeout=20s \
       start interval=0s id=VirtualIP-start-interval-0s timeout=20s \
       stop interval=0s id=VirtualIP-stop-interval-0s timeout=20s;
   pcs resource create --no-default-ops --force -- Website ocf:heartbeat:apache \
     configfile=/etc/httpd/conf/httpd.conf statusurl=http://127.0.0.1/server-status \
     op \
       monitor interval=10s id=Website-monitor-interval-10s timeout=20s \
       start interval=0s id=Website-start-interval-0s timeout=40s \
       stop interval=0s id=Website-stop-interval-0s timeout=60s;
   pcs resource group add apachegroup \
     my_lvm my_fs VirtualIP Website

6. Display the pcs command you can use to re-create only the IPaddr2 resource. To display only one configured resource, specify the resource ID for that resource.

   # pcs resource config VirtualIP --output-format=cmd
   pcs resource create --no-default-ops --force -- VirtualIP ocf:heartbeat:IPaddr2 \
     cidr_netmask=24 ip=198.51.100.3 \
     op \
       monitor interval=10s id=VirtualIP-monitor-interval-10s timeout=20s \
       start interval=0s id=VirtualIP-start-interval-0s timeout=20s \
       stop interval=0s id=VirtualIP-stop-interval-0s timeout=20s

Procedure

1. Display the initial values of the configured parameters for resource VirtualIP.

   # pcs resource config VirtualIP
    Resource: VirtualIP (type=IPaddr2 class=ocf provider=heartbeat)
     Attributes: ip=192.168.0.120 cidr_netmask=24
     Operations: monitor interval=30s

2. Change the value of the ip parameter for resource VirtualIP.

   # pcs resource update VirtualIP ip=192.169.0.120

3. Display the values of the parameters for resource VirtualIP after you have modifed the value of the ip parameter.

   # pcs resource config VirtualIP
    Resource: VirtualIP (type=IPaddr2 class=ocf provider=heartbeat)
     Attributes: ip=192.169.0.120 cidr_netmask=24
     Operations: monitor interval=30s

1. Add a ping resource to the cluster. The ping resource uses the system utility of the same name to test if a list of machines (specified by DNS host name or IPv4/IPv6 address) are reachable and uses the results to maintain a node attribute called pingd.
2. Configure a location constraint for the resource that will move the resource to a different node when connectivity is lost.

Procedure

1. Configure a ping resource. You configure the resource as a clone so that the resource will run on all cluster nodes.

   # pcs resource create ping ocf:pacemaker:ping dampen=5s multiplier=1000 host_list=gateway.example.com clone

2. Configure a location constraint rule for the existing resource named Webserver. This causes the Webserver resource to move to a host that is able to ping gateway.example.com if the host that it is currently running on cannot ping gateway.example.com.

   # pcs constraint location Webserver rule score=-INFINITY pingd lt 1 or not_defined pingd

Procedure

1. On one node of the cluster, create the resource clone.

   When you create a clone of a resource, by default the clone takes on the name of the resource with -clone appended to the name. The following command creates a resource of type apache named webfarm and a clone of that resource named webfarm-clone.

   # pcs resource create webfarm apache clone

2. Remove the clone of a resource or a resource group. This does not remove the resource or resource group itself.

   # pcs resource unclone webfarm

Procedure

1. On the new node to add to the cluster, perform the following tasks:

   1. Install the cluster packages. If the cluster uses SBD, the Booth ticket manager, or a quorum device, you must manually install the respective packages (sbd, booth-site, corosync-qdevice) on the new node as well.

      [root@newnode ~]# dnf install -y pcs fence-agents-all

      In addition to the cluster packages, you will also need to install and configure all of the services that you are running in the cluster, which you have installed on the existing cluster nodes. For example, if you are running an Apache HTTP server in a Red Hat high availability cluster, you will need to install the server on the node you are adding, as well as the wget tool that checks the status of the server.

   2. If you are running the firewalld daemon, execute the following commands to enable the ports that are required by the Red Hat High Availability Add-On.

      # firewall-cmd --permanent --add-service=high-availability
      # firewall-cmd --add-service=high-availability

   3. Set a password for the user ID hacluster. It is recommended that you use the same password for each node in the cluster.

      [root@newnode ~]# passwd hacluster
      Changing password for user hacluster.
      New password:
      Retype new password:
      passwd: all authentication tokens updated successfully.

   4. Execute the following commands to start the pcsd service and to enable pcsd at system start.

      # systemctl start pcsd.service
      # systemctl enable pcsd.service

2. On a node in the existing cluster, perform the following tasks.

   1. Authenticate user hacluster on the new cluster node.

      [root@clusternode-01 ~]# pcs host auth newnode.example.com
      Username: hacluster
      Password:
      newnode.example.com: Authorized

   2. Add the new node to the existing cluster. This command also syncs the cluster configuration file corosync.conf to all nodes in the cluster, including the new node you are adding.

      [root@clusternode-01 ~]# pcs cluster node add newnode.example.com

3. On the new node to add to the cluster, perform the following tasks.

   1. Start and enable cluster services on the new node.

      [root@newnode ~]# pcs cluster start
      Starting Cluster...
      [root@newnode ~]# pcs cluster enable

   2. Ensure that you configure and test a fencing device for the new cluster node.

Procedure

1. Set the health-node-strategy cluster property to define how Pacemaker responds to changes in node health.

   # pcs property set node-health-strategy=migrate-on-red

2. Create a cloned cluster resource that uses a health node resource agent, setting the allow-unhealthy-nodes resource meta option to define whether the cluster will detect if the node’s health recovers and move resources back to the node. Configure this resource with a recurring monitor action, to continually check the health of all nodes.

   This example creates a HealthIOWait resource agent to monitor the CPU I/O wait, setting a red limit for moving resources off a node to 15%. This command sets the allow-unhealthy-nodes resource meta option to true and configures a recurring monitor interval of 10 seconds.

   # pcs resource create io-monitor ocf:pacemaker:HealthIOWait red_limit=15 op monitor interval=10s meta allow-unhealthy-nodes=true clone

Procedure

1. On every node in the cluster, add the user to the haclient group:

   # usermod -a -G haclient <username>

   Replace <username> with the name of the user to whom you are granting permission.

1. Execute the pcs acl role create…​ command to create a role which defines the permissions for that role.
2. Assign the role you created to a user with the pcs acl user create command. If you assign multiple roles to the same user, any deny permission takes precedence, then write, then read.

Procedure

1. This procedure requires that the user rouser exists on the local system and that the user rouser is a member of the group haclient.

   # adduser rouser
   # usermod -a -G haclient rouser

2. Enable Pacemaker ACLs with the pcs acl enable command.

   # pcs acl enable

3. Create a role named read-only with read-only permissions for the cib.

   # pcs acl role create read-only description="Read access to cluster" read xpath /cib

4. Create the user rouser in the pcs ACL system and assign that user the read-only role.

   # pcs acl user create rouser read-only

5. View the current ACLs.

   # pcs acl
   User: rouser
     Roles: read-only
   Role: read-only
     Description: Read access to cluster
     Permission: read xpath /cib (read-only-read)

6. On each node where rouser will run pcs commands, log in as rouser and authenticate to the local pcsd service. This is required in order to run certain pcs commands, such as pcs status, as the ACL user.

   [rouser ~]$ pcs client local-auth